Posted on by Jean-Luc Aufranc (CNXSoft) - 33 Comments on NanoPi R5S preview – Part 2: Ubuntu 20.04 (FriendlyCore)

NanoPi R5S preview – Part 2: Ubuntu 20.04 (FriendlyCore)

I started the NanoPi R5S review with an unboxing, a teardown, a quick try of the pre-installed OpenWrt-based FriendlyWrt, and some iperf3 benchmarks on the 2.5GbE interfaces that were rather disappointing. I test further I switched to the Ubuntu 20.04-based FriendlyCore image since I’m more familiar with Debian-based operating systems, and some tools will not run on OpenWrt. Note the performance is still not quite optimal, and that’s why I call this a preview since numbers should improve in the next few months as more people tweak the software.

OpenWrt optimizations?

But before jumping to Ubuntu, I gave an updated version of FriendlyWrt a try as FriendElec told me they had added some optimizations:

We have made some optimizations on the new image, such as NIC interrupt settings, and offload support…

So I downloaded “rk3568-eflasher-friendlywrt-20220526.img.gz” found on Google Drive, flashed it to a microSD card with USBImager, and booted it to the router.

It will automatically flash the image to the eMMC flash, If you connect a monitor you can follow the result. Once it’s done, remove the microSD card, and power cycle the router.

eflasher friendlywrt

You can check the status by connecting an HDMI monitor (as shown above), or checking out the LEDs on the device. It’s very fast and the installation to the eMMC flash takes only a few seconds.

The main changes were made to the 40-net-smp-affinity file. In the pre-installed  FriendlyWrt, it looks like that:


While the new 40-net-smp-affinity file is indeed different:


Willy Tarreau explains the changes made for the eth1 interface:

It involves RPS …. i.e. they receive this IRQ on core 2 and redistribute the incoming traffic to cores 0,1,3. That’s the right way to use RPS. However you have to manually assign iperf and watch the first core that saturates. If it’s saturating core 2 with ksoftirqd first, make sure that iperf runs on any of the other 3. If core2 is slightly idle, try to put iperf on it. If putting it on it makes ksoftirqd pop up, then they’re hindering each other, and you’d rather change the RPS setting to free another core and use it for iperf.

I did try this method before testing and switching to Ubuntu, and my results were even worse with the new FriendlyWrt image:


So this would have to be revisited.

M.2 NVMe SSD installation in NanoPi R5S

I’ve just purchased an APACER AS2280 (AP256GAS2280P4-1) PCIe Gen 3.0 x4 SSD that can achieve up to 1,800 MB/s sequential read and up to 1,100 MB/s sequential write speeds on the right hardware.

Apacer M.2 2280 PCIe SSD

The installation is straightforward as I just had to loosen four screws to remove the bottom cover, install the SSD, and fasten it with the provided screw.

NanoPi R5S M.2 NVMe SSD

Installing Ubuntu 20.04 FriendlyCore on NanoPi R5S

I first attempted to install FriendlyCore using the eflasher image.

eflasher friendlycore

It looked good, so I restarted the router, but then I noticed the WAN interface link would not show on the TP-Link switch, and only the power LED was turned on (It happens the latter is normal for the FriendlyCore/Ubuntu image). I tried again, going into eflasher UI settings by clicking on Finish, but still no luck.

Eflasher FriendlyCore install NanoPi RS5

So instead I downloaded the “SD” image to boot directly from the microSD card and run the OS from there. It works fine. If you intend to use NanoPi R5S for multiple purposes and expected a desktop environment in the Ubuntu 20.04 image, you’ll be disappointed, as the HDMI output is currently only used to access the terminal.

Ubuntu 20.04 FriendlyElec Login

FriendlyCore system information

You’ll find the boot log on CNX Software Pastebin. I logged in with SSH using pi/pi credentials (username/password), and upgraded the system to the latest packages with:


Let’s run some commands to get system information:


It looks all good, except the NVMe drive has not been mounted automatically. Let’s find more details with inxi:


Only eth0 WAN port is up, and eth1/eth2 2.5GbE ports are down, and not configured at all. FriendlyElec appears to mostly focus on the FriendlyWrt image, and they told me optimizations were not implemented on FriendlyCore yet, so most people should probably use FriendlyWrt instead since it will be easier to configure the network and router settings. I can see the Apacer AS2280P4 SSD is detected, and it turns out it’s not formatted out of the box, so I just formatted it with mkfs.ext4.

Benchmarking NanoPi R5S

Let’s run SBC Bench on the router to benchmark the CPU and potentially find some issues:


I launched this almost immediately after boot, so the dmesg output should be there in full (see boot load earlier in this preview/review), but the script is missing some information from it. The full output from sbc-bench.sh script can be found on pastebin, and we can notably see the “1992” MHz advertised frequency is tested to be 1845 MHz in reality, so some optimization may be possible here.

7zip is still faster than on NanoPi R2S router (3871), or about a 23% boost in performance, while AES-256-CBC 16KB is about 22% faster (704,872.45 vs 861,334.19kH/s)

NVMe benchmarking

I tested the NVMe SSD three times with iozone 3 three times, one with 100MB file:


then a 500MB file:


and finally a 1GB file:


The results are more or less consistent across all three tests without massive variations, and in last we’ve got about 380MB/s for read and write, well below the SSD advertised write/read speeds, and results for ODROID-M1, but that’s because of the PCIe 2.0 x1 interface used in this design, instead of the PCIe Gen 3.0 x2 interface used in the Hardkernel board.

Here’s lspci output for reference:

2.5GbE interfaces configuration and benchmarking

NanoPi R5S router review

Since only the eth0 Gigabit Ethernet “WAN” interface is configured out of the box, we have to configure the two 2.5GbE ports manually. I used the same testbed as in the first part of the review with FriendlyWrt, namely a Ubuntu 20.04 laptop with a Realtek RTL8156BG USB 3.0 to 2.5GbE dongle connected to eth1, and UP Xtreme i11 mini PC connected to eth2. Instead of using a bridge interface like in FriendlyWrt, I’ll configure two different subnets: 192.168.2.0 for eth1, and 192.168.3.0 for eth2.

Let’s create two new files in /etc/network/interfaces.d/:

  • eth1

  • eth2


Restart the network as follows:


Now install a DHCP server


edit /etc/dhcp/dhcpd.conf file with our two subnets:


… before restarting the dhcp server:


At this point the laptop and mini PC should get their IP address from NanoPi R5S on the respective subnets. We can start benchmarking the interfaces:

iperf3 download (Rx from R5S point-of-view) using eth1 connected to the laptop:


That’s a bit slower than what I got (1.85 Gbps) in OpenWrt, and there are retransmissions. I also monitored the system with sbc-bench.sh during the transfer:


The system performs at its maximum advertised frequency during the test, and I don’t any obvious bottleneck shere.

We can also check some information and stats with ethtool:


We did get some rx_mac_missed.

Let’s do that in reverse (Tx):


It looks quite better than in OpenWrt (1.12 Gbps).


IRQ percentages are much lower but I suppose that’s normal for Tx.

Let’s switch to eth2 connected to UP Xtreme i11:


Oh great! It’s the first time I get a decent 2.35 Gbps transfer, so there’s hope!


Unless I’m mistaken the 25% of IRQ should mean a core is fully utilized handling those.

Let’s try Tx:


1.59 Gbps. Not quite perfect, but still better than in OpenWrt.


Again the CPU operates are full speed, and far from 100% utilization so the bottlenecks must be somewhere else.  We can again check eth2 info and stats with ethtool.



We’ve got more rx_mac_missed here. So there will be some tweaks to improve the performance, but based on my experience with RTL8156B adjusting settings is really tricky, and experienced people don’t seem to agree on what to adjust, I’m talking about Realtek engineers working on RTL8156/8125 drivers vs regular readers here that are experts in networking.

Configuring NAT between the two 2.5GbE interfaces

Since the 2.5GbE interfaces don’t work optimally with iperf3, I did not bother testing the router performance in FriendlyWrt, but several people still asked. So I’ll show how I configured NAT in Ubuntu 20.04, and still test NAT performance bearing in mind it will certainly have improved in a few weeks or months.

We’ll need to enable IP forwarding and NAT. I used instructions adapted from a post on networkreverse.

Edit /etc/sysctl.conf to enable IP forwarding (uncomment the following line):


Apply the changes:


Now let’s enable NAT:


We can now ping UP Xtreme i11 on 192.168.3.0 subnet from my laptop on 192.168.2.0 subnet:


If you want to make the changes permanent:


Let’s try iperf3 between UP Xtreme i11 and my laptop with the data routed through the NanoPi R5S router.


768 Mbps in one direction and 937 Mbps in the other.


Monitoring with sbc-bench.sh show the processor runs at 1992 MHz (or 1845 MHz real), and again the 25% IRQ should mean one core is fully utilized to handle IRQs.

The mpstat command shows this should handled by core #0


And this can be confirmed by using top and htop.

top htop irq

NanoPi R5S power consumption

I’ve just got a wall power meter to check power consumption:

  • Idle – 5.1 Watts
  • Iperf3 to eth1 – 6.3 to 6.7 Watts
  • NAT test between laptop and mini PC – 6.2W

The numbers above are with Ubuntu 20.04 and the NVMe SSD installed.

I’ve also tested on NanoPi R5S with OpenWrt and no SSD:

  • Idle – 4.6 Watts
  • Iperf3  – 6.0 to 6.2 Watts

Note that since I’m using a wall power meter any efficiency loss in the power adapter (Khadas VIM4 USB-C power adapter) will be included and the values may be higher than with a USB-C power meter. It might also be possible to optimize power consumption with settings.

Final words

I’ll stop here for today. Optimizations should include changing the firmware to get Rockchip cores to run at 1992 MHz, and adjusting various settings related to PCIe and Ethernet settings, most of which I’m not familiar with (yet).

I’d like to thank FriendlyElec for sending review samples of NanoPi R5S mini router. The router with metal enclosure can be purchased on FriendlyElec website for $75, or you can get the board only for $59. The router can also be found on Aliexpress from several sellers, some of which also sell a 4GB RAM version, which is odd since FriendlyElec only sells the model with 2GB RAM at this time.

 

Share this:
FacebookTwitterHacker NewsSlashdotRedditLinkedInPinterestFlipboardMeWeLineEmailShare

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK 5 ITX RK3588 mini-ITX motherboard

33 Replies to “NanoPi R5S preview – Part 2: Ubuntu 20.04 (FriendlyCore)”

  1. > which is odd since FriendlyElec only sells the model with 2GB RAM at this time.

    In description at one of those sellers from Ali I see note:

    > Nanopi R5S 4GB Ram are presale, and it is expected to be shipped at June 15 

    Reply

  2. At first I was concerned about the bad ramlat perf report that could have indicated a deeper issue, but that’s just due to a missing gcc optimization. @tkaiser, I’ve just added a makefile to ramlat address this, let’s just run “make” now.

    I’ve run similar network tests on my StationPC-M2 (RK3566@1.8) with its native NIC, and monitored the performance and CPU usage. IRQs are only delivered to core 0. It correctly reaches line rate at Gbps in Tx (minus overhead since iperf3 only reports payload, i.e. 941 Mbps), and 95% in Rx. The CPU usage reaches 18% of a core on Tx if the same as the IRQ, 14% on another core, 49% for Rx on the same core as ksoftirq, and 67% when running on a different one. The CPU usage varies greatly based on the IRQ rate, hence the IRQ coalescence. This obviously improves with multiple streams but given that iperf3 cannot use more cores, I guess it quickly becomes the bottleneck in Rx at 2.5 GbE here, which confirms your measurements. Using RPS to deliver IRQs to other cores doesn’t help here as what is saved on one core cannot be reused by iperf. Running iperf on the mcbin which has 4 times the memory bandwidth saturates around 3Gbps while other tools have no difficulty filling the 10G link with HTTP traffic. This indicates that there are limitations caused by the SoC’s RAM bandwidth + NIC driver, and others by iperf’s limited architecture which requires strong single-threaded performance to use moderate links.

    Reply

  3. Hello,

    I’m looking for a relatively simple and affordable way to set up a control, especially parental control, for a home network. As I want to control all connections, I would like to install the control router between the fiber box and the ISP box. Can an R5S do the job in your opinion?

    Looking forward to your feedback.

    Best regards,

    Renaud

    Reply

  4. > the “1992” MHz advertised frequently is tested to be 1845 MHz in reality

    Quick digging through sbc-bench results collection shows that so far only Radxa and Firefly managed to get RK3568 to clock at advertised speeds:

    Reply

      1. Raw Markdown table data is available to be fed into whatever converter to end up with something sortable.

        Sbc-bench’s intention is producing insights and not numbers/graphs for brainless consumption. There are so many different use cases and the only number where sorting would make sense is the 7-zip score which vaguely represents ‘server tasks in general’. But if someone intends to run a server both questioning the numbers and switching to a better benchmarking mode is a great idea. 🙂

        Reply

        1. just suggesting it might be useful (several github sites, this), if tables increase in size (low, high, mean/mode/median [(n) values first]). Some examples show, this would be 15-30 lines javascript ( compared to browser extensions for several os/provider or external programs )
          ‘a/p benchmarking’ needs knowledge about hardware internals for being useful ( this depending on companies and policies, Rockchip seems getting China’s Rpi Ltd./ARM for people’s (advanced) education, at the moments )

          Reply

  5. In real life performance only works for devices that you work with interactively. For a router or NAS it seems to me the power efficiency is also important.

    Did you run the test with the below setting:
    $ cat /sys/module/pcie_aspm/parameters/policy
    default [performance] powersave powersupersave

    (I think there are more settings needed to trim the power consumption of an NVMe, but this is a big one.)

    Reply

    1. It’s set to default:

      Reply

    2. > /sys/module/pcie_aspm/parameters/policy

      This affects everything PCIe (also the two 2.5GbE NICs on the faster PCIe lanes). For initial benchmarking using anything other than ‘performance’ would be weird while for ‘real world operation’ settings need to be determined that provide a good compromise between consumption and sustained/peak performance.

      For SSD power consumption the SSD in question is the most important factor and there’s more than ASPM, check for example the ‘Supported Power States’ using ‘nvme’ or ‘smartctl -a’ :

         Transcend TS2TMTE220S:
         St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
          0 +     9.00W       –        –    0  0  0  0        0       0
         
         Samsung SSD 970 EVO+:
         St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
          0 +     7.80W       –        –    0  0  0  0        0       0
          1 +     6.00W       –        –    1  1  1  1        0       0
          2 +     3.40W       –        –    2  2  2  2        0       0
          3 –   0.0700W       –        –    3  3  3  3      210    1200
          4 –   0.0100W       –        –    4  4  4  4     2000    8000

      Reply

      1. You are right in making the distinction between initial tests and subsequent tests. With mature software a powersaving mode should have increased performance where the (current) performance mode is a practical limit.

        I have found no way to check if ASPM works right. The way I see it my device advertises 0,0040W when idle (P3 -> 0.0450W). What I measure when using NVMe instead of an SD-card is my SBC still uses around 0.5W extra when idle, even using the powersupersafe. I hope 0.3-0.4W extra savings are still possible.

        Currently, I don’t see the NVME as a positive addition since it significantly increases the power consumption, which is important for my application. The performance of an A55 SBC might be enough for some applications, but is nothing to rave about.

        The 2,5GB ethernet connections might be a sales point provided the energy consumptions stays within limits. When used 24/7 as router or NAS it is an important issue.

        Reply

        1. I found it hard to measure consumption with more complex scenarios just with a primitive meter only outputting actual amps/volts. Now happy with NetIO equipment I had to write a monitoring plugin for to not only measure/graph ‘current’ data but also ‘consumption since last check’ since being way more precise with low loads after being averaged over a time period.

          Major downside: you need a lot of patience since after adjusting settings it requires a long(er) time span for the new averaged consumption value to settle. But you get precise consumption data and not just fluctuating values you almost always choose the wrong ones to write down.

          Reply

  6. I was thinking the optimizations, first is to enable the rss queue flag to y on the r8125 driver source code, and also disable the ASPM, the last thing is to bind the irqs to the cpus

    And also it will be very nice to add the xt_FLOWOFFLOAD.

    I ordered this sbc on ali and cost 72usb, it’s a very nice sbc

    Reply

  7. Looks like there is a lot of work to do on performance for the R5S right now. I know it’s new but these results are not impressive. I’ll give it some time and if FriendlyWrt improves when OpenWrt 22.03 is officially released that might be the time to get one.

    Reply

    1. Thanks. That kinda kills the device for me though, way to much for a device that will idle most of the day.

      The numbers are *bare* power or with nvme?

      Reply

        1. I’ve also tried with OpenWrt and no SSD.
          That’s 4.6 Watts while idle, 6.0 Watts with iperf3.

          Reply

    1. They have not released the schematics so it’s unclear.

      The wiki reads “2 Pin 1.27/1.25mm RTC battery input connector for low power RTC IC HYM8563TS”. So I understand the 2-pin connector is just for the battery, and HYM8563TS is on-board already, but this would have to be confirmed.

      Reply

      1. I can confirm the RTC is there.

        It’s just the battery that is not included.

        Reply

        1. Thanks Jean! I’ve got quite a few uses for the NanoPi R5S if it comes with the RTC capability. But you’re right that the documentation on the RTC and RTC battery input is lacking.

          Reply

  10. I’m a bit surprised to see that the driver for the NIC is r8125 version 9.008, because that means that they use the out-of-tree driver by Realtek while Linux 5.10 should have native support for the 2.5 GBE NICs in mainline with the r8169 kernel module. I’m wondering if it would make any difference if you used the mainline driver for those NICs.

    I have used the r8125 driver on my desktop for quite some time and while I had no performance issues, it flooded my kernel logs with error messages that I could only suppress with some kernel command line parameters (btw. to give you an idea what I mean by “flooding” – within just 2 hours my kernel logs grew to almost 3GB!). Now that I upgraded my distro and use a newer kernel which includes 2.5 GBE support for my NIC by default in the r8169 driver, I don’t have this issue anymore and there’s no need to tweak the kernel command line just to work around this issue.

    Reply

  11. Sorry for asking what might be stupid questions;

    1) Can you access the USB 3.0-ports from Docker?

    So if I want to setup Home Assistant running on a Docker container and use 1* Z-Wave USB-stick and 1* Zigbee USB-stick can I expose both of them to the Home Assistant Docker container?
    I guess right?
    Mount a USB drive in a Docker container (sevenbridges.com)

    2) I was thinking of either using a “USB2LCD” to monitor the IP-address and temperature or / and a small touch screen to be able to view the actual output.

    However I guess since the Wiki mentions “Plug the USB2LCD module to the USB interface of NanoPi-R5S and power on[…]” that I need to sacrifice an USB 3.0-port and it does not connect to an internal header (and even if it did the chassi doesn’t seem to have a way to connect to them)?

    So guess the smaller touch screen / monitor would be a better option since I would need the USBs for Z-Wave / Zigbee.
    NanoPi R5S – FriendlyELEC WiKi

    Reply

  13. How do you use an m.2 SSD with openwrt? I’ve tried both SATA and NVMe and neither show in “Mount Points”.

    Reply

  14. Sorry FriendlyElec – not touching it.

    Its just kind of a waste since they actually never fix any serious firmware related issues or provide to mainline.

    Give this 2-4 months or even a year and it will be left to rot on bugged drivers and old OS snapshots at the point of release.

    They also never care to make their hardware stable at release so that both added together is an absolute no-go to ever touch their stuff again for anthing other then throwing your money away for some playing around.

    Imagine the Rasperry-Pi foundation doing that. Its really a shame because they could be SO much better if they would actually care for that … but no they dont.

    The only other ARM manufacturer of SBCs I trust somewhat is HARDKERNEL

    Reply

  15. Hello, I have a 2.5Gb wired connection between my RTL8125B chip 2.5Gb ethernet card and Nano Pi R5S on my computer. According to iperf3 results, I see 2.37Gb connection speed between the two in my tests. However, my disk connected via usb 3.0, which I have opened for sharing with samba4, which I have installed Openwrt, is between 70-80 megabytes for file copying and downloading. According to iperf3 test speeds, shouldn’t I reach at least 250 megabyte file copy speed per second? By the way, I am using the rk3568-eflasher-friendlywrt-22.03-docker-20221123.img.gz image. Could it be due to this image? Where should I check? Thanks.

    Reply

    1. > shouldn’t I reach at least 250 megabyte file copy speed per second?

      Not if the disk is the bottleneck or inappropriate settings like (missing) Samba tunables or (missing) IRQ/SMP affinity settings. SBC vendors usually don’t care about either (3rd parties like Armbian in the meantime too) so you’re on your own to figure out what’s going on with atop/htop (and /proc/interrupts) searching for CPU/IRQ bottlenecks and testing the disk in question (all HDDs use ‘Zone Bit Recording’ and become slower when filled with data since throughput on the inner tracks is lower than on the outer)

      Reply

      1. I did a file transfer test between nano pi r5s 2.5gb ethernet and windows 11 2.5gb ethernet on my computer, when I look with htop, the cpu comes to 70% 100% and openwrt ram suddenly shows 4gb ram 99% and cached ram 99% used. What exactly does this mean and what do I need to check in samba settings, it starts at 288 megabytes per second and goes down. 

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Boardcon Rockchip and Allwinner SoM and SBC products
Boardcon Rockchip and Allwinner SoM and SBC products