NanoPi R5S preview – Part 2: Ubuntu 20.04 (FriendlyCore)

I started the NanoPi R5S review with an unboxing, a teardown, a quick try of the pre-installed OpenWrt-based FriendlyWrt, and some iperf3 benchmarks on the 2.5GbE interfaces that were rather disappointing. I test further I switched to the Ubuntu 20.04-based FriendlyCore image since I’m more familiar with Debian-based operating systems, and some tools will not run on OpenWrt. Note the performance is still not quite optimal, and that’s why I call this a preview since numbers should improve in the next few months as more people tweak the software.

OpenWrt optimizations?

But before jumping to Ubuntu, I gave an updated version of FriendlyWrt a try as FriendElec told me they had added some optimizations:

We have made some optimizations on the new image, such as NIC interrupt settings, and offload support…

So I downloaded “rk3568-eflasher-friendlywrt-20220526.img.gz” found on Google Drive, flashed it to a microSD card with USBImager, and booted it to the router.

It will automatically flash the image to the eMMC flash, If you connect a monitor you can follow the result. Once it’s done, remove the microSD card, and power cycle the router.

You can check the status by connecting an HDMI monitor (as shown above), or checking out the LEDs on the device. It’s very fast and the installation to the eMMC flash takes only a few seconds.

The main changes were made to the 40-net-smp-affinity file. In the pre-installed  FriendlyWrt, it looks like that:

While the new 40-net-smp-affinity file is indeed different:

Willy Tarreau explains the changes made for the eth1 interface:

It involves RPS …. i.e. they receive this IRQ on core 2 and redistribute the incoming traffic to cores 0,1,3. That’s the right way to use RPS. However you have to manually assign iperf and watch the first core that saturates. If it’s saturating core 2 with ksoftirqd first, make sure that iperf runs on any of the other 3. If core2 is slightly idle, try to put iperf on it. If putting it on it makes ksoftirqd pop up, then they’re hindering each other, and you’d rather change the RPS setting to free another core and use it for iperf.

I did try this method before testing and switching to Ubuntu, and my results were even worse with the new FriendlyWrt image:

So this would have to be revisited.

M.2 NVMe SSD installation in NanoPi R5S

I’ve just purchased an APACER AS2280 (AP256GAS2280P4-1) PCIe Gen 3.0 x4 SSD that can achieve up to 1,800 MB/s sequential read and up to 1,100 MB/s sequential write speeds on the right hardware.

The installation is straightforward as I just had to loosen four screws to remove the bottom cover, install the SSD, and fasten it with the provided screw.

Installing Ubuntu 20.04 FriendlyCore on NanoPi R5S

I first attempted to install FriendlyCore using the eflasher image.

It looked good, so I restarted the router, but then I noticed the WAN interface link would not show on the TP-Link switch, and only the power LED was turned on (It happens the latter is normal for the FriendlyCore/Ubuntu image). I tried again, going into eflasher UI settings by clicking on Finish, but still no luck.

So instead I downloaded the “SD” image to boot directly from the microSD card and run the OS from there. It works fine. If you intend to use NanoPi R5S for multiple purposes and expected a desktop environment in the Ubuntu 20.04 image, you’ll be disappointed, as the HDMI output is currently only used to access the terminal.

FriendlyCore system information

You’ll find the boot log on CNX Software Pastebin. I logged in with SSH using pi/pi credentials (username/password), and upgraded the system to the latest packages with:

Let’s run some commands to get system information:

It looks all good, except the NVMe drive has not been mounted automatically. Let’s find more details with inxi:

Only eth0 WAN port is up, and eth1/eth2 2.5GbE ports are down, and not configured at all. FriendlyElec appears to mostly focus on the FriendlyWrt image, and they told me optimizations were not implemented on FriendlyCore yet, so most people should probably use FriendlyWrt instead since it will be easier to configure the network and router settings. I can see the Apacer AS2280P4 SSD is detected, and it turns out it’s not formatted out of the box, so I just formatted it with mkfs.ext4.

Benchmarking NanoPi R5S

Let’s run SBC Bench on the router to benchmark the CPU and potentially find some issues:

I launched this almost immediately after boot, so the dmesg output should be there in full (see boot load earlier in this preview/review), but the script is missing some information from it. The full output from sbc-bench.sh script can be found on pastebin, and we can notably see the “1992” MHz advertised frequency is tested to be 1845 MHz in reality, so some optimization may be possible here.

7zip is still faster than on NanoPi R2S router (3871), or about a 23% boost in performance, while AES-256-CBC 16KB is about 22% faster (704,872.45 vs 861,334.19kH/s)

NVMe benchmarking

I tested the NVMe SSD three times with iozone 3 three times, one with 100MB file:

then a 500MB file:

and finally a 1GB file:

The results are more or less consistent across all three tests without massive variations, and in last we’ve got about 380MB/s for read and write, well below the SSD advertised write/read speeds, and results for ODROID-M1, but that’s because of the PCIe 2.0 x1 interface used in this design, instead of the PCIe Gen 3.0 x2 interface used in the Hardkernel board.

Here’s lspci output for reference:

2.5GbE interfaces configuration and benchmarking

Since only the eth0 Gigabit Ethernet “WAN” interface is configured out of the box, we have to configure the two 2.5GbE ports manually. I used the same testbed as in the first part of the review with FriendlyWrt, namely a Ubuntu 20.04 laptop with a Realtek RTL8156BG USB 3.0 to 2.5GbE dongle connected to eth1, and UP Xtreme i11 mini PC connected to eth2. Instead of using a bridge interface like in FriendlyWrt, I’ll configure two different subnets: 192.168.2.0 for eth1, and 192.168.3.0 for eth2.

Let’s create two new files in /etc/network/interfaces.d/:

• eth1

• eth2

Restart the network as follows:

Now install a DHCP server

edit /etc/dhcp/dhcpd.conf file with our two subnets:

… before restarting the dhcp server:

At this point the laptop and mini PC should get their IP address from NanoPi R5S on the respective subnets. We can start benchmarking the interfaces:

iperf3 download (Rx from R5S point-of-view) using eth1 connected to the laptop:

That’s a bit slower than what I got (1.85 Gbps) in OpenWrt, and there are retransmissions. I also monitored the system with sbc-bench.sh during the transfer:

The system performs at its maximum advertised frequency during the test, and I don’t any obvious bottleneck shere.

We can also check some information and stats with ethtool:

We did get some rx_mac_missed.

Let’s do that in reverse (Tx):

It looks quite better than in OpenWrt (1.12 Gbps).

IRQ percentages are much lower but I suppose that’s normal for Tx.

Let’s switch to eth2 connected to UP Xtreme i11:

Oh great! It’s the first time I get a decent 2.35 Gbps transfer, so there’s hope!

Unless I’m mistaken the 25% of IRQ should mean a core is fully utilized handling those.

Let’s try Tx:

1.59 Gbps. Not quite perfect, but still better than in OpenWrt.

Again the CPU operates are full speed, and far from 100% utilization so the bottlenecks must be somewhere else.  We can again check eth2 info and stats with ethtool.

We’ve got more rx_mac_missed here. So there will be some tweaks to improve the performance, but based on my experience with RTL8156B adjusting settings is really tricky, and experienced people don’t seem to agree on what to adjust, I’m talking about Realtek engineers working on RTL8156/8125 drivers vs regular readers here that are experts in networking.

Configuring NAT between the two 2.5GbE interfaces

Since the 2.5GbE interfaces don’t work optimally with iperf3, I did not bother testing the router performance in FriendlyWrt, but several people still asked. So I’ll show how I configured NAT in Ubuntu 20.04, and still test NAT performance bearing in mind it will certainly have improved in a few weeks or months.

We’ll need to enable IP forwarding and NAT. I used instructions adapted from a post on networkreverse.

Edit /etc/sysctl.conf to enable IP forwarding (uncomment the following line):

Apply the changes:

Now let’s enable NAT:

We can now ping UP Xtreme i11 on 192.168.3.0 subnet from my laptop on 192.168.2.0 subnet:

If you want to make the changes permanent:

Let’s try iperf3 between UP Xtreme i11 and my laptop with the data routed through the NanoPi R5S router.

768 Mbps in one direction and 937 Mbps in the other.

Monitoring with sbc-bench.sh show the processor runs at 1992 MHz (or 1845 MHz real), and again the 25% IRQ should mean one core is fully utilized to handle IRQs.

The mpstat command shows this should handled by core #0

And this can be confirmed by using top and htop.

NanoPi R5S power consumption

I’ve just got a wall power meter to check power consumption:

• Idle – 5.1 Watts
• Iperf3 to eth1 – 6.3 to 6.7 Watts
• NAT test between laptop and mini PC – 6.2W

The numbers above are with Ubuntu 20.04 and the NVMe SSD installed.

I’ve also tested on NanoPi R5S with OpenWrt and no SSD:

• Idle – 4.6 Watts
• Iperf3  – 6.0 to 6.2 Watts

Note that since I’m using a wall power meter any efficiency loss in the power adapter (Khadas VIM4 USB-C power adapter) will be included and the values may be higher than with a USB-C power meter. It might also be possible to optimize power consumption with settings.

Final words

I’ll stop here for today. Optimizations should include changing the firmware to get Rockchip cores to run at 1992 MHz, and adjusting various settings related to PCIe and Ethernet settings, most of which I’m not familiar with (yet).

I’d like to thank FriendlyElec for sending review samples of NanoPi R5S mini router. The router with metal enclosure can be purchased on FriendlyElec website for $75, or you can get the board only for $59. The router can also be found on Aliexpress from several sellers, some of which also sell a 4GB RAM version, which is odd since FriendlyElec only sells the model with 2GB RAM at this time.

 

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress