Create Minimal Debian Upstream Images with Debos and Armbian

20 Replies to “Create Minimal Debian Upstream Images with Debos and Armbian”

1. > Uncompressed images are still over 1GB and come with Armbian-specific tools, kernel and bootloader.

   To save a lot of space for the uncompressed image with Armbian simply choose btrfs instead of ext4 at image creation time. You can also save additional space by using Armbian’s customization features and skip most of if not all packages referenced in the $PACKAGE_LIST_ADDITIONAL variable.

   And by skipping ‘Armbian-specific tools’ you choose both lower performance and less reliability. I really wonder whether those people obsessed by small image sizes and ‘Debian upstream’ ever did performance comparisons or checked what Armbian does to increase life expectancy of the install media…

   Reply

   1. I consider armbian to be the Ubuntu of the embedded SoCs. Is the center of the distros universe for many reasons. Nevertheless, I like to see other alternatives from time to time, because the more the better in terms of choices for different use cases. Also people learn by doing those things. So I’m glad seeing new stuff and people try to do whatever they like to do.

      Reply

      1. > I consider armbian to be the Ubuntu of the embedded SoCs. Is the center of the distros universe for many reasons.

         I couldn’t agree less since for me Armbian is not even a distro 🙂

         It’s a flexible build system able to generate bootable Linux images from scratch in various flavors: 3 x Debian and 3 x Ubuntu. Majority of work deals with collecting and consolidating kernel patches (and as such Armbian is often months or even years ahead of ‘upstream Debian’ wrt hardware support for vast majority of interesting boards). The other important part is SBC specific tweaks — see below. Other than that Armbian is either pure Debian or pure Ubuntu.

         > Also people learn by doing those things

         Sure, learning is important. But I would try to focus on the important stuff and that’s:

         Most SD cards and especially those with small capacity suck when it’s about random IO. They were made for sequential transfers but perform poorly when a huge amount of data has to be written in a random fashion. Installing/updating packages is exactly this: almost pure random I/O. As such I fail to understand the desire to generate ‘minimal’ SBC images at least when it’s about general purpose distros like Debian/Ubuntu.

         Also important: ‘Write amplification’ when writing various small chunks of data on flash storage like SD cards is magnitudes higher compared to writing a whole image containing all the necessary stuff in a sequential fashion. Generating a ‘minimal’ image and afterwards installing the needed packages will not only take way longer but also increases wear out on the SD card drastically. The relevant learning here is: if you love your SD card (or flash storage in general) use a flexible build system and add all the packages to a reasonably large SD card image which then gets written sequentially in a single step.

         How to deal with low memory situations and horribly low performing storage? Avoid swap on SD card of course. After a lot of research on ZRAM this is default on all Armbian images now. Swap to compressed block devices in RAM performs awesome and causes no harm to SD card. The whole stuff is fully configurable in Armbian and while defaults are rather conservative even a massive ZRAM overcommitment can be configured and works beautifully (taks that need 3GB RAM can run on 1GB boards with Armbian with a slight performance decrease though). What’s there with ‘upstream Debian’? Swap on SD card or nothing.

         I’ve already mentioned ‘Write amplification’. A logfile where a few bytes get written to every few seconds will result in an awful lot of writes at the flash layer below (likely hundred times or even more). That’s what armbian-ramlog is for, keeping logs in RAM and writing them all in a batch to storage every hour and as such resulting in magnitudes less writes at the flash layer. This combined with other measures like increased commit interval for filesystem writes (keeping amount of ‘dirty pages’ high) increases life expectancy of SD cards dramatically.

         Same happens on Armbian desktop images where browser caches are located in RAM and ‘profile sync daemon’ is enabled for every local user resulting in browser ‘profiles’ being kept in RAM since otherwise ‘surfing the web’ will be both slow and also drastically increases wear on SD card (same for eMMC — we have several reports of worn out eMMC on Linux Desktop installs, and SD cards dying is more rule than exception with this use case)

         Speaking about performance: isn’t it great that with majority of kernels on ARM IRQ/SMP/HMP affinity is broken? If all interrupts are processed on cpu0 and CPU intensive tasks also sit there then with high system load this CPU core can become a nice bottleneck slowing everything down (especially true on big.LITTLE designs where cpu0 is almost always a little core). Distributing the IRQ processing on different CPU cores is one of the Armbian tweaks (unfortunately nobody there caring any more of such stuff in the meantime).

         And then part of ‘Armbian-specific tools’ is also something allowing to diagnose/report problems: armbianmonitor. IMO prerequisit in support situations to get a clue what’s going on on a remote SBC where someone struggles with something.

         IMO that’s the important learnings (not related to Armbian ‘as distro’ but just stuff Armbian partially focuses/focused on and implements). Debootstrapping Debian in yet another variant can also be fun but if the outcome is something performing a lot lower and sending SD cards to the bin more frequently… then you focused on the wrong goals IMO.

         SBC are different than ‘real PC’. Not addressing this is IMO just an excercise in ignorance. 🙂

         Reply

         1. You’re right! I’ve miss-used the word distro. Of course it’s a build system. I do the same mistake often for SBC and BSP.

            Since I’m building quite a lot of distros (I mean proper distros this time :p) for different architectures like ARM and x86, I’ve had to build distros that were a few MBs in size that the only thinks they did was just to run a single process in the initramfs, up to distros of 10ths of GBs that were running a huge application framework on top. This is why I’ve said there’s room for everything. Whatever the client needs. And sometimes they ask for weird stuff.

            Also things like RAM logs and storage buffers are pretty much the standard in most of the projects I’ve worked with (using eMMC though, not many projects had SD cards). Also the affinity or the rtirq for preempt-rt kernels is also something I’m using a lot since I’ve mostly worked on audio devices for the last year. I haven’t used ZRAM yet, because I didn’t had to, but I would like to have a project at some point to use it! But in my case these are custom distros for custom SBCs.

            It’s nice that Armbian has all these things! I wasn’t aware of those tools in the image, I’ll have a look to compare. Anyway, it’s pretty obvious that is made from people that are aware of the embedded domain and its needs and it’s not just a naive port from the desktop to the embedded. As I’ve said is my favorite to use when I want to test something on a SoC that is supported and I’m also using a lot of patches from there when I need to work with that SoC.

            Anyway, I like seeing new things whatever that is and I like when I have time to look at it and be able to keep even 1 thing from the whole thing.

            Reply

            1. > Armbian … made from people that are aware of the embedded domain

               Don’t think so. Armbian’s origins are ‘energy efficient and inexpensive server stuff’… on ARM since ~5 years ago something like a Cubieboard, Cubietruck, CuBox i4 or a BananaPi seemed like an ideal choice for small servers due to ‘native SATA’ and so on while x86 offers were way more expensive and wasted a lot more energy.

               Project matured, in early 2016 we started to support H3 boards (most probably the biggest mistake in the project’s history) but still the focus was on server, NAS, now also Linux as desktop and shortly after also this retro gaming crap. I believe ’embedded’ with its own constraints was never a consideration since why would an embedded project need an Ubuntu or Debian userland or an installation size of around 1 GB?

               The ‘collecting and consolidating kernel patches’ approach Armbian is a lot about provides a great benefit for embedded projects since the mainlining process on ARM takes an awful lot of time and once a certain hardware is fully supported by mainline kernel it’s likely already EoL. Here I’m with you that Armbian might provide some benefits for the embedded domain (providing a kernel with hardware support for a specific SoC/board 6 to 18 months prior to ‘being mainlined’) but other than that I don’t think there’s much focus on this. And the real heroes here aren’t the Armbian guys but those people contributing to mainline Linux kernel and providing their efforts in a usable way (thinking of Icenowy, jernej, Ondrej, wens — just to name few of the sunxi guys — and countless others)

               It’s dealing with the limitations. I’m a server guy for example and learned a lot the last few years dealing with these small beasts. One direct benefit is being able to cramp 3 times more VMs in a virtualization host compared to before 🙂

               And a side product is improving things on ARM and trying to make stuff like ZRAM a default on Linux boxes with small memory footprint (hey, even Windows 10 does something like this today!). Not sure where this will end though since currently nobody from ‘team Armbian’ takes care about such issues any more. The sole focus of the project is adding more and more hardware to the build system to ensure it’s impossible to provide proper support. When moving to newer kernel versions for example performance often decreases but nobody cares about.

            2. why h3 boards support was a mistake?

2. As tkaiser pointed out, if you want to get this “into the bushes” on making your own image, the Armbian build system supports doing just about anything you could hope for. And, skipping the Armbian utilities/bsp packages would be disastrous on several boards, and at the very least ill-advised on the rest.

   Reply

3. Hi, I am Gustavo Padovan, from Collabora. Thanks for trying this out, most of the time our people are using Debian to run debos (I usually do it from Fedora). This issue is new to us, can you confirm that your user has access to /dev/kvm? A ‘ls /dev/kvm’ will tell us that.
   I’m trying to understand where in the stack this bug is so I can work with the team here to fix this if it happens to be in debos itself. I’ll be happy if you can check that for us.

   Thansks,

   Gustavo

   Reply

   1. Hi,
      Here’s the output:

      jaufranc@cnx-laptop-4:~$ ls -l /dev/kvm crw-rw---- 1 root kvm 10, 232 มิ.ย. 23 09:42 /dev/kvm jaufranc@cnx-laptop-4:~$ groups jaufranc adm dialout cdrom sudo dip plugdev lpadmin sambashare vboxusers docker kvm libvirt

      1 2 3 4

      jaufranc@cnx-laptop-4:~$ ls -l /dev/kvm crw-rw---- 1 root kvm 10, 232 มิ.ย.  23 09:42 /dev/kvm jaufranc@cnx-laptop-4:~$ groups jaufranc adm dialout cdrom sudo dip plugdev lpadmin sambashare vboxusers docker kvm libvirt

      The user (jaufranc) is part of kvm group, so it should have read/write access.

      Reply

      1. The user inside the container will need access to /dev/kvm, so it needs to be part of kvm group.
         Could you try to add “–group-add $(getent group kvm | cut -d: -f3)” to docker command, i.e.

         docker run --rm --interactive --tty --device /dev/kvm --group-add $(getent group kvm | cut -d: -f3) --user $(id -u) --workdir /recipes --mount "type=bind,source=$(pwd),destination=/recipes" --security-opt label=disable godebos/debos orangepi0p2.yaml

         1	docker run --rm --interactive --tty --device /dev/kvm --group-add $(getent group kvm | cut -d: -f3) --user $(id -u) --workdir /recipes --mount "type=bind,source=$(pwd),destination=/recipes" --security-opt label=disable godebos/debos orangepi0p2.yaml

         Reply

         1. Thanks that did the trick for the permission issue, but there’s another problem:

            docker run --rm --interactive --tty --device /dev/kvm --group-add $(getent group kvm | cut -d: -f3) --user $(id -u) --workdir /recipes --mount "type=bind,source=$(pwd),destination=/recipes" --security-opt label=disable godebos/debos orangepi0p2.yaml qemu-system-x86_64: -virtfs local,mount_tag=virtfs-5,path=/recipes/scripts,security_model=none: cannot initialize fsdev 'virtfs-5': failed to open '/recipes/scripts': No such file or directory open /tmp/fakemachine-686249639/result: no such file or directory

            1 2 3

            docker run --rm --interactive --tty --device /dev/kvm --group-add $(getent group kvm | cut -d: -f3) --user $(id -u) --workdir /recipes --mount "type=bind,source=$(pwd),destination=/recipes" --security-opt label=disable godebos/debos orangepi0p2.yaml qemu-system-x86_64: -virtfs local,mount_tag=virtfs-5,path=/recipes/scripts,security_model=none: cannot initialize fsdev 'virtfs-5': failed to open '/recipes/scripts': No such file or directory open /tmp/fakemachine-686249639/result: no such file or directory

            Do I need to create this directory manually inside the docker image? Or maybe the build was interrupted, and restarting it causes this kind of problems?

            Reply

            1. Do you have scripts and overlays directories in the orangepi0p2.yaml directory?
               orangepi0p2.yaml relies on some scripts and overlays.
               You can retrieve them from https://gitlab.collabora.com/fdanis/debos-armbian2debian/tree/master/1-ArmbianKernel

            2. Oops. OK I missed that part. I through debos would just automatically retrieve anything it wanted. That works now. I’ll edit the post accordingly.

            3. It did not complete successfully.

               2019/06/28 09:30:43 ==== Update U-Boot menu ==== 2019/06/28 09:30:44 u-boot-update | P: Checking for EXTLINUX directory... found. 2019/06/28 09:30:44 u-boot-update | P: Writing config for vmlinuz-4.19.38-sunxi64... 2019/06/28 09:30:44 u-boot-update | P: Updating /boot/extlinux/extlinux.conf... 2019/06/28 09:30:44 ==== raw ==== 2019/06/28 09:30:44 Action <code>raw</code> failed at stage Run, error: Failed to read /scratch/mnt/usr/lib/linux-u-boot-next-orangepizeroplus2-h5_5.83_arm64/sunxi-spl.bin Powering off.

               1 2 3 4 5 6 7

               2019/06/28 09:30:43 ==== Update U-Boot menu ==== 2019/06/28 09:30:44 u-boot-update | P: Checking for EXTLINUX directory... found. 2019/06/28 09:30:44 u-boot-update | P: Writing config for vmlinuz-4.19.38-sunxi64... 2019/06/28 09:30:44 u-boot-update | P: Updating /boot/extlinux/extlinux.conf... 2019/06/28 09:30:44 ==== raw ==== 2019/06/28 09:30:44 Action <code>raw</code> failed at stage Run, error: Failed to read /scratch/mnt/usr/lib/linux-u-boot-next-orangepizeroplus2-h5_5.83_arm64/sunxi-spl.bin Powering off.

               Full output: https://pastebin.com/xq3ZNmvZ

            4. Armbian encode a version number in path to u-boot binaries, from your pastedbin the version currently installed is 5.85.
               So, you should replace 5.83 by 5.85 in raw actions, lines 75 and 80 of your recipe.

            5. Thank you. All good now!

4. Hi,

   I’m not sure how exactly debos works in Docker, but to get it going on your Ubuntu you need to create empty files at the locations it expects (i.e. kernel modules). Ubuntu has those modules compiled into the kernel while Debian ships them as modules, and it seems debos at the moment verifies the file is there before doing a modprobe call.

   Reply

   1. I have to clarify this only applies to running debos outside of the docker image and directly in the host Ubuntu system.

      Reply

5. Recently I took a different approach to accomplish more or less the same to get Ubuntu 18.04 minimal onto an Allwinner board. Instead of running a builder I just downloaded the ARM (netboot) installation initrd from Ubuntu and ran it using qemu emulating the full ARM processor itself. It is quite slow to run the installer this way but in the end you’ll have the same Ubuntu installation as you’d have running the installer on general i386/AMD64 architectures which I find quite favorable because the resulting base installation is exactly as I have on my PC.

   I used ‘https://gist.github.com/takeshixx/686a4b5e057deff7892913bf69bcb85a’ as a reference to get things up and running. Downloaded a kernel and the right initrd for the installer.

   When the installer was done I compiled u-boot for the Allwinner board and also compiled the most recent 4.19 mainline kernel without modules but with cherry-picked drivers that I needed. Put u-boot, the kernel and the installation on an SD card. It works really well.

   I think you can also run the original Debian installer inside ‘qemu-system-arm’ to get a base Debian installation for your Allwinner board,

   EDIT: just tried it. Downloaded ‘vmlinuz’ and ‘initrd.gz’ from http://ftp.nl.debian.org/debian/dists/stretch/main/installer-armhf/current/images/netboot/ and then ran (on my amd64 Ubuntu 18.04 host):

   dd if=/dev/zero of=disk.img bs=1M count=4096
   qemu-system-arm -kernel vmlinuz -initrd initrd.gz -append “root=/dev/ram” -no-reboot -nographic -m 1024 -M virt -hda disk.img

   You can go through the installer this way up to the point where it cannot find the virtio network card. The ubuntu installer did not have that problem.

   Reply

6. Unfortunately, The kernel and u-boot packages isn’t working, so I update the mirror and the debian suite in “/debos-armbian2debian/1-ArmbianKernel/overlays/armbian/sources.list.d” to

   deb http://mirrors.jevincanders.net/armbian/apt/ buster main buster-utils buster-desktop

   1	deb http://mirrors.jevincanders.net/armbian/apt/ buster main buster-utils buster-desktop

   And I update the architecture to armhf and the name packages to “packages: [ linux-image-current-sunxi, linux-dtb-current-sunxi, linux-u-boot-orangepipc-current ]
   ”

   To find the name packages I was search in mirror url and localize the packages names in dists directory.

   Finally, I update the raw action in yaml file to a new bin file:

   – action: raw
   origin: filesystem
   description: Raw do arquivo bin
   source: /usr/lib/linux-u-boot-current-orangepipc_22.05.3_armhf/u-boot-sunxi-with-spl.bin

   Reply

Leave a Reply

Connect with:

Your email address will not be published. Required fields are marked *

Comment *

Name *

Email *

Website