CNXSoft: Guest post by Blu about setting up arm64 toolchain on 64-bit Arm hardware running a 32-bit Arm (Armv7) rootfs.
Life is short and industry progress is never fast enough in areas we care about. That’s an observation most of us are familiar with. One would think that by now most aarch64 desktops would be running arm64 environments, with multi-arch support when needed.
Alas, as of late 2019, chromeOS on aarch64 is still shipping an aarch64 kernel and an armhf userspace. And despite the fine job by the good folks at chromebrew, an aarch64 chromeOS machine in dev mode an otherwise excellent road-warrior ride, is stuck with 32-bit armhf. Is that a problem, some may ask? Yes, it is aarch64 is the objectively better arm ISA outside of MCUs, from gen-purpose code to all kinds of ISA extensions, SIMD in particular. That shows in contemporary compiler support and in the difference in quality of codegen. Particularly with the recent introduction of ILP32 ABI for aarch64 (AKA arm64_32) and Thumb2 notwithstanding (unsupported under aarch64), there won’t be a good reason one would want to build aarch32 code on an aarch64 machine soon.
As a long-time arm chromebook user, I have been using a sandboxed arm64 compiler on my chromebooks. A recent incident (read: author’s sloppiness) destroyed that sandbox on my daily ride while I was away from home, so I had to do it again from scratch, over a precious mobile data plan. This has led me to document the minimal set of steps required to set up the sandbox, and I’m sharing those with you. Please note, that the steps would be valid for any setup of 3.7-or-newer aarch64 kernel & armhf userspace, but in the case of chromeOS some extra constraints apply the root fs is sternly immutable to the user.
Under chromeOS dev-mode a writable fs space is available under /usr/local, so this is where we’ll set up our sandbox.
|
1 2 3 |
$ cd /usr/local $ mkdir root64 $ cd root64 |
We need to choose a good source of arm64 packages some (non-rolling) distro with up-to-date package base will do. That casually means Debian or Ubuntu we go with the latter due to its better refresh rate of packages. Next we need to decide which version of the compiler we want, as that will determine what system libraries we’ll need. For personal reasons I’ve decided upon gcc 8.2.0. Here’s the list of packages one needs to download from http://ports.ubuntu.com/ubuntu-ports/pool/main/ for the gcc version of choice:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
-rw-r--r--. 1 chronos chronos 2180000 Apr 9 2018 binutils-aarch64-linux-gnu_2.30-15ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 2391032 Apr 9 2018 binutils-dev_2.30-15ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 5711228 Sep 18 2018 cpp-8_8.2.0-7ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 6465184 Sep 18 2018 gcc-8_8.2.0-7ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 428092 Apr 9 2018 libbinutils_2.30-15ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 2270660 Apr 17 2018 libc6_2.27-3ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 2051800 Apr 17 2018 libc6-dev_2.27-3ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 474272 Apr 17 2018 libc-bin_2.27-3ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 36228 Sep 18 2018 libcc1-0_8.2.0-7ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 58632 Apr 17 2018 libc-dev-bin_2.27-3ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 34292 Sep 18 2018 libgcc1_8.2.0-7ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 856036 Sep 18 2018 libgcc-8-dev_8.2.0-7ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 212856 Dec 2 2018 libgmp10_6.1.2+dfsg-4_arm64.deb -rw-r--r--. 1 chronos chronos 520464 Aug 14 2018 libisl19_0.20-2_arm64.deb -rw-r--r--. 1 chronos chronos 37156 Jan 24 2018 libmpc3_1.1.0-1_arm64.deb -rw-r--r--. 1 chronos chronos 202380 Feb 8 2018 libmpfr6_4.0.1-1_arm64.deb -rw-r--r--. 1 chronos chronos 371988 Sep 18 2018 libstdc++6_8.2.0-7ubuntu1_arm64.deb -rw-r--r--. 1 chronos chronos 45430 Sep 13 2015 zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb |
In case of difficulties locating those packages on the main Ubuntu repo, one can use the global registry http://ports.ubuntu.com/ubuntu-ports/ls-lR.gz.
Next we ‘deploy’ those packages. We can do it one by one, e.g. if we’re curious to check what’s in each package:
|
1 2 3 |
$ ar x package.deb $ tar xf data.tar.xz $ rm -v control.tar.{xz,gz} data.tar.xz debian-binary |
Or en masse:
|
1 |
$ for i in *.deb; do ar x $i ; tar xf data.tar.xz ; rm -v control.tar.{xz,gz} data.tar.xz debian-binary ; done |
That was the straight-forward part. The next part is less so.
An aarch64 kernel can natively run statically-linked binaries no problem. Such binaries are a rarity on desktop Linux, though most of the binaries you’ll encounter or compile yourself will be dynamically-linked. For each architecture supported by the platform, there must be a corresponding dynamic loader. On arm multi-arch nowadays those would be:
|
1 2 |
/lib/ld-linux-aarch64.so.1 /lib/ld-linux-armhf.so.3 |
For aarch64 and armhf, respectively; on non-multi-arch there would be only one, naturally. If you inquire about any given dynamically-linked binary, you’d see those loaders reported immediately after the ELF type:
|
1 2 3 4 5 |
$ file a.out # some freshly built armhf dynamically-loaded binary a.out: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, stripped $ file a.out # same code built for aarch64 a.out: ELF 64-bit LSB pie executable ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=e999425cae14ba4fa0ad8bf44ad1c50bb32e012e, stripped |
The part that reads “dynamically linked, interpreter /lib/ld-linux-armhf.so.3” tells what dynamic loader should be used to launch each binary, with path and all. As we cannot modify /lib on chromeOS, we cannot place an aarch64 loader there, so we will need to revert to other means.
As many are familiar, the binary of gcc is just a facade that invokes a bunch of actual business binaries to carry out a build. In the context of our sandbox, those are:
|
1 2 3 4 |
/usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/cc1 /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/collect2 /usr/local/root64/usr/bin/as /usr/local/root64/usr/bin/ld |
Now, the first two were delivered through our .deb packages, while the latter two are normally mere symlinks to the actual aarch64-linux-gnu-* binaries, but we don’t have those symlinks yet, and we don’t need them as symlinks anyway.
Each time our arm64 gcc will try to execute one of those, it will fail due to lack of system-wide loader for aarch64. So how do we solve that? Easily we manually specify the loader that should be used with any given arm64 binary we want to run. So let’s shadow those binaries by some scripts of the same names to do our bidding, while preserving the original binaries for cc1 and collect2 as .bak files at their original locations:
|
1 2 3 |
$ cd usr/lib/gcc/aarch64-linux-gnu/8 $ mv cc1 cc1.bak $ mv collect2 collect2.bak |
Eventually we want the following scripts at their corresponding locations:
- /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/cc1:
12#!/bin/bash/usr/local/root64/lib/ld-linux-aarch64.so.1 /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/cc1.bak $@
- /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/collect2:
12#!/bin/bash/usr/local/root64/lib/ld-linux-aarch64.so.1 /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/collect2.bak $@
- /usr/local/root64/usr/bin/as:
12#!/bin/bash/usr/local/root64/lib/ld-linux-aarch64.so.1 /usr/local/root64/usr/bin/aarch64-linux-gnu-as $@
- /usr/local/root64/usr/bin/ld:
12#!/bin/bash/usr/local/root64/lib/ld-linux-aarch64.so.1 /usr/local/root64/usr/bin/aarch64-linux-gnu-ld $@
Last but not least, we need a gcc top-level script to set up some things for us (placed somewhere in PATH, ideally):
- gcc-8.sh:
12345#!/bin/bashroot=/usr/local/root64export PATH=$root/usr/bin/:$PATHexport LD_LIBRARY_PATH=$root/lib/aarch64-linux-gnu/:$root/usr/lib/aarch64-linux-gnu/$root/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1 $root/usr/bin/gcc-8 $@ --sysroot=$root
Let’s test out contraption with the following sample code:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
#include <stdio.h> #if __LP64__ #define PLATFORM_STR "arm64" #else #define PLATFORM_STR "armhf" #endif int main(int argc, char** argv) { fprintf(stdout, "hello from " PLATFORM_STR "\n"); return 0; } |
First, with the native armhf gcc-8.2.0 compiler, courtesy of chromebrew:
|
1 2 3 4 |
chronos@localhost /usr/local/root64/misc $ gcc-8.2 test.c chronos@localhost /usr/local/root64/misc $ ./a.out hello from armhf chronos@localhost /usr/local/root64/misc $ |
Next with the arm64 gcc-8.2.0 we just set up:
|
1 2 3 4 |
chronos@localhost /usr/local/root64/misc $ ./gcc-8.sh test.c chronos@localhost /usr/local/root64/misc $ LD_LIBRARY_PATH=/usr/local/root64/lib/aarch64-linux-gnu/:/usr/local/root64/usr/lib/aarch64-linux-gnu/ ../lib/ld-linux-aarch64.so.1 ./a.out hello from arm64 chronos@localhost /usr/local/root64/misc $ |
Please note, that since we have not exported an arm64-aware LD_LIBRARY_PATH yet, we have to specify that on the command line, right before invoking the aarch64 loader.
Well, that’s all, folks. I hope this comes of use to some aarch64+armhf users out there.
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress
Seems like a perfect example of why you don’t get yourself locked into an ecosystem where you can’t just install your own kernel and userland.
That’s entirely a matter of priorities. I don’t want to ‘install my own kernel and userland’ — I’d rather leave that to distro providers, like for instance google, who are quite good at what they do. The majority of chromeos dev-mode users are fine with chromeos and rich 3rd party repos like chromebrew. This is an article for those who want aarch64 on top of that, demonstrating it’s trivial to get aarch64 apps and full toolchains going on chromeos.
> rich 3rd party repos like chromebrew.
With a whole 1026 packages apparently.
>demonstrating it’s trivial to get aarch64 apps and full toolchains going on chromeos.
Doesn’t look trivial. i.e. in the time you’ve taken to hack your ubuntu toolchain into chromeos I could have installed all of the pre-built cross toolchains in Debian and have GCC9 for a bunch of targets along with qemu to run the binaries locally if I needed to.
This seems like “making a-rod for your own back”.
> With a whole 1026 packages apparently. *Apparently* chromebrew is not trying to be a debian repo replicant — it is driven by the needs of a handful of devs (as in chromeos *dev-mode*). So it’s only the packages that those chromeos users want that find their way into the repo. For the needs of *those* devs, it’s quite a nice repo; from my experience with the repo, it’s provided the vast majority of the packages I’ve needed so far. > Doesn’t look trivial. i.e. in the time you’ve taken to hack your ubuntu toolchain into chromeos I could have… Read more »
>*Apparently* chromebrew is not trying to be a debian repo replicant You said it was *rich* not me. Only having 1026 packages probably means coverage of packages there is pretty shallow.. and a quick scan through the list of packages (https://github.com/skycocker/chromebrew/tree/master/packages) doesn’t seem to suggest any richness either. A lot of it seems to be toy stuff like quakespasm. >the approach is quite trivial, yes. IMHO writing an article that’s multiple paragraphs and includes images to explain how to do something contradicts with the claim that what is described in the article is trivial. Maybe you mean “straightforward”. >It’s a… Read more »
> You said it was *rich* not me. Yes, and I’ll repeat it — it’s rich for my purposes and those of devs like me. Clearly not rich for people who deal with absolutes. > Maybe you mean “straightforward”. No, dgp, I mean trivial, as in ordinary, as in nothing extraordinary to get an arm64 package running on an armv7 userspace. If you need to translate that to ‘straightforward’ so it could fit your worldviews — be my guest, I’m not stopping you. > I hate to go all meta on you but.. And here I thought you were nothing… Read more »
>it’s rich for my purposes and those of devs like me. Yet you have to write a wordy description of how to get basic tools on your system. I’m surprised your boss hasn’t spotted you constantly messing with that crap, forced the company credit card into your hand and told you to just buy a macbook already. For devs like me.. just my AndroidStudio instance would make your chromebook burst into flames somewhere between hitting the build button and the projected week it would take to build. >No, dgp, I mean trivial, as in ordinary, That’s not what trivial means… Read more »
I appreciate linguistic bikeshedding and political drivel, but that about concludes our totally-not-meta discussion. If you figure out something meaningful to say on the subject, I’ll participate. Until then, enjoy your day, dgp.
Usually I close the tab and forget about it because this is always what you do and it’s easy to tell you’re getting to this point because you try spinning your “I’m the only *dev* in the world” (No true Scotsman) elitist nonsense, calling people kids and so on… but I just found this tab open after looking for another so lets have some more fun. >I appreciate linguistic bikeshedding Words mean things. If I tell my boss a change is trivial that does not mean that it will take a whole day to implement and require the full test… Read more »
dgp, there’s just one individual on this site I’ve called ‘a kid’.. And here’s but an example of why: > Even if trivial was redefined to ordinary that still isn’t ordinary. You have to manually put your machine into dev mode and then manually extract packages and faff around to get the bastardised packages to run. Sure, let’s “redefine” trivial: https://www.merriam-webster.com/dictionary/trivial trivial, adjective 2 : commonplace, ordinary > manually put your machine into dev mode Another trivial step, done once and forgotten about by chromeos dev-mode users.. > manually extract packages and faff around to get the bastardised packages to… Read more »
Hey.
https://github.com/ryanwoodsmall/crosware
For context: “Tools, things, stuff, miscellaneous, etc., for Chrome OS / Chromium OS “
Nice job! I was not aware of it — I’ll check it out ASAP.
As a question why have Android OS boxes stayed 32 bit software while the SoC are 64 bit ?
Most phone vendors don’t want to deal with ISA transitions, and aarch32 and aarch64 are different ISAs. Also, due to (1) larger pointers in 64-bit code and (2) absence of the compressed arm ISA known as Thumb on aarch64, most arm64 binaries tend to be larger. Larger binaries, larger distros, less space for photos, mp3s and apps. Lastly, some pointer-chasing code can be slower in 64-bit again thanks to larger data records, but that is addressed by arm64_32. The natural answer to all those points above under linux is called multi-arch support — on arm that’d be aarch64 kernel and… Read more »
Is there an easy way figuring out if my android phone has an armhf userspace?
Not an android user here, but chances are that if you have a recent android and armv8 hw, userspace is already effectively multi-arch (multi-lib in android terms). Get an android app with a known 64-bit version, i.e. Geekbench, and see what it says on your phone.
Thanks blu
Well honestly I don’t care enough to go that far finding out 😉
Just out of curiosity what are you using if not android?
I was a sworn s30/40 user for many years, then firefoxos until it went under, at which point I was forced into ios. I’ve written (system-level) code for android, but I’ve never owned an android phone : )
On the subject of android, checking the timelines again, it turns out android got 64-bit multi-lib earlier than I remembered — since lolipop.
I also had the impression it was 64 bit from a long time but got curious.
I never was a nokia fan couldn’t wait till android is coming. It never happened, till i lost interest…
Lol then Samsung came now im with android as i cant use ios (due to allergy 😛 ) its for me the least bad of the horrifying options. No more with Samsung from along time though
>As a question why have Android OS boxes stayed 32 bit software while the SoC are 64 bit ? TBH I doubt it has anything to do with size and was just a timing issue. Almost all apps that ship with native binaries only shipped with armv7 and x86 and they didn’t want to make that work (i.e. ship multiple versions of all of the android system libraries, make the interactions work correctly, get AndroidStudio etc compatible) at that time and it was much easier just to continue with armv7 for the userland. Now that no one makes new armv7… Read more »
> Almost all apps that ship with native binaries only shipped with armv7 and x86
Many native Android apps lacked x86 support and that’s why Intel had to write an emulator to run ARM code on their pitiful attempt at making x86-based SoC for phones.
>Many native Android apps lacked x86 support That might have been the case back at android 2 but the x86 emulator has been the main development environment for a lot of people for a long time now so if you don’t have x86 libraries you’ll be annoying almost every android developer for no good reason. I don’t even think you could have a 100% on the metal native app back then. A game I worked on a good 8 or more years ago shipped with arm and x86 libraries. >and that’s why Intel had to write an emulator to run… Read more »