Radxa Fogwise Airbox AI box review – Part 2: Llama3, Stable Diffusion, imgSearch, Python SDK, YOLOv8

After checking out Radxa Fogwise Airbox hardware in the first part of the review last month, I’ve now had time to test the SOPHGO SG2300x-powered AI box with an Ubuntu 20.04 Server image preloaded with CasaOS as well as Stable Diffusion and Llama3 containers.

I’ll start the second part of the review by checking out the pre-installed Stable Diffusion text-to-image generator and Llama3 AI chatbot, then manually install imgSearch AI-powered image search engine in CasaOS web dashboard, test the Python SDK in the command line, and run some AI vision models, namely Resnet50 and YOLOv8.

Radxa Fogwise Airbox OS installation

Radxa only provided an Ubuntu Server 20.04 image last month with only the basics pre-installated. The company has now improved the documentation and also made two images available for the Radxa Fogwise Airbox:

• Base image (1.2GB) – Based on Ubuntu Server 20.04; contains only Sophon base SDK and backend.
• Full image (9.5GB) – Same as above, but adding the Radxa LLM frontend, CasaOS, and demos for common LLMs.

Beginners should go with the second even if it takes longer to download, as it will make everything much easier to test out of the box. So that is the image I went with (Radxa Airbox B5 0606), and I used USBImager to flash it in a 32GB microSD card, before inserting the microSD card into the Fogwise Airbox and monitoring the installation with the serial console as I did in the first part of the review.

After installation is complete, we can see airbox.local’s TCP port 81 is open and we can access the CasaOS dashboard using radxa and radxa as login credentials.

Using Radxa Fogwise Airbox with CasaOS

It will show some system information (I also connected a USB hard drive), but the most important is that Stable Diffusion and Lllama3 are already installed. You may consider changing CasaOS’s username and password, and accessing the Linux terminal through SSH with username (linaro) and password (linaro) to change the password there as well…

Here is some extra information about the system from inxi.

Only 2.99GB RAM is available to the system, although I have a machine with 16GB of RAM. That’s because the RAM is shared between the NPU (Neural Processing Unit), VPU (Video Processing Unit), and VPP (Graphics Acceleration Unit). We can check the settings with the memory_edit utility as follows:

That means we are left with 4096 MB for the system with this device tree file. If needed, it can be changed with the script as follows:

I have not changed it before the current memory configuration works fine for the AI models used.

Stable diffusion test

Let’s now click on the Stable Diffusion icon in CasaOS dashboard to start and open it. The first time I tried it looked like it would not work and Firefox was unable to connect. That’s simply because it takes time for the container to start, so you’ll want to wait a minute or two before trying again.

The web interface for Stable Diffusion is built with Gradio which explains why the window is called “Gradio”. The first step is to select the Model and Controlnet and click on the Load Model button although there’s not much of a choice here because each dropdown menu only has a single choice. It will take a little while to load the model (around 1 minute), and then we can try the text-to-image generator. I started with one of the examples provided at the bottom with both a prompt:

1girl, ponytail ,white hair, purple eyes, medium breasts, collarbone, flowers and petals, landscape, background, rose, abstract

and a negative prompt:

ugly, poor details, bad anatomy

plus various parameters including denoising strength and CFG (classifier-free guidance scale) scale.

It took about 7 seconds to create the image. I then tried my own text prompt creating an image with a penguin surfing at a beach with some palm trees.

It also took 5 to 7 seconds to create the image. Adding an input image may be quite fun and it allows us to better understand the denoising strength and CFG scale parameters. All processing is done locally, you don’t have to worry about sharing personal photos online. I used a stock photo with older people as a test.

I set the denoising strength to the minimum to keep most of the original image, and the CFG scale to 0 to give it some flexibility. The resulting image is very close to the original.

If I boost the denoising strength and play around with the CFG scale to get an acceptable result… Stable Diffusion still takes the input as a guide, but it has much more flexibility when creating a photo.

I’ve noticed that when human subjects are small in a photo the results don’t look that good. Let’s try another photo.

That’s better. Adjusting the denoising strength will create images further and further from the original. The GFC scale can create “monsters” with funny fingers and physical attributes. All tests I did completed in 5 to 10 seconds.

Llama3 on Fogwise Airbox

Time to shut down the Stable Diffusion container, and start Llama3. We’ve presented with a text prompt and a submit button at the bottom of the page. So I asked whether Llama3 knew anything about CNX Software.

Most of it gives a pretty good summary of what CNX Software does, although the starting year is wrong. But I was told (on X), that I should not expect Llama3 to spew accurate information. I guess it’s some art project then 🙂

We don’t have performance metrics in the web interface, so I shot a screencast in Ubuntu to show the rendering speeds.

I then asked Llama3 to translate a relatively short text into English but this stopped in the middle of the answer with a warning reading “reach the maximal length, Llama3 would clear all history record”.

So stopped the container, changed the memory limits from 256 to 2048, and restarted Llama3.

But I got the same issue. Radxa told me it’s possible to change that:

Our Llama3 is fixed length input due with TPU design, for now is 512 length, if the total input + output > 512 the model would clear all of history information, if you want to increase the length of model, you can compile it to 1024 or more follow this link: https://github.com/sophgo/LLM-TPU/tree/main/models/Llama3 (chinese), but it would cost more inference time, or you can set –enable_history to False to ignore the history

I don’t think I can do that in CasaOS, but I’ll try again later in the command line.

Installing and running imgSearch in CasaOS

We’ve only used some preinstalled apps so far. But we can install extra apps manually including “Radxa whisper”, “Radxa ImageSearch”, and “Radxa chatdoc”. I’ll go with the imgSearch image search implementation

The first step is to click on the + icon and select “Install a customized app”

Then we need to add the parameters for the docker image:

• Docker Image – radxazifeng278/radxa_imgsearch_app:0.1.0
• Title – Image_Search
• Web UI port – 9007 (you can select any unused TCP port)
• Host Port – 9007
• Container Port – 8501
• Host Devices – /dev
• Container Devices – /dev
• CPU Shares – Medium

Now click on the “Install” button to start the installation process which only takes several seconds.

We now have a new app called “Image_Search”.

We can click on the app to start it, however the first time I tried, it would get stuck forever in a loop showing “Running”

If I check the log in Ubuntu 20.04 we can see a few out-of-memory errors:

We can also check the log in CasaOS and the program is continuously being killed and restarted.

The trick is to change the memory limit in the app settings. I changed it to 2048.

We can click on Save which will reinstall the docker container with the new parameters, and this time around it can run:

My first idea was to select a directory on the hard drive attached to the Radxa Fogwise Airbox AI Box, but it’s not implemented that way, as instead, we need to manually upload a list of files. I was told the Steamlit Python framework used for this demo does not have a widget to load files from a directory. Nevertheless, I created a “Test 1” gallery with around 3 photos to get started. It could process the photos within a few seconds.

However, I was unable to run the test due to an error that reads “could not open ./results/EN/Test 1/index.faiss for reading: No such file or directory“.

If I go to the results/EN directory in the Terminal for the container, there’s a “Test 1” folder, but it’s empty.

Then I thought maybe, it’s not liking the space in the gallery name. So I changed that to “test2” and it worked after uploading a few recent pictures.

Then I created a “CNXSoftware” gallery trying to add all images from 2023 (about 4,000 of those), but it was stuck and nothing seemed to happen. So I just uploaded a few hundred from the directory, and I was able to search for “block diagram” from the list of photos and got relevant results.

Checking out the Python Virtual environment on Fogwise Airbox

Time to shut down all containers running in CasaOS, and try the Python Virtual environment that should give the user more flexibility. I’ll be following the instructions for Llama3 since I have some unfinished business.

At first, I did this in the root partition (/), but I quickly ran out of space.

So I switched to the 25GB “data” partition instead. Adding an M.2 2230 NVMe SSD to the Fogwise Airbox might be a good idea since AI models are large, and you may not want to delete or move the files around all of the time…

Let’s get the Llama3 Python demo:

The Llama3 8B model can be downloaded as follows:

Let’s now setup the environment and install dependencies:

We can now start the Llama3 demo in the terminal (Note: it will take a while before we get to the prompt, so be patient):

Let’s ask the same question as before:

The AI box generates 9.566 token/s.  if you want to reproduce the Gradio demo as in CasaOS, we can start the web_demo.py Python script instead:

All good.  I still have a 512-byte limit with either method:

How to increase Llama3 answer limit

Let’s see if we can increase the limit to 1024 and to what extent it impacts performance. Note that I first started those instructions on my Ubuntu 22.04 laptop with an Intel Core i5-13500H, 16GB RAM, and about 26GB of free space. But once I read the instructions at the end where we are told that:

Generating the bmodel takes about 2 hours or more. It is recommended to have 64G memory and over 200G of disk space, otherwise OOM or no space left errors are likely.

It’s not going to work. If only I had a machine with 64GB RAM. Oh! wait… I do!  So I installed Ubuntu 24.04 on Cincoze DS-1402 embedded computer with an Intel Core i9-12900E processor, 64GB DDR5, and a 300GB Ubuntu partition. That should do. All instructions below are done in the x86 host unless otherwise stated.

We’ll first need to install the compiler:

We’ll now need to ask permission to download the Llama3 model by filling out the form at https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/tree/main. Note that it requires a manual permission. I asked on Saturday, and I could download it on Sunday. We need to install git-lfs (Large File Storage) first:

Then we have to generate a token to download the model and make sure to select “Read access to contents of all public gated repos you can access”. Let’s run the following command:

and input your username and token to retrieve the code. I did that in the root directory for the current user.

The first time I tried with Python 3.12 preinstalled in Ubuntu 24.04, but the solution requires Torch-1.12.0 not available in Python 3.12. So I had to install Python 3.10 with miniconda3:

After restarting the system will see (base) added to the prompt and Python 3.10 version is used:

We can go back to the Radxa instructions to create a virtual environment in the LLM-TPU/models/Llama3 directory:

Now copy modeling_llama.py to the transformers library in venv2, install a few extra dependencies (apparently missed by the requirements.txt file)

We will need to edit compile/files/Meta-Llama-3-8B-Instruct/config.json with our selected length:

and copy the file to the Llama3 directory from Huggingface:

Now generate the onnx file using the downloaded Llama3 and a 1024-byte length:

This part could be completed in a little over 6 minutes:

The next step would be to exit the Python virtual environment.

We can now start the compilation. I exited Docker above and also rebooted the Cincoze DS-1402, and the compilation part must take place in Docker. So we’ll need to reattach it, load the environment setting, and finally start the compilation:

There will be some warnings about the onnx model check, but those can be discarded:

This step could be completed in under 40 minutes on my machine:

Let’s copy the new model to the Fogwise Airbox:

Cincoze DS-1402 job is now done, so we need to continue on the Fogwise Airbox. First, let’s test the model:

It looks OK, but we can try to run the demo in the terminal:

It working fine:

But you’ll notice the speed has gone down with about 7.2 tokens generated per second. If I try to translate a few paragraphs in Thai, I’m still hitting the new 1024-byte length limit, but it goes further: