Generative AI on NVIDIA Jetson Orin, Jetpack 6 SDK to support multiple OSes

NVIDIA has had several announcements at ROSCon 2023 related to robotics & embedded with highlights including generative AI on the NVIDIA Jetson Orin module and the Jetpack 6 SDK will be released next month (November 2023) with supports for Ubuntu as usual, but also other operating systems and platforms such as Debian, Yocto, Wind River, Redhawk RTOS, and Balena.

Generative AI on NVIDIA Jetson Orin

There’s been a lot of hype in the last year about generative AI thanks to services such as ChatGPT, Google Bard, or Microsoft Bing Chat. But those rely on closed-source software that runs on powerful servers in the cloud. As we noted in our article about the “AI in a box” offline LLM solution there are some open-source projects such as Whisper speech-to-text model and Llama2 language models that could be run on embedded hardware at the edge, but as noted by some readers platforms based on Rockchip RK3588S or the Raspberry Pi 4 often lacks the processing power and the memory capacity to run useful generative AI models, although some the RK3588 boards now ship with 32GB RAM.

But NVIDIA Jetson AGX Orin is a beefier embedded AI computer with up to twelve Cortex-A78AE cores, a combination of a powerful GPU with AI accelerators delivering up to 275 TOPS, and up to 64GB RAM, that makes it suitable for generative AI and replace the legacy, more rigid, CNNs (Convolutional Neural Networks).

Developing a CNN model is a time-consuming endeavor as you’d typically need to provide plenty of data to train your model and it’s more rigid with fixed features since it’s not able to learn by itself. Generative AI does not require additional training, or what NVIDIA calls “zero-short learning”, and such models can learn to recognize things that they had not seen before.

It also enables a language interface, so instead of displaying boxes around detected objects, there’s a chatbot that can answer questions such as “How many boxes were unpacked in the last hour?” or “How many boxes per minute did the worker in orange vest unload?”. So generative AI is a much more powerful and useful solution and is likely to make CNN models obsolete on platforms with enough resources to handle the new technology.

NVIDIA benchmarks its generative AI implementations on Jetson Orin and an Intel Xeon Platinum 8480+ (56 cores, 112 threads @ 3.80 GHz, 350W TDP) server testing large language models (LLM), vision language models, vision transformers, and stable diffusion on both platforms, eventually finding out that the NVIDIA Orin delivers 1.7 times the performance of the Xeon-based system. I supposed the Orin was set to high power mode or 60W.

The NVIDIA Jetson Generative AI Lab provides tools and tutorials for deploying open-source LLMs (e.g. llama2), diffusion models to generate stunning images from text, vision language models (VLMs), and vision transformers (ViTs) that combine vision AI and natural language processing for an interactive description of the scene using a chatbot. Alternatively, developers can make use of the TAO Toolkit to create AI models for edge with minimal coding (low-code platform) and optimize vision AI models, including ViT models.

Jetpack 6 SDK

The NVIDIA Jetpack 5.0.2 production release was outed last November with Ubuntu 20.04 and support for the new Jetson Orin, and the JetPack 6 SDK is expected for release next month with support for Canonical’s optimized and certified Ubuntu OS (which version we are not told…), and for the first time other operating systems that include Wind River Linux, Concurrent Real’s Redhawk Linux and various Yocto-based distributions.

NVIDIA says AI developers will not be required to perform a full Jetson Linux upgrade to switch the JetPack 6 SDK so the upgrade will be easier and less time-consuming.  We don’t have that many other details for now, but the diagram above also shows new system services (IoT Edge Stack, Diagnostics, Fleet management…) and enhanced security with secure boot, memory encryption, and so on. Additional details should be made available at the time of the release

Further information on generative AI and the new JetPack SDK may also be found in the relevant blog post on NVIDIA’s website. The company also released the NVIDIA Isaac ROS 2.0 and NVIDIA Isaac Sim 2023.1 for robotics applications.

Jean-Luc Aufranc (CNXSoft)

Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress