The Teflon project is an open-source TensorFlow Lite delegate that currently supports VeriSilicon Vivante NPU and aims to support several other AI accelerators over time. The project has recently been upstreamed to Mesa meaning AI inference with open-source software will work out of the box in supported hardware once Mesa 24.1 is released in May 2024.
Teflon relies on the Etnaviv open-source driver better known for working with Vivante GPUs found, for instance, in NXP i.MX processors, but Tomeu Vizoso also worked on implementing Vivante NPU support since May 2022 thanks to a sponsorship from Libre Computer and documented the development progress on this blog.
The VeriSilicon Vivante “VIPNano-QI.7120” 5 TOPS NPU is found in the Amlogic A311D processor, and Tomeu tested Teflon with the Etnaviv NPU drive on both the Libre Computer AML-A311D-CC Alta and Khadas VIM3 single board computers with decent performance although still slower than with the closed-source binary provided by VeriSilicon.
- MobileNet V1 (UINT8) – ~15 ms inference speed
- MobileNet V2 (UINT8) – ~15.5 ms inference speed (about 6 times faster than the CPU and twice as slow as the proprietary driver)
- SSDLite MobileDet (UINT8) – ~53 ms inference speed (about 4 times faster than the CPU and 3 times slower than what the proprietary driver)
Getting upstreamed to Mesa provides several benefits. First, the userspace driver will be automatically packaged by distros using Mesa 3D graphics library such as Debian, Ubuntu, Fedora, and Yocto, so the Vivante NPU will be supported out of the box with the open-source driver. Teflon will benefit from the Mesa project’s infrastructure (Gitlab, continuous integration, etc…) and leverage the Mesa codebase with features such as NIR intermediate representation and the Gallium internal API. You’ll find more details in the blog post linked in the second paragraph of this post.
All that means is that we now have a relatively inexpensive ($65 on Amazon) Arm SBC running mainline Linux with a 5 TOPS NPU supported by open-source drivers.
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress
Tomeu has done some amazing work but also credit goes out to the others he mentioned on his blog. Neil Armstrong and BayLibre have been instrumental in making this even possible since a whole platform bringup in multiple open source projects is a major feat. Now that the initial support is in mesa, I expect the performance to exceed the proprietary implementations in a year’s time. Good times ahead. And thanks to those who purchase the our SBCs that make this work and other open source work possible.
!remind 1year
Some improvements with regards to performance: https://blog.tomeuvizoso.net/2024/02/etnaviv-npu-update-16-nice-performance.html
Support for the Etnaviv NPU driver for the NXP i.MX 8M Plus SoC has just been upstreamed too: https://blog.tomeuvizoso.net/2024/11/etnaviv-npu-update-21-support-for-npu.html