Sony IMX500-based smart camera works with AITRIOS software

Raspberry Pi recently received a strategic investment from Sony (Semiconductor Solutions Corporation) in order to provide a development platform for the company’s edge AI devices leveraging the AITRIOS platform.

We don’t have many details about the upcoming Raspberry Pi / Sony device, so instead, I decided to look into the AITRIOS platform, and currently, there’s a single hardware platform, LUCID Vision Labs SENSAiZ SZP123S-001 smart camera based on Sony IMX500 intelligent vision sensor, designed to work with Sony AITRIOS software.

LUCID SENSAiZ Smart camera

SENSAiZ SZP123S-001 specifications:

• Imaging  sensor – 12.33MP Sony IMX500 progressive scan CMOS sensor with rolling shutter, built-in DSP and dedicated on-chip SRAM to enable high-speed edge AI processing.
• Focal Length  – 4.35 mm
• Camera Sensor Format – 1/2.3″
• Pixels (H x V) – 4,056 x 3,040
• Pixel Size, H x V – 1.55 x 1.55 μm
• Networking – 10/100M RJ45 port
• Power Supply – PoE+ via RJ45 port
• Dimensions – 55 x 40 x 35 mm
• Weight  48 grams
• Temperature Range – Operating:  -10 to +55°C; storage: -30 to +60°C
• OS- Unnamed RTOS

According to a press release in Japanese, the Raspberry Pi’s AITRIOS development platform will be a camera module based on the same Sony “IMX500 / IMX501” intelligent vision sensor, so possibly the Raspberry Pi Camera Module 4 or Raspberry PI AI Camera 1… with support for Sony’s AI edge software.

The LUCID Vision Labs SENSAiZ SZP123S-001 smart camera is available now for $335 and works exclusively with the AITRIOS software platform. Image processing and AI image analysis run directly to the camera without the need for external processing or memory using the DSP and SRAM on the IMX500 sensor. A WiFi camera and an outdoor camera are also planned, both based on the same IMX500 sensor, but not currently available.

Sony AITRIOS Software

AITRIOS applications can be developed using Console web application, purchased in the “Portal” as SaaS, and the SDK on GitHub. Sony’s developer website for the AITRIOS platform describes its main components:

• Portal – Web application that serves as the gateway to the provision of AITRIOS services.
• Console – Web application that provides various functions for edge AI devices. Use the web UI, or control Console using the REST API or SDK. It includes functions to deploy AI models and vision applications to edge AI devices, AI model retraining, fetching inferred metadata and captured images, and state monitoring and device management.
• Edge AI Device –  Device equipped with an intelligent vision sensor from Sony Semiconductor Solutions Corporation such as the SENSAiZ SZP123S-001 describes above, or the upcoming Raspberry Pi development platform.
• Cloud SDK – SDK for developing vision service cloud applications. Use the API to take advantage of the edge AI device capabilities in Console.  The Cloud SDK is published on GitHub (Note: private repo that you can access after registering). Sony recommends using GitHub Codespaces, but software development can also be done with Visual Studio Code and Docker on your local PC.
• Vision and Sensing Application SDK – SDK for developing AI models and post-vision applications to deploy on edge AI devices.  Sony provides functions to retrain, deploy, and evaluate AI models, and for post vison applications, functions to deploy and evaluate applications, and debug functions before deployment. It’s also published on a private repository on GitHub.
• Base AI Model – A pre-trained AI model available for purchase in Portal such as Human Body Detection
• Camera FW – The Edge AI device firmware is provided by Sony
• Device Certificate –  Authentication file for using edge AI devices that you get by email after purchasing the device after asking one from Sony.

All downloads, support, and tutorials require registration on the developer’s website. I’ve registered to try to access resources, but even after successful registration, you still need to request access to the private GitHub repositories, and also purchase a license for the AITRIOS software:

• $50 Basic fee – A fixed monthly fee is charged as a Console usage fee.
• $13 monthly connection fee per device
• Model converter – $2 each time
• Model retrainer – $0.17 per minute

There’s no trial offer that I can see, so it’s a pain to evaluate at this time as you’d need to go through many hoops. Hopefully, this will change once the Raspberry Pi platform launches.

Yet the platform should prove to be interesting, as Edge Impulse just uploaded a video from the TinyML 2023 conference showing the Sony IMX500 using AITRIOS software compared to using ESP-Eye or a Raspberry Pi with a camera module and the performance in terms of frames per second is an order of magnitude better with the Sony devkit getting 4.5 fps against 0.3 fps for ESP-Eye, and slightly better than the 3.1 fps for the Raspberry Pi, but power consumption is way lower at about 26mW per inference against around 95 mW per inference on the Pi.

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress