Upgrading Embedded Linux Without Bricking – ELCE 2012

Arnout Vandecappelle, senior embedded software architect at Essensium/Mind, talks about ways to greatly decrease the risk of bricking your board/device during upgrade thanks to gubies scripts and tools at the Embedded Linux Conference in Barcelona, Spain, on November 6, 2012.

Abtract:

Embedded systems are often modified remotely, e.g. to upgrade the firmware or change the configuration. This may however break the system and render it inaccessible, which is a major problem if the device is hard to reach physically. Unfortunately, no catch-all failsafe solution exists to make sure that the device stays accessible remotely even if a modification goes wrong. Instead, the possible failures have to be anticipated and covered. This talk discusses some of the frequently occurring failures, how they can be detected and handled. These include power failure, kernel crashes, network failure and data corruption. We include examples of concrete use cases. Finally, there is room for discussion about possible alternative or more generic solutions than the ones proposed.

This talk is geared towards system architects and developers who want to improve the quality of their product.

1. Failure mechanisms
   • Power failure during firmware update. Solutions:
     • Add fail-safe firmware
     • Detect failed power
     • Atomic update of firmware images
     • Use journalling filesystem for writable data
   • Bad firmware. Solutions:
     • Fall back on previous (known good) firmware
     • Fail-safe firmware that can do upgrades
     • Upgrade script included in upgrade image
     • Watchdog reboot + boot fail-safe after bad boot
   • Communication errors. Solution: verify data before writing (e.g. GnuGP). If memory is lacking to store the full firmware before upgrade, split in chunks for streaming upgrade
     
2. Boot loader upgrade
   
   • if boot loader is broken, no recovery is possible =>  avoid bugs and features in the bootloader.
   • Upgrade of boot loader with (bootable) backup media
3. Package-based upgrade
   
   • Advantage – results in smaller upgrade files
   • Drawbacks – Difficult to predict what is installed exactly, more places where something can go wrong, and no package manager is truly atomic (but nixos package manager comes close).

You can download the slides for this presentation, as well as checkout gupies (Generic UPgrade Infrastructure for Embedded Systems), a set of set of scripts and source code that can be used for upgrading the firmware of embedded systems maintained by Arnout.

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress