Despite wear-leveling techniques, eMMC flash memories tend to wear out over time as they have limited write cycles. So we’ve seen in the past the importance of wear estimation in eMMC flash chips, and methods to limit write operation such as disabling logging when possible or write the log to RAM with log2ram in order to extend the life of flash-based storage devices.
My Xiaomi A1 smartphone basically became unusable after a little over a year due to eMMC flash issues, but that was not that big of an issue since I could just get another phone and the most important data is saved in the cloud nowadays.
That’s one thing when it happens to a phone, and another when it happens to your car, as some Tesla S & X owners realized when they lost access to the control screen in the car because the eMMC flash was worn out.
You’d think this should not stop people from driving, but since most features are controlled by the touch screen, one driver had the experience while driving in the rain and his car fogged out, and obviously, the navigation system does not work either leading another person to pay $175 to reflash the firmware. They had not figured out the eMMC flash was dying at the time, so the fix only worked for a couple of months.
So what happened exactly? InsideEVs reports this affects vehicles based on Tegra 3 based MCU (Media Control Unit) v1 that runs Linux, and stores the logs in /var directory since Tesla engineers did not disable those logs nor redirected them to the RAM. It worked well during the first few years, but apparently, firmware got larger from 300 MB to 1GB over the years leaving less space for the logs. That’s a problem because wear-leveling is less effective if there’s less space as the same sectors keep being written too.
A company called Gruber Motors offers to repair those boards by replacing the 8GB eMMC flash with a larger 16GB flash in order to give it more space to perform wear-leveling operations. The alternative is to purchase a new $1,800 MCU, not including installation fees.
As you may be aware, Tesla switched to Intel for their MCUv2 board, and they selected a 32GB eMMC flash so it should help. despite the new board adding extra features such as YouTube and 3D gaming. But Jason Hughes, from 057 Technology, explains excessive (and unnecessary) logging still happens in the software of the new cars, and to avoid further issues he currently moves the logging to RAM in his cars. However, this penalizes performance a little since RAM is limited.
Hopefully, Tesla will soon release a fix so that the eMMC flash in their MCU boards last longer.
Via Hackaday. Thanks to Zoobab for the tip.
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress
I don’t really know, if logs in question are of any use for diagnostic or troubleshooting purposes for Tesla, but what’s a point of logging to RAM? You loose those logs on power loss anyway, so why not log directly to /dev/null instead? That shouldn’t penalize any performance, for sure.
> what’s a point of logging to RAM?
It’s not really ‘logging to RAM’ but the goal is to keep logs in memory for some time and then write them back to the underlying storage in a batch since this dramatically decreases Write Amplification which is the real problem when writing constantly small amounts of data to flash memory.
If you write every few seconds a few bytes to the filesystem and sync the writes each time then the real amount of writes at the flash layer below is easily thousands of times higher compared to buffering these logs in RAM and writing them to flash storage every hour in a batch.
But I guess you’re right and writing those MCU logs to /dev/null wouldn’t make much of a difference since nobody will look at them anyway. Most probably that’s just a symptom for software development in the automotive world being as crappy as everywhere else…
So it’s write cache, focused on logs, so to speak.
Well, purely from a forensic point of view, anything’ that is not written directly on non-volatile media immediately is not much of a use – on impact’ that involves power loss you loose exactly the information you are interessted in.
Logging to RAM makes sense because you’ll want those logs during development, testing and so on and you don’t want to test one build configuration but actually ship another especially when disabling logging might trigger issues you can’t reproduce in your development build. There are millions or billions of devices logging stuff out to a UART that goes nowhere because of this.
Remember when you’d go to the train station and see some display with a blue screen of death? The new version of that is going to be a black screen with some Debian/Raspian boot up output, some messages about insufficient space or refusal to mount without fscking and a prompt from the initramfs recovery environment.
Or the famous press ctrl + d to continue on certain in flight entertainment / cabin management system touchscreens
youtube and gaming in the car ? Better be careful next time you cross such a car on your road!
Well so basically they’ve put some vital functions like air blowing to remove the fog on the entertainment system. Nice design… Maybe they simply didn’t have enough room left on the main system and considered it was secondary and could be installed somewhere else. This is developed exactly like a shitty web site, not like a reliable embedded system should be. Think that cars are supposed to live roughly 30 years once sold, you definitely understand that these ones will hardly live more than 5 years if they depend so much on such poor software.
As a fix they will log directly to the cloud. If the servers go down the entertainment screen will go black….
Edit: forgot to ask, is there a touchscreen directly on the required/essential system of the car? Could they have put ac controls to some other gui? Or would they need to add proper hw knobs? I guess i have to drive my diesel till normal e cars are available. I dont want to drive with an app.
>you definitely understand that these ones will hardly live more than 5 years
>if they depend so much on such poor software.
Before the eMMC dies you’ve already lived with the kia factory reject build quality for a few years so presumably being charged thousands of dollars each month to keep it running becomes less painful.
In automotive they have these ASIL (automotive safety integrity levels). For some reason, that it doesn’t also make sense to me, ECUs like the aircon are in the low integrity level, as also the speedometer and other functions. Only some critical parts like a few lights are ASIL-D, to the rest are integrated in the IVI.
But yeah, it doesn’t make sense to me either. If your IVI breaks then it actually makes your car useless and it’s ok!
Maybe Tesla should have used the Toradex flash lifetime estimation tool mentioned on cnx a few weeks ago. 🙂
https://www.cnx-software.com/2019/08/16/wear-estimation-emmc-flash-memory/