iWave Systems i.MX8M Mini Devkit Targets Low-cost Facial Recognition Systems

33 Replies to “iWave Systems i.MX8M Mini Devkit Targets Low-cost Facial Recognition Systems”

1. Why such an old Linux kernel version? 4.14.98 was well over a year ago now

   Reply

   1. The worst is not that it was release over a year ago, but that it contains 7653 bugs that were since fixed in mainline and that the only two possible reasons for this kernel to have been left for dead for more than one year are either that they don’t care, which is not acceptable from a serious vendor, or that their kernel is so heavily patched that it cannot even stand security and stability updates, which is not an acceptable reason either. In any case, nobody should use this unmaintained BSP under such conditions!

      Usually NXP takes care of mainlining their code, so you’d be far much safer with an up-to-date 5.4 which IS maintained.

      Reply

      1. How much would a mainlined kernal add to SoC products production costs?

         Reply

         1. You mean if the silicon vendor already has a mainlined kernel?

            Reply

            1. Yes how much costs would it add..

            2. It’s not necessary a big deal in costs. It is more a question about comfort. It’s simple to just don’t care about coding standards etc.

            3. There may be significant costs in the short term as a company would have to rebase its code to be compliant with mainline Linux guidelines. They’d have to pay developer time to do it, and it may delay other projects. It’s further complicated because of closed-source binary blobs so they have to rely on other partnering companies (e.g. GPU IP) to do the same.

               But over the long term, it should pay off, as new features and security fixes are added to new Linux releases without the company having to be involved.

            4. But when they’re in an LTS branch, there is almost nothing to do. Just a “git rebase v4.14.181”, fix the rare conflicts which systematically mean that some code they changed was actually affected by a bug, and that’s all. If *this* is too hard, either they’re totally incompetent and nobody must trust them to deliver critical fixes, or their BSP is a pile of crap which ruins everything and it cannot simply be trusted at all.

            5. Okay, the costs in short term are true. I still don’t get why people aren’t doing it in the first place. It’s the best strategy you can have.

      2. That’s not what is going on. Their tech support people are trained on how to use and debug the released BSP. It costs a lot of time and money to train their people so NXP does not constantly update their BSP. The whole point of a BSP is so that you and tech support are looking at the same code while they try and help you fix your problem.

         Two different definitions of support are being conflated
         1) support as in mainline support. Is the code in mainline and does it work.
         2) support as in tech support. Will NXP accept a problem report and provide you a solution

         iMX code is in mainline and their developers take good care of it. That is the first definition.

         NXP will only provides human tech support on their released BSPs. That is the second definition. There is considerable time and effort involved with training their tech support people. Plus you and the tech support people have to be looking at the same code when you report a bug.

         Reply

         1. There is a corollary to this…

            If you ask NXP “do you support kernel 5.6”, they will answer no.. We only support the BSP. That is because you are asking that question to the tech support people and they answer in their work context.

            But then if you try mainline 5.6 you’ll discover that it works (most of the time). And you’ll say iMX is supported in mainline, why did they tell me it wasn’t!?! This is because you are now using the other definition of support.

            Reply

         2. … and that has been an ongoing discussion for a long time about commercial interest of companies compared to open source efforts and success of (partly to mostly) unpaid development work, but in either case leading into cultural improvements and growth of knowledge?
            What’s a silicon company’s philosophy?
            What is Allwinners philosophy (now)?
            http://www.allwinnertech.com/index.php?c=about&a=index&id=24
            What are NXP’s customer relations like?
            https://www.nxp.com/company/our-company/about-nxp/sustainability:CORP_SOCIAL_RESP

            Reply

         3. Jon, I’m not speaking about rebasing their BSP on top of each major release, I know that this is a major pain. I’m speaking about doing the very minimally required hygiene which is to rebase to the up-to-date version in the LTS branch they’re using (here it ought to have been 4.14.181 and not 4.14.98). Greg KH now maintains these kernels for 6 years and that’s awesome for every vendor. But what’s the point if they stick to a bogus, outdated version forever ?

            At work we ALWAYS rebase our patches on top of the latest LTS kernel of the same branch. We’re not doing it for each LTS kernel, we’re just picking the updated LTS kernel the day we emit a new maintenance release. We probably face 2 or 3 conflicts per branch in a 3 years life cycle, that’s ridiculously low, and this keeps our customers safe. It’s extremely cheap, I’d consider it not only a negligence, but even a fault *not* to do it.

            Please keep this in mine: every time you get a product running on an outdated kernel, you must imagine the product’s developers wearing dirty underpants that were last changed when they picked that kernel.

            Here these developers have been wearing the same brown underpants for 15 months now.

            Reply

         4. > It costs a lot of time and money to train their people so NXP does not constantly update their BSP.

            But if their BSP is based on an LTS kernel what’s the point of doing this when they ‘forget’ to update this LTS kernel to latest version available?

            Reply

            1. Just merge the the LTS branch in yourself, that’s what I do. The problem here is in keeping you and the support team all on the same piece of code. They don’t want it to be a moving target.

   2. Differentiate between what the tech support people that work at NXP will answer questions on vs what you can do on your own. The iMX8 is fully working in mainline. You can build a fully up to date kernel anytime you want to. That is different than saying to iMX tech support – your code is broken, here’s instructions on how to break it, now give me a fix. They’ll only do that for their released BSP. What NXP tech support is not going to do is debug your random mainline build for you. If you want to use mainline then you are on your own. But… iMX mainline is quite good since that is where the NXP developers do all of their work so it doesn’t have very many problems.

      So here’s how it works. Go ahead and use mainline. Say you find a kernel problem. Then you have to back off to their supported BSP and reproduce it there. Then you can file a problem report with NXP tech support and they will fix it in the released BSP. The fix will also appear in mainline since it is the same people making the fix.

      This doesn’t work for you if the kernel bug you encounter on mainline does not reproduce on their BSP. That’s possible because many other companies besides NXP work on mainline and one of them may have broken NXP’s code. In this case you have to track the bug down yourself with git bisect and then complain to whoever committed the code that broke things.

      Reply

      1. My point above is that their BSP is broken because they refuse to apply the required fixes on it and everyone in the world can figure how to break it, possibly remotely root it (because surely in these 7600 bugs some are remotely exploitable). By deploying this unmaintained code in the wild, you’re knowingly putting your customers at risk.

         It would be nice if some serious vendors (and I consider NXP as serious) would provide their BSP as patch sets and they would themselves ask their customers to apply them on top of branch X.Y. This way, if a customer calls their support saying “we’re on 4.14.98 with your BSP”, their support could say “so the bug probably is among the 7653 known ones, please update to 4.14.181, try again and report once you’ve done your homework”.

         Reply

         1. You just have to do the merge onto the latest kernel yourself. These tech support guys are not core kernel developers. They can’t deal with the constant code churn from mainline. Merging up to the current LTS versions is your job, not theirs. Also consider that every customer is going to merge up to a different LTS version.

            There is also a testing guarantee going on here. NXP has run a lot of tests on their BSPs. They don’t want to keep doing that over and over with each LTS point release. Maybe they will update the BSP once a year or so, but don’t count on it.

            A BSP is not something that you would ship. A BSP is a common set of code that you and the NXP tech support group use to communicate bugs on so that you are both looking at the same code.

            Look at the bright side, NXP keeps their mainline support up to date. You can merge right up to Linus’ release of the day and it will work (most of the time). That’s where Allwinner fails. Allwinner sends out the BSP but then there is no way to merge it into mainline since you can’t port the closed source pieces. You are simply stuck on their BSP forever with no solution for merging kernel fixes.

            Reply

            1. > You just have to do the merge onto the latest kernel yourself(…) A BSP is not something that you would ship

               This is the broken part of the process, which is demonstrated by every single router, NAS, modem or whatever blackbox vendor shipping with the exact version of the BSP they validated because “it comes from the SoC vendor who knows what they’re doing and we don’t want to risk to lose support”. Remember 3.3.8 as initially provided by OpenWRT that was found running in virtually any sub-5watt device even years after 3.3 was dead ?

               > NXP has run a lot of tests on their BSPs. They don’t want to keep doing that over and over with each LTS point release

               This is flawed as well: they validated that their code works as expected on a kernel having 7653 known bugs. For sure there is always a non-null risk of regression when updating, but in my experience it’s extremely rare in an LTS branch, so anyone would rather take the risk of getting one more bug and fix 7653 than the opposite.

               > A BSP is a common set of code that you and the NXP tech support group use to communicate bugs

               One more reason for just providing a patch set to apply on top of “latest 4.14.x” rather than making one believe 4.14.98 is a validated version of whatever, this version is broken and vulnerable and will sadly be found in many products just because of this approach.

               In addition, when a customer gets a series of patches to apply on top of something he really feels where the difference is between what is supported by the vendors and what is not. Here the boundary between the two is burry.

               > Look at the bright side, NXP keeps their mainline support up to date

               I know, and that’s why seeing the broken process in parallel really pisses me off. These guys have understood how to play well with Linux except on one point which is getting things fixed.

               This reminds me of Marvell’s BSPs. They are utter shit breaking so many things that you never know if the crashes you get come from their patches, the old kernel they started from, or the hardware itself, and since everything is mangled together it’s a real mess to rebase them. Fortunately, Marvell is not that bad at feeding mainline, they’re just extremely slow and their doc is lagging behind quite a lot. So once their products get old you can start to use them reliably (my mcbin only started to work without crashing with mainline 5.4).

            2. There is no perfect solution. It is simply not practical for the SOC vendor to track continuous kernel churn on a daily basis in their BSPs. NXP has a pretty good compromise in keeping everything in a shape where is it pretty easy to merge. About the only thing you could fault them for is not updating their BSPs once a year or so. But that costs money and they may not have the customers/revenue to justify that work for every BSP.

               The core concept here is to get everything into mainline so that the customer can do the update. And that’s where all of the Chinese vendors fail completely. If Allwinner doesn’t have the staff capable of mainlining why don’t they hire someone like bootlin to do it for them? Allwinner certain has the money for pay for this. For a company the size of Allwinner it’s a minor expense to pay and the benefits are great.

               Hers’ a lesson I wish the Chinese SOC vendors would learn. Customers that ship buy your chips. If software problems prevent them from shipping they aren’t going to buy anything. Getting your customer to shipping status is critically important, far more important that keeping some random driver closed sourced. Do you really think your competitors can’t write the same drivers? Of course they can and they have all done it. The only thing closed source achieves is to prevent your own customers from shipping. And then if they do ship you are forcing them to ship with code that is full of known security holes. At least give them the courtesy of providing a path where they can upgrade to an LTS release and close those holes.

            3. NXP is a reliable and highly professional vendor, extended customer support and aware of social and environmental responsibility. But if this skills on hardware not include, that there is software related trust in continuation on security updates, especially with ai-related recognition coding, what’s then their advantage over (formerly) less security concerned Chinese vendors, for example.
               Outsourcing that revision of mainline security means talking ’bout networking experts with longanimity. LTS releases updating is therefore on different levels delayed, compared to instant mainline (that business customers can include on their own responsibility and comparable investment into such being “up-to-date”).
               No supporter of closed source for every day means of communication, but knowing that there are many people working in commercial software business. AI recognition is worth commercial security, because that justifies trust into that relation.
               One question is on nature of bugs of older kernel versions and influence to already shipped products. Another. What updated driver and security improvements customers been missing since around a year?

            4. > What updated driver and security improvements customers been missing since around a year?
               Nobody should have to manually validate each of these 7653 fixes. The purpose of LTS is “take it all and trust us, if we fail we’ll fix it, it’s always safer up to date”. That doesn’t mean you must update all the time. It just means that the bare minimum the vendor must do is provide something that is up to date when shipped. If it takes two weeks of QA after the last change, nobody will complain that the kernel is 2 weeks old. Here we’re talking about a kernel that was last rebased before the product even existed! They’re not giving a good example, really!

            5. Got the point, probably, so maybe better just give it a while
               https://xkcd.com/1820/

            6. It may also be a business decision… as I can see there’s a service called “Vigiles Software: Keeping Your Linux BSP Secure”: https://www.nxp.com/support/support/nxp-engineering-services/vigiles-software-keeping-your-linux-bsp-secure:VIGILES

               I found it through an upcoming webinar that reads:

               Security should be considered throughout the lifetime of an embedded system. Continually changing threat environments, new deployment modes, and third-party software updates mean that your BSP software can no longer remain static and “frozen.”

               https://register.gotowebinar.com/register/2712071180302143755

               So based on the discussion above, I understand they provide a fixed BSP for which their support staff is trained, and then customers can either apply the patchsets themselves or with some help using VIGILES. The service is offered by Timesys.

            7. > So based on the discussion above, I understand they provide a fixed BSP for which their support staff is trained, and then customers can either apply the patchsets themselves or with some help using VIGILES.

               Then at the very least they should not propose 4.14.98 as the base but 4.14.0. The problem with 4.14.98 is that nobody knows the current versions and this one looks high enough to appear as the up-to-date one. When you see 4.14.0 there’s no doubt at least. Or if they need to be on top of 4.14.98+ because of some API changes, at least they must drop that 98 from the BSP kernel’s name.

            8. Reliable security (responsible for users behind business-to-business) with purchasing cost of a system is not overall cost for longterm support. That’s probably where “authorize access to a commercial/industrial space, home/office, transportation, banking, and government sites” description changes its static meaning?

            9. And by the way another proof that it doesn’t work is that iWave is their customer here and simply relies on the outdated BSP provided by the SoC vendor.

            10. give it a signet “not confidable on that BSP revision”?
                Who would?

         2. It is easy to generate a patch from most BSPs. Look in the kernel makefile and check out the kernel tree for the tag you see there. Then copy the BSP onto that checked out tree. Do ‘git diff’. Now you have your patch. Commit it, then merge in future tags if you want. That usually works but sometimes you will have to do it in smaller chunks, jumping three months ahead usually works, jumping two years ahead almost always fails (ie merge conflicts).

            I do wish all of the SOC vendors would stop the tarballs and simply provide git trees. That is especially important for Android. I really hate getting 8GB Android tarballs. Use a manifest!!!!! and only send me the deltas to AOSP.

            Allwinner has all of their stuff in internal git trees. They are idiots for not creating git mirrors for the public releases. They do have these but apparently you have to be Chinese to get access.

            Reply

   3. It’s based on latest NXP GA Release:
      https://source.codeaurora.org/external/imx/imx-manifest/tree/?h=imx-linux-sumo

      imx-4.14.98-2.0.0_ga.xml

      Reply

      1. Not familiar with codeaurora. Just talked to the postmarket OS people. They told me that codeaurora is basically the dump for Qualcomm etc.

         Reply

   4. Just posted an interview with NXP and Timesys that goes into more details about NXP Linux BSP and maintenance:
      https://www.cnx-software.com/2020/07/04/interview-nxp-linux-bsp-and-timesys-vigiles-maintenance-service-security-updates/

      Reply

2. iWave – never get schematics for SOM. After purchase they will constantly ask Zoom and solicitate their service.

   Reply

Leave a Reply

Connect with:

Your email address will not be published. Required fields are marked *

Comment *

Name *

Email *

Website