Interview – NXP Linux BSP and Timesys Vigiles Maintenance Service & Security Updates

I’ve been interviewing Ed White, Manager of NXP’s Professional Support and Engineering Services, and Akshay Bhat, Director of Engineering, Security Solutions at Timesys by email to find out more about NXP Linux BSP development process, and how Timesys can help to keep it updated and secure with its Vigiles service.

Q1. CNX Software readers recently discussed NXP Linux BSP update status. One person specifically noted Linux 4.14.98 used in the BSP was well over a year old, and there were various opinions about the topic, including one person suggesting NXP only provides a stable BSP and it was the ultimate responsibility of the customer to merge Linux security patchsets. Could you explain the typical development process for NXP Linux BSP, and why the company chose not to update the patchsets regularly?

Answer: The kernel strategy for NXP’s i.MX family BSPs closely follows the annual cadence of kernel.org’s LTS kernel selection. As soon as kernel.org establishes the next official LTS kernel version, NXP transitions our internal development to that particular kernel. However, the migration of the kernel is only one aspect of our next major release. There may be a number of associated updates to be included, such as a new version to Yocto, updates to U-Boot, and many other package changes we integrate into the Yocto release specific to the i.MX BSP. These factors, plus our rigorous testing process create an inevitable delay between the community version of the latest LTS kernel release and NXP’s i.MX board support package (BSP) based on that same kernel.

We must also consider a number of other factors that come into play between our planned cadence of Linux LTS kernel updates. NXP may introduce new products, or there may be updates to various packages, and of course, there are issue resolutions (including LTS minor version updates) to be considered. Our engineering team must balance all these factors while maintaining consistent quality standards for the entire i.MX product family being supported by each BSP release.

We (NXP) provide support for each new LTS kernel for at least one (1) year after delivering the initial general availability (GA) release of that version. From the change factors I just described, one can expect to see several minor updates in between the major LTS kernel version updates. These typically come as complete BSP releases, and occasionally as a specific patch update. Customers developing i.MX-based products should monitor NXP.com to ensure they have the latest BSP release and updates.

I read a discussion from one of your readers about an NXP partner product released in May of 2020 based on our L4.14.98 BSP. The discussion implied NXP has not provided a more current Linux kernel version BSP for i.MX – which isn’t true. Since releasing the L4.14 kernel version BSP, We have delivered an L4.19 LTS kernel version BSP as well as a few minor version updates in-between. In late 2019 the development team was already migrating to LTS kernel version L5.4, which was initially released by us in Q1, 2020.

In defense of any NXP partners or customers developing products based on i.MX products, they each have their own development and test cycles and typically select the NXP BSP version that matches their production schedule and goals. Despite having a product “officially” developed on a specific NXP BSP kernel version does not mean they haven’t been monitoring their BSP for the latest vulnerabilities and applying the appropriate updates. Let’s discuss how NXP is enabling the capability to have the latest and most secure SW platform – without a wholesale BSP update – by using the Vigiles tool powered by Timesys.

Q2. Soon after this discussion, I discovered Timesys Vigiles service that makes sure the Linux BSP is secure with the customer’s own patches. Could you provide an overview of the development workflow?

Answer: NXP/Timesys has two offerings to help maintain device security:

1. Vigiles is a security & vulnerability notification and reporting tool for monitoring your software. Vigiles includes many features like links to patches, mitigation information, alerts for new CVEs, collaboration tools, etc. Using information provided by Vigiles end customers can secure the BSPs themselves (e.g.: patch/upgrade packages and kernel).

   

   Note: Vigiles is pre-integrated in NXP i.MX Yocto BSP releases. Customers can register for a Vigiles Prime 30-day evaluation to experience all the features of the tool in their own project. (After 30 days, it converts to a less feature-rich, free version.) See sample demo report.

2. The BSP Maintenance service is a managed service whereby customers provide their hardware and BSP sources along with any custom patches to NXP/Timesys team for security maintenance. As part of BSP maintenance service customers get:

• A subscription to Vigiles Prime: Security & vulnerability notification and reporting tool for monitoring your software. Vigiles Prime is used to collaborate between customer and BSP Maintenance teams on planned security releases.
• Complete BSP update (software release) twice a year (by default, and the cadence can be changed if needed) on a mutually agreed timeline
• Minor kernel version upgrade for security and bug fixes
• User space security patching & package updates
• Each update is validated and tested on the customer’s hardware
• Release notes and test reports included with each update
• Customer-provided hardware is maintained in our Timesys board farm
• In the event something critical happens between updates…
• On-demand update for emergency security fixes (one per year included)

More about the process answered under Q4, BSP maintenance section.

Q3. New CVE (Common Vulnerabilities and Exposures) get discovered daily, so the system needs to be updated regularly to be kept secure. How often does a product managed through Vigiles typically get security updates?

Answer:

• Vigiles subscribers: Vigiles’ curated database is updated on a daily basis. Customers can subscribe to daily/weekly/monthly notifications to be alerted about CVEs applicable to components in their BSP. Customers will need to use the information available in Vigiles to DIY patch/upgrade components to address the vulnerabilities. Since customers are driving the process, they also choose the exact time and frequency to roll out security updates in their products.
• BSP maintenance subscribers: Customers subscribing to the full BSP maintenance offering have the option to select their desired upgrade cadence. The typical options are: monthly, quarterly, semi-annual, or yearly. The cost of BSP maintenance varies based on the selected option. The service also includes one emergency release for addressing any high severity vulnerability outside of the scheduled release cadence.

Q4. Some open-source tools, including the Yocto Project, come with CVE checkers. That means some customers could apply the latest Linux patches for Linux 4.14.xxx and run a CVE checker to find out potential vulnerabilities themselves. Would Vigiles still make sense in that case? What would be the added benefits of the offer?

Answer: Yes, the Vigiles and BSP maintenance services still make sense for customers like you are describing.

Our Vigiles services provide significantly more accurate and timely vulnerability data and more accurate software component analysis than open-source CVE checkers, which dramatically cuts analysis and mitigation time. Our services also include the collaboration, mitigation, and update tools and services that are needed for an efficient security and maintenance process. These are based in large part on industry best practices derived from thousands of software projects we have supported over the years, all at a very low-cost.

While Vigiles offers the above benefits over open-source CVE checkers to reduce the burden in accurately monitoring and finding fixes, there is still effort/skill required for patching, resolving conflicts, and testing. Hence, customers might want to go with the managed BSP maintenance service to completely offload the BSP security maintenance burden to NXP/Timesys. This would provide significant cost savings and result in a more secure BSP.

Someone taking a DIY approach with open-source CVE checkers can certainly try to recreate that and reinvent the wheel, but it would not make sense to do so.

To provide specifics about these added benefits:

• Vigiles:
  • Vulnerability Data Accuracy: The open-source tools rely on the NVD database. The NVD data has quite a few false positives and missed CVEs due to data entry errors and/or outdated information. E.g.: When a CVE fix is backported to an LTS kernel version, the NVD database is not updated with this information. (Slide 17, 22 of ELC presentation).
    
    Vigiles pulls vulnerability data from multiple sources such as NVD and Security Bulletins (Ubuntu, Debian), which is run through in-house-developed curation algorithms to reduce false positives (by cross-verifying available fixes against git commits to confirm affected version ranges, etc.), map vulnerabilities to package config options and limit to affected platforms. In addition, the Timesys security research team manually monitors vulnerabilities in packages from build systems such as Yocto and Buildroot, SoC vendor advisories, and mailing lists such as oss-security, and adds/curates vulnerability information as needed. All of these steps lead to improved accuracy and coverage while reducing false positives in our curated database (up to 40% improvement over NVD for certain packages).
  • Vulnerability Data reporting delay: Oftentimes, there is a delay in when a vulnerability is reported to when it gets published and analyzed by NIST NVD. This can result in high severity vulnerabilities being reported in the news but not in open-source tools relying on NVD (Slide 23 of ELC presentation). Since Vigiles augments multiple sources for vulnerability data, we report up to four weeks earlier than NVD in some cases.
  • Tool accuracy: The CVE checking tools themselves have issues with the way NVD feeds are parsed and versions are compared. Recent improvements have been made but have not been backported to older versions of Yocto, leading to a false sense of security. (Slide 19, 20 of ELC presentation).Vigiles is maintained for older releases of Yocto and Buildroot and does not suffer from similar deficiencies.
  • Ease of use: Open-source tools such as the one built into Yocto do not give a high-level overview of which package has which vulnerability. Further, they don’t differentiate build time vs. run time packages, leading to a lot of manual and cumbersome investigative clean-up effort. Vigiles provides an easy to read a summary by package along with a detailed section that can be searched/sorted. Further, we only report CVEs for packages installed on the target.
  • Filtering: Most times, even though a vulnerability is applicable to a package, the feature resulting in the vulnerability might not be enabled in the build. Vigiles lets you upload kernel and u-boot configs and automatically filters out CVEs not applicable based on config options being used reducing analysis time by 4X. Further, one can filter based on severity or attack vector to help prioritize the ones that need to be addressed first.
  • Links to patches: Finding CVEs is only a part of the process. The real benefits come from identifying if a fix is available, whether a newer version has been released with the fix, has it been backported, and so on. This detailed analysis is a time-consuming process when you are dealing with hundreds of CVEs.Vigiles also gives links to patches and suggests which version of the software to upgrade to address the vulnerabilities.
  • Value-added features: Vigiles offers team collaboration and communication tools such as adding notes, whitelisting CVEs as not applicable to a project, and so on. It also highlights changes from one software component scan to another, gives a history plot, can send email alerts on the desired frequency, export reports in various formats, search the curated database, to list a few of the many features. All these features were added based on customer feedback, our own need to be efficient when addressing vulnerabilities and insights gained from the thousands of software projects Timesys has supported over the years.

Vigiles is more than a monitoring tool, it provides an end-to-end workflow to address vulnerabilities at volume. This enables a device maker to increase product security at a significantly lower cost than attempting to DIY with open source CVE checkers.

• BSP maintenance: The challenges/effort associated with DIY security maintenance is as follows:
  • Kernel security: The upstream kernel maintainers do a great job of backporting security and bug fixes to LTS branches. So migrating to the latest minor version of the LTS kernel typically addresses most security vulnerabilities. Taking 4.14.98 i.MX release as an example, there are ~6659 NXP patches on top of the mainline 4.14.98. Under BSP maintenance we would rebase the NXP patches on top of the latest 4.14 LTS kernel (eg: 4.14.183). This typically results in many conflicts since the same code can be modified by both NXP patches and in 4.14.183. We analyze which changes to keep from upstream and which ones to keep from NXP and resolve the conflicts.DIY route of security maintenance: As you can see, resolving conflicts requires a deep understanding of how the driver works. This is a highly specialized skill-set that is not widespread among end-customers who mainly consume BSPs.
  • User space security: Many times, upgrading a package might result in API changes that require end-user application changes. We work with the customer and backport or upgrade packages on a case-by-case basis, ensuring minimum burden to the end customer.DIY route of security maintenance: If patches are available, investigate if they can be applied cleanly. If they cannot, you then need to figure out how to backport and ensure a given patch addresses a vulnerability without breaking any functionality.
  • Testing and Reports: A driver test framework along with driver tests is made available to end customers and is pre-configured to test the peripherals based on end-customer hardware. NXP/Timesys runs this test suite with the updated BSP on the customer hardware. For user space, we run tests from either Yocto ptest or built-in test suites available in packages and verify the tests continue to pass. We provide vulnerability and test reports as part of the release which can be used for any required compliance (e.g.: FDA).DIY route of security updates: Customers need to write or integrate their own driver tests and configure them for their platform and validate the BSP. Further, generating testing and before/after vulnerability reports is a cumbersome process without prior investment in automation.
  • Support: We work as an extension to the customer team and are available for answering any security vulnerability related questions.DIY route of security maintenance: Engage with the open-source community and hope for a response or hunt for information on mailing lists, etc.

Having the right tools, automation and expertise is key to bringing the cost of BSP maintenance down and reducing security risk. In short, NXP/Timesys services capture industry best practices to help bring more secure products with reduced maintenance cost allowing end customers to focus on product value add.

Q5. Vigiles is specifically targeted at NXP Linux BSP, but what about other SoCs with mainline Linux support or a BSP with a Linux LTS kernel? Would you be able to provide support? If so, would the silicon vendor need to be involved, or any client could hire your services for support?

Answer: For NXP BSPs, Vigiles and/or BSP maintenance can be purchased directly through NXP.

For other vendors’ BSPs, Vigiles and/or BSP maintenance can be purchased through Timesys. The silicon vendor does not need to be involved, although Timesys has longstanding relationships with many vendors. Any end-customer can hire Timesys for their specific security or BSP maintenance needs. Timesys has expertise in maintaining BSPs on various SoCs. Further, if an end-customer is not on an LTS kernel, Timesys can first migrate them to an LTS version under an additional service contract and then start BSP maintenance to keep the device secure.

Biography of interviewees

Ed White, Manager of NXP’s Professional Support and Engineering Services. Ed has worked in the semiconductor business for over 24 years in predominantly marketing and product management roles. An employee of NXP since 2011, he is currently managing NXP’s global Professional Support and engineering services business, based in Austin, TX. Ed has an undergraduate degree in Computer Science (BSCS) and a Master’s degree in Business Administration (MBA).

Akshay Bhat, Director of Engineering, Security Solutions at Timesys. Akshay has experience with embedded systems that spans a broad range of industries with a focus on board bring-up, driver development, and software security. Akshay received his MS in Electrical Engineering from NYU Polytechnic University.

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress