The dreaded security bugs… call the Yocto Exterminator!
The dreaded security bugs… call the Yocto Exterminator!
By: Alan Levy
Lead Consultant, Embedded Systems
17th May 2017
In modern electronic product development, many of the key features tend to be realised in the software or firmware stage. These are the processes that turn a lump of metal, plastic and silicon into a useful device.
My speciality is embedded software development and my role is focussed on embedded operating systems, which more often than not these days means Linux. As a result, the security of embedded Linux systems is a key issue and one that I personally spend a lot of time thinking about.
It would seem reasonable to me that a customer should be able to expect two things from the embedded software in the product they have just bought. Firstly, that it has been developed by engineers who know what they are doing, and secondly, that it should be adequately bug free and secure. In other words, fit for purpose and right from the outset.
In reality, the time taken, in any product development project, to reach key milestones is crucial and any way of speeding up the process always seems attractive to project managers. By contrast, any professional software engineer will tell you that taking shortcuts to speed up development tends to go hand in hand with an increase in both bugs and security risks.
Even if your software appears to be solid on day one, bugs do emerge over time and security flaws are revealed. Suddenly, that top of the range consumer product you put on the market several months ago becomes the target of choice for every hacker on the planet.
At that point (and if you’re sensible, long before then), your top priority is to decide what to do about it.
A perfectly acceptable reaction is to say “just fix the bugs and update the devices as quickly as possible”. Sadly, in practise, a secure remote software update is a very complicated proposition and therefore it is done infrequently or, sometimes, not at all. Worse still, software update mechanisms can themselves introduce reliability and security holes if they are not properly designed and implemented. Homebrew update schemes also have a track record of creating, rather than solving, such problems.
What’s in an update?
Historically, embedded products have typically been updated by downloading the entire system image from a website and reprogramming everything every time. This has significant costs in terms of communications bandwidth, device downtime and difficulty of recovery from update errors.
But a full image update is not the only approach. Ideally, your update scheme needs:
• A simple, secure, modular and automated update mechanism so that the customer doesn’t need to worry about it.
• A way to track the statuses of numerous third party packages that make up a Linux distribution and a method that works out how to apply any changes to your software.
• An approach that minimises the costs involved in obtaining patches and bug fixes for third party packages. These patches and bug fixes will then need to be integrated into your software and validated.
At this point, you are probably wondering how to deal with this. Well, the answer is not to invent your own scheme but to use the solutions that other people have spent years developing and fine-tuning.
This is where ‘Yocto’ enters the picture. Yocto is a software toolset designed to build highly customised, small embedded Linux distributions for resource limited hardware. The name ‘Yocto’ was actually chosen because it is the naming prefix for the smallest measurement scale (10-24) in the SI system of units.
Yocto is based on layered, configurable scripts known as “recipes” that are “baked” to generate the software. In my opinion, Yocto has a number of advantages from a security perspective when compared with similar toolsets such as ‘Buildroot’. In particular, it is simple to use, extremely flexible, and it offers embedded Linux developers access to a massive base of open-source Linux software packages. That includes a number of reliable, secure, tried and trusted package-based update mechanisms.
What’s in a package?
• Every application, system utility and service (including the Linux kernel) is a package. Just as with Linux on desktops and servers, you can update individual packages rather than the entire system image. This minimises the time and effort to rebuild/re-test the software as well as the bandwidth and time required to update the devices in the field.
• Package maintenance is done by the teams that develop the corresponding packages. Package updates are also regularly integrated into the Yocto project and then made available to product developers. This means that all of the details of maintaining and securing packages and recipes are taken care of by people who understand them. All a product developer then has to do is use the latest recipes from the public Yocto Git repositories.
• Devices can also be updated using standard Linux package managers, such as ‘RPM’ (itself just another updateable package). These package managers support cryptographic signing and secure updates. As a result, if an update fails for any reason, the target system is still up and running. You also get the opportunity to try again instead of presenting the end user with a useless electronic brick.
Of course, Yocto won’t solve all of your problems but it does give you a robust starting point. When it comes to product development, it can save you months of development effort, making the project manager happy and putting you on the front foot. Most importantly, it might save you from that embarrassing security bug that would otherwise haunt you for the rest of your career.
For more information about Yocto, you can read Alan’s white paper on the subject here.
In modern electronic product development, many of the key features tend to be realised in the software or firmware stage. These are the processes that turn a lump of metal, plastic and silicon into a useful device.
My speciality is embedded software development and my role is focussed on embedded operating systems, which more often than not these days means Linux. As a result, the security of embedded Linux systems is a key issue and one that I personally spend a lot of time thinking about.
It would seem reasonable to me that a customer should be able to expect two things from the embedded software in the product they have just bought. Firstly, that it has been developed by engineers who know what they are doing, and secondly, that it should be adequately bug free and secure. In other words, fit for purpose and right from the outset.
In reality, the time taken, in any product development project, to reach key milestones is crucial and any way of speeding up the process always seems attractive to project managers. By contrast, any professional software engineer will tell you that taking shortcuts to speed up development tends to go hand in hand with an increase in both bugs and security risks.
Even if your software appears to be solid on day one, bugs do emerge over time and security flaws are revealed. Suddenly, that top of the range consumer product you put on the market several months ago becomes the target of choice for every hacker on the planet.
At that point (and if you’re sensible, long before then), your top priority is to decide what to do about it.
A perfectly acceptable reaction is to say “just fix the bugs and update the devices as quickly as possible”. Sadly, in practise, a secure remote software update is a very complicated proposition and therefore it is done infrequently or, sometimes, not at all.Worse still, software update mechanisms can themselves introduce reliability and security holes if they are not properly designed and implemented. Homebrew update schemes also have a track record of creating, rather than solving, such problems.
What’s in an update?
Historically, embedded products have typically been updated by downloading the entire system image from a website and reprogramming everything every time. This has significant costs in terms of communications bandwidth, device downtime and difficulty of recovery from update errors.
But a full image update is not the only approach. Ideally, your update scheme needs:
• A simple, secure, modular and automated update mechanism so that the customer doesn’t need to worry about it.
• A way to track the statuses of numerous third party packages that make up a Linux distribution and a method that works out how to apply any changes to your software.
• An approach that minimises the costs involved in obtaining patches and bug fixes for third party packages. These patches and bug fixes will then need to be integrated into your software and validated.
At this point, you are probably wondering how to deal with this. Well, the answer is not to invent your own scheme but to use the solutions that other people have spent years developing and fine-tuning.
This is where ‘Yocto’ enters the picture. Yocto is a software toolset designed to build highly customised, small embedded Linux distributions for resource limited hardware. The name ‘Yocto’ was actually chosen because it is the naming prefix for the smallest measurement scale (10-24) in the SI system of units.
Yocto is based on layered, configurable scripts known as “recipes” that are “baked” to generate the software. In my opinion, Yocto has a number of advantages from a security perspective when compared with similar toolsets such as ‘Buildroot’. In particular, it is simple to use, extremely flexible, and it offers embedded Linux developers access to a massive base of open-source Linux software packages. That includes a number of reliable, secure, tried and trusted package-based update mechanisms.
What’s in a package?
• Every application, system utility and service (including the Linux kernel) is a package. Just as with Linux on desktops and servers, you can update individual packages rather than the entire system image. This minimises the time and effort to rebuild/re-test the software as well as the bandwidth and time required to update the devices in the field.
• Package maintenance is done by the teams that develop the corresponding packages. Package updates are also regularly integrated into the Yocto project and then made available to product developers. This means that all of the details of maintaining and securing packages and recipes are taken care of by people who understand them. All a product developer then has to do is use the latest recipes from the public Yocto Git repositories.
• Devices can also be updated using standard Linux package managers, such as ‘RPM’ (itself just another updateable package). These package managers support cryptographic signing and secure updates. As a result, if an update fails for any reason, the target system is still up and running. You also get the opportunity to try again instead of presenting the end user with a useless electronic brick.
Of course, Yocto won’t solve all of your problems but it does give you a robust starting point. When it comes to product development, it can save you months of development effort, making the project manager happy and putting you on the front foot. Most importantly, it might save you from that embarrassing security bug that would otherwise haunt you for the rest of your career.
For more information about Yocto, you can read Alan’s white paper on the subject here.
Further Reading