Because of the interaction between several software and hardware products, continuous delivery in IoT environments has specific challenges not found in other environments. In this report, I will share the experience and challenges in enabling continuous delivery practices for Philips Hue and other IoT products. I will share what I learned about deploying firmware to customers’ devices, releasing mobile apps and doing backend deployments, while ensuring everything works together seamlessly.

1.     INTRODUCTION

For your web backend and frontend development, you probably have a continuous delivery process in place. But how about your mobile apps? Do you have a continuous delivery process in your mobile app development team? And what about the embedded firmware in your hardware products?

At Philips Hue, we build the world’s most popular smart home lighting system. The software in our products, and in most other IoT products, is a combination of embedded firmware, mobile apps, and web backend and frontend. The tools, practices, risks and delivery timelines between these software stacks are very different, but they all need to work together seamlessly.

In this report, I will share the experience and challenges in enabling continuous delivery practices for Philips Hue and other IoT products. These experiences were gathered over several years, and the ideas presented mostly came from the colleagues I worked with over the years.

I will share what I learned about deploying firmware to customers’ devices, releasing mobile apps and performing backend deployments, while ensuring everything works together seamlessly. I will share details on handling hardware diversity and automated acceptance testing in a IoT environment. Furthermore, I will share what I learned about user acceptance testing for our products.

2.     Background

Philips Hue is one of the leading brands in smart home technology. Our lighting products are used and loved by millions of customers all over the world. Our customers expect flawless quality and frequent updates with new functionality.

Besides supporting many different lamps, Philips Hue also supports switches and sensors. Even its central hub, the Hue bridge, has already received a hardware update with improved capabilities. Our challenges don’t just lie in this hardware diversity, but every single point in our system receives software updates. While continuous delivery has become standard practice for our backend, it is more challenging in mobile app and embedded firmware development.

As a product owner for mobile apps at Philips Hue, I have experienced these challenges first hand, and I have seen our development teams devise creative solutions to address these challenges.

3.     System Architecture Overview

Systems architecture of smart home technology platforms varies widely. However, most of them contain at least the following three elements:

• Hardware appliance(s) in the home of the user.
• Mobile app to control the hardware (usually via Bluetooth or Wifi).
• Server backend, to enable additional services, and to control the appliances remotely.

Figure 1 below provides a visual overview of the high-level system architecture of the Philips Hue smart home platform.

 

Figure 1: Philips Hue System Architecture Overview

To install and use the Philips Hue smart home system, the user will typically take the following steps:

• Install the hardware appliance(s) in their home:
  1. Connect the Philips Hue bridge to their wifi router. The bridge communicates wirelessly with Hue lamps.
  2. Place the Hue lamps in a fixture.
• Install the mobile app of the appliance manufacturer, and make a connection from the mobile app to the appliance(s). When the user opens the Philips Hue app for the first time, the app scans the wifi network for the Philips Hue bridge, and makes a connection to it.
• When the user is successfully connected to the appliance(s), they can control the appliance(s) directly. In the case of Philips Hue, the user can turn Hue lamps on and off and change the brightness and color of the lamps.
• Enable the server backend functionality. To enable the control of Philips Hue lamps when out of home, the user needs to create and login to their account, and couple the Philips Hue bridge to their account.

The software stacks in each of the components vary greatly:

• The hardware appliances contain embedded software, usually written in a lower-level language, such as C or C++. The update frequency of embedded software is typically a few months to a few years.
• The mobile app is written in the language and framework that is supported by the mobile OS manufacturer. Because of the ubiquity of the iOS and Android operating systems, this software is typically written in Objective-C/Swift or Java/Kotlin respectively. The update frequency of mobile apps is typically weekly to a few months.
• The number of programming languages used in server backends is too long to list, but Java is one of the most widely used. The update frequency of server backends is typically hourly to weekly.

4.     Deploying Software Updates

During the first installation of a smart home product, the embedded software in the hardware appliance can easily be over half a year old. The embedded software is installed in the factory typically as a last step during manufacturing. Additionally, the process that follows can easily take over six months: shipping from factory to central warehouse, shipping from central warehouse to shop, placing of the product on the shop shelves, and finally the user buying the product and taking it home.

This means that users are often confronted with a request for a software update right after installing the product in their homes. And that is only the beginning: to provide new functionality, software updates are continuously rolled out, in different frequency to users.

4.1       Firmware Deployments

To update the software of a hardware appliance means deploying firmware to the appliance. This can be done in two ways (Fig. 2):

• Via the server backend: the firmware is uploaded by the developer to the backend, and is downloaded and installed by the appliance.
• Via the mobile app: the firmware is embedded in, or downloaded by, the mobile app, and is downloaded and installed by the appliance.

Figure 2: Firmware Deployments Via Server Backend And Mobile App

It is best practice to enable firmware updates both from a server backend as well as via a mobile app. This has a clear advantage:

When the hardware appliance is connected to the internet, updates can be performed automatically by the backend server. Moreover, when the hardware appliance is not connected to the internet, the firmware of the hardware appliance can still be updated via the mobile app.

Note that the hardware appliance should be able to provide basic functionality without internet connection. For example, Philips Hue lamps do not need to be connected to the internet to work.

Learning 1: enable firmware deployments both via a server backend and mobile app.

4.2       Mobile App Deployments

Deploying a new version of a mobile app is typically easier than deploying new firmware to hardware appliances in customers’ homes. There are several services, including open source frameworks like Fastlane [Fastlane], that can be used to automate deployment of mobile apps to users.

However, automating the deployment process is just one part of the puzzle. Another challenge is maintaining software compatibility between different system components. In continuous delivery practices, multiple versions of the same service often run in parallel to ensure compatibility. Running multiple versions of the same firmware in parallel on a hardware appliance is often impossible due to hardware limitations, such as memory.

Figure 3: Update Bridge By Embedding Firmware In The App

Compatibility between the hardware appliance and the mobile app can be achieved by installing the compatible version of the firmware via the mobile app, see Figure 3. When offering new functionality to customers, it is typically enabled via an update of the mobile app. Usually, this new functionality in the mobile app requires the latest firmware version of the hardware appliance. By installing this latest firmware version via the mobile app at startup, we prevent that the mobile app needs to be backwards compatible (with older firmware). It also prevents that the hardware appliance needs to be forwards compatible (with newer app versions).

Learning 2: deploy the latest compatible firmware version via your mobile app.

Note that backwards compatibility of the hardware appliance (or forwards compatibility of the mobile app) is not guaranteed this way: in this model, once the firmware is updated, all clients that connect to it must update to a version that is compatible. However, this is usually not a problem, since most users of a mobile app should be able to update to the latest version available.

The method described above does have another drawback: users updating the mobile app are confronted with a mandatory software update in case the firmware of the hardware appliance is out of date. A mandatory update is very inconvenient if all you wanted to do is to just turn your Hue lamps on.

On the other hand, in case the update is not forced, the app should be compatible with all the older firmware versions in the field, including the first release which contained the very limited functionality. To prevent this problem, the Philips Hue app allows a user to postpone the mandatory update, but restricts the functionality to just the basics of turning lamps on and off and tuning brightness.

Learning 3: allow users to postpone mandatory firmware updates temporarily, to give access to the most basic use cases.

Even if customers can postpone mandatory firmware updates temporarily, it’s still better to prevent a mandatory update. Therefore, it is best to roll out firmware updates via the server backend, before the mobile app is released that requires that firmware. At Philips Hue, we typically rollout a new firmware update a few weeks before we release our mobile app that needs this latest firmware. By the time the customer updates their mobile app, the required firmware is already on the hardware appliance, and no mandatory update is necessary.

Learning 4: update the firmware of a hardware product, before you update the mobile app that uses it.

This process is also beneficial for new product releases. Typically, when we release new hardware that people can add to their existing system, we make sure that the Hue bridge has the latest firmware, and the mobile apps are updated a few weeks before the products are available in stores. When customers return home with their purchase, they will not be bothered by a mandatory software update at installation time, but have a frictionless installation experience.

5.     Hardware Diversity

Firmware updates and mobile app updates keep the user on the latest version of your software, but what happens when the hardware is updated? Not all users will immediately purchase new hardware at each update, so this means that the software needs to deal with a diversity of hardware in the field.

There are several ways to deal with this diversity. First, the best solution to deal with hardware diversity is to abstract away from it in software, so it is invisible to other parts of the system. For example, there are two major versions of the Philips Hue bridge, the v1 bridge running a custom operating system on an ARM processor, and the v2 bridge running the OpenWRT operating system on a MIPS processor. Both differences in underlying CPU and the operating system are abstracted away, as the bridge offers a single API for both versions.

Secondly, in case diversity cannot be abstracted away, the hardware appliance can expose its capabilities. The Philips Hue bridge, for example, exposes the maximum number of light scenes that can be stored, because the light scene capacity differs between bridge v1 and v2. The impact on the user experience in this case is limited: both users of bridge v1 and bridge v1 can create scenes until the maximum capacity is reached.

Lastly, in certain cases it is inevitable that the hardware diversity impacts the user experience. For example, Philips Hue has several different types of lamps: lamps that can give white and coloured light, lamps that can give different shades of white, and lamps that are only dimmable, with a fixed white tone. It is impossible to completely abstract these differences away.

For example, a colour picker for a lamp that can’t show any colours doesn’t make sense. And what about setting the colour for a group of lamps of different types? The easiest technical solution in that case is to restrict the user interface to what the most limited lamp can do. But that would restrict users’ ability to control colours of Hue bulbs which feature colour functionality. Therefore, the best solution is to let all lamps perform to the maximum of their capabilities, and choose a sensible alternative for lamps that have limited capabilities.

Learning 5: minimize impact of hardware diversity via software abstraction, exposing capabilities, or choosing sensible alternatives.

6.     Automated Acceptance Testing

Automated acceptance testing is a fundamental practice of continuous delivery. From [ContinuousDelivery]: “The acceptance test suite as a whole both verifies that the application delivers the business value expected by the customer and guards against regressions or defects that break preexisting functions of the application.” Because in smart home products, the hardware is installed in users’ homes, and owned and maintained by the user, the hardware is a key part of the product. This introduces additional complexity in automated acceptance testing compared to pure service-based software environments.

One option is to include the hardware in automated acceptance tests. To do this, a reference hardware setup needs to be created, which must be kept intact and consistent. The advantage of using a consistent hardware setup, is that it tests against the real environment in production, the environment that users also have. Therefore, it allows to test the basic functionality of the smart home product end-to-end.

There are also several disadvantages to using real hardware in your acceptance test suite:

• Hardware can fail
• Hardware setups are costly to maintain
• Interaction with hardware is slower than pure software based tests
• Interference with other hardware, especially in using wireless communication
• Speeding up acceptance tests by running tests in parallel requires creating multiple identical hardware instances.

Figure 4: Test Automation Pyramid And Corresponding Test Doubles

Because of the above drawbacks, hardware should only be used in a limited number of tests. If you structured your tests according to the “test automation pyramid” from [SucceedingWithAgile] (see the left side of Figure 4), your hardware setups should be limited to the UI layer of your automated acceptance test suite. 

Learning 6: use real hardware for UI tests in your automated acceptance test suite.

If real hardware should only be used in the UI layer of your acceptance test suite, what do you use in the service layer? At Philips Hue, hardware can be emulated using a system emulator. The system emulator is also used in automated acceptance tests. The following disadvantages with real hardware setups are solved by using the system emulator.

• No hardware failures
• Easier to maintain than hardware
• Interaction can be faster than with real hardware
• No interference
• Multiple instances of the emulator allow running tests in parallel

Learning 7: create emulation software for your hardware, to be used for the service layer in your automated acceptance test suite.

Note that emulator software can still have the disadvantage of being slow to interact with. Therefore, when functionality can be verified on a unit test level in code, this is still preferred over automated acceptance tests with a system emulator.

7.     User Acceptance Testing

User acceptance testing in a mobile or cloud environment is typically done by supplying beta versions of software to end users, or by A/B testing new functionality with a small percentage of users in a production environment. In IoT environments, this production environment includes physical hardware, in different configurations, in users’ homes. The most complicated releases, are the releases of new hardware products, which also enable new functionality in other parts of the system, such as mobile applications.

At Philips Hue, user acceptance testing is performed with an active community of beta users. This community also gets access to hardware to test the combination of both hardware and software.

Learning 8: use real hardware in your user acceptance tests.

Maintaining a community of beta users is time consuming and expensive, but well worth the investment, considering the feedback we receive. Due to the confidentiality constraint of new product launches, many beta users are internal Philips employees. Note that to be effective, the beta community needs to be diverse, consisting of different types of users in different locations, using different hardware configurations.

8.     Acknowledgements

I would like to thank my colleagues at Philips Hue, for sharing their knowledge and insights with me. Most of the ideas shared in this report came from them. Specifically, I want to thank Jeroen Mols for reviewing and giving feedback on this report.

Next to my colleagues, I would like to thank Rebecca Wirfs-Brock and Joseph Yoder for their encouragement to submit this report, and Irina Tsyganok for her valuable reviews and feedback.

9.     References

[ContinuousDelivery] Jez Humble and David Farley, 2010. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley Professional.

[SucceedingWithAgile] Mike Cohn, 2009. Succeeding with Agile. Addison-Wesley Professional

[Fastlane] Fastlane. 2018. Fastlane: app automation done right. [ONLINE] Available at: https://fastlane.tools. [Accessed 20 April 2018].