From pocket sized payloads to dodgy device specifications, we discuss Daizy’s open-sourced solution to the complexities of handling communications with IoT devices, and the positive impact this is having on large scale IoT deployments.
Unlocking communication and analytics for every IoT device
At Daizy, we want to make using IoT devices super-consumable, and a big part of that is providing data from devices that can be readily ingested into downstream systems and analytics.
Domestic connected devices – USB charged and on a beefy Wi-Fi connection – can afford to send data in a format such as JSON that can be easily parsed by a server, ingested into data pipelines and even read by a human.
However, the constraints of LPWAN technologies means that devices cannot send data that is in a suitable format to use immediately. The long battery life of these devices is achieved precisely because they send tiny amounts of data. In the case of Sigfox, that’s just 12 bytes in every payload, and for LoRa typically 51 (larger payloads are possible, but most devices work to the worst-case scenario). Even if payload size is not constrained by the protocol – for example devices using cellular data – they are usually on a metered connection and still need to conserve battery life, so minimising payload size is crucial for sensor longevity and reduced maintenance costs.
Pocket sized payloads
There are a variety of tricks that can be deployed to minimise payload sizes. For example, if a number only needs to be between 0 and 100, such as a percentage, you can represent it in 7 bits instead of 8, which in a 12 byte payload would mean being able to fit 13 numbers instead of 12. Even if the number needs to be between 100 and 200, you can still use 7 bits and specify that the decoding needs to subtract 100 for the final result. Similarly, dividing a whole number is a good way of representing a decimal (floating point) number that traditionally would need 4 bytes. “Pseudo-numbers” – text that just happens to be numeric digits, like a serial number or a firmware version – can be represented as a series of 4 bit nibbles. In this way, for example, a 15 digit IMEI number can be packed into 8 bytes, that would otherwise need 15 bytes if it were represented as standard ASCII.
These tactics and others are employed in the vast majority of LPWAN payloads, and while they are great for minimising payload sizes, the downside is that every type of device comes with its own set of specifications that need to be understood and implemented in code to decode the device data.
Decoding payloads with Destruct-JS
To help us decode such payloads, Daizy created and open-sourced destruct-js, a Javascript library that allows us to specify the decoding rules in a simpler declarative manner that reads like a specification more than code, and avoid some of the common mistakes made in parsing binary data. Using destruct-js reduces the time we need to implement new device specifications, improves reusability and readability, and it can also do the same trick in reverse, mapping data objects to binary payloads so that we can send downlinks to devices.
Alas, the complexity doesn’t end there. The majority of devices will send multiple different payloads – for example one that is sent when the device is turned on, one as a regular “heartbeat”, one that contains readings etc. Writing decoders for such situations can get complicated – our most complex device modelled so far can be in any one of 50-plus modes sending nearly 70 different types of payload!
Making sense of device specifications
Even with our experience of writing a large number of payload decoders, the job is difficult without good documentation. In the best cases payload specifications from device vendors are clear, concise and come with plenty of examples. Unfortunately, this is not always the case – many specifications are little more than unformatted notes, or example C code buried deep in electronic specifications. Even if you have a good specification, it’s not unusual for a device to have undocumented behaviour, or for the documentation to contain examples that contradict it.
Organisations undertaking IoT deployments often overlook the complexity involved in this decoding of such payloads, and most IoT platforms leave it to the end user to work out the details for themselves. Even though the initial technical complexity is not huge – it’s just numbers after all – it’s a lower-level skill that is more readily found in developers working with embedded software than your typical application developer, and the challenges of documentation and complexity of device use-cases can make it even more difficult.
Removing the complexity of IoT device management with Daizy Aware
As a Daizy customer, the Daizy Aware programme means that our in-house device onboarding experts deal with this complexity, so you don’t have to, and we’ll work with you to make sure that your device payloads are understood, decoded and turned into normalised data suitable for ingesting into your systems.
