Firmware Programming – How to Format Serial Communication Data

Firmware Programming – How to Format Serial Communication Data

In this article you will discover how to format data such that it can be efficiently transmitted, received, and processed via a serial communication interface.

Every embedded systems engineer will have to, at some point, transmit data over a serial communication bus like UART or SPI in which the data is transmitted and received one byte at a time.

The question when approaching such a communication system is how to format the data such that it can be transmitted, received, and processed as intended.

For simple commands that don’t require any supporting data (perhaps you want to switch on/off an LED connected to an Arduino), a popular approach is to simply send over a single character as a command.

FREE GUIDE: Introduction to Microcontrollers

Using the Arduino as an example, you would send over ‘a’ to turn the LED on, and ‘b’ to turn the LED off.

The extent of your protocol definition could then be the following two lines on the Arduino:

As long as the communication requirements stay this simple, this may be all you need. The trouble arises as more complex and variable information needs to be sent over, like what value to set for an analog output pin, or the result from reading an ADC.

The approach many engineers reach for at this point is to simply “pack” the values into a byte array.

As an example, if you wanted to transmit at 16-bit ADC reading using C, you would pack it into a byte buffer like so:

On the other side, the receiver needs to undo that byte packing and reconstruct the original value. There are several apparent problems with this approach, the most immediate being the inherent complexity, which also makes it difficult to scale.

What if you also need to send over the time the data was captured, or other related values that go with the ADC reading? The following snippet would now look somewhat like this:

This process that ensures the data is properly encoded and decoded is tedious, manual, and prone to errors. Defines and comments can help alleviate these issues, but ultimately when it comes to transmission of grouped and/or structured data, this approach takes too much time and is too difficult to maintain to be useful.

Another approach is to try using a struct to store the data you want, and packing the struct into a buffer instead:

Use memcpy to pack the struct into an array. With some caveats (some care needs to be taken to ensure the memory locations work out), this can work and be more useful than trying to pack multiple separate variables into a buffer one at a time.

There is one major flaw. This approach increasingly anchors us to only transmitting between two systems with the same endianness, language, etc. Any other combination requires even more code to be written to do conversion, handle edge cases, etc.

Both the low level approaches discussed above (packing individual variables or an entire struct) have the advantage of potentially being very fast and consuming small amounts of memory, which some applications may require.

However, if the device in use allows it, the best solution is to go with a serializer, which is a library that can take some data structure, convert it to an array of bytes, and convert it back to a data structure.

To circumvent the issues of the two previous approaches, the serializer should have a few other qualities:

• The serialized data should be platform-independent, meaning the same buffer can be generated from Python or C, and the encoding and decoding functions can extract or encode the same information to/from said buffer regardless of language or platform
  • This means the data framework will define its own types that map to existing types on the target platforms
  • Another consequence of this is that the definition of the data structure is also platform agnostic. Some serializers may use something like JSON to define the structure, while others will come up with their own solution.
• The serializer should be easy to implement and use, requiring only a few lines of code and defined variables to decode/encode arbitrarily large data streams. The user should not need to understand the format of the serialized buffer in order to employ the encoding/decoding functions.

One of the more popular implementations of such a serializer is Google’s protocol buffers, which has libraries (both from Google and third parties) that support Java, Python, C, C++, Go, and more.

This powerful framework enables simple, straightforward transmission and reception of structured data between various applications potentially written in different languages or running on different targets.

The protocol has a ton of powerful features, but to show how easy it is to configure and use in a simple application, I will demonstrate how to go about transmitting data from a microcontroller (firmware written in C) to a client written in Python, using the same example I built upon earlier.

The first step is to create the definition for the data structure to use. All protocol buffers implementations use a file with the .proto extension to accomplish this.

Since we are transmitting ADC data, this file could be called adc.proto with the following contents:

To break down the above:

• At the top of the file we define the version of the protobuf syntax to use. Proto3 is the newest
• In the message adc_data{} block, we define the data structure for our ADC data. Note that the values are all still present but they are written as an unfamiliar ‘uint32’. This is one of the protobuf-specific data types defined in the documentation. This will map to a uint32_t on C and an int() in Python
• Finally, in the message msg_oneof block, we define a message that can contain one of many different message types. The benefit of this is it allows a decoder to be able to process one of potentially many message types without knowing which one is being transmitted ahead of time. With only a single message type being employed here, this is not strictly necessary, but I left it in as an example of how protobuf makes scaling to larger and more complex applications much easier.

This file forms the foundation of our protocol buffers specification on both the microcontroller side and the PC side.

The next step is to use this file to generate boilerplate code that each application will include in order to map the fields defined above to language-specific structures, and to provide the definitions to enable encoding/decoding.

Get your FREE Ultimate Guide - How to Develop and Prototype a New Electronic Hardware Product in 2024

The program used to do this can vary with the language used. For Python, Google’s own protocol buffer compiler (protoc) can be used. For embedded firmware that supports C++, protoc can be used as well.

But for those writing code in strictly C, there is a third-part library nanopb that implements protocol buffers in pure C, and is otherwise geared towards embedded systems.

This example will use protoc to generate the Python code and nanopb to generate the microcontroller code. The specifics of how to do this can be found in the documentation for each program, but in short it simply involves running a command-line program that runs on the adc.proto file.

A source and header file is generated from nanopb for inclusion in the embedded project (note that nanopb also specifies some general library files that also need to be included).

For Python a single .py file is created to be imported into the application script (note that this file will automatically import the Python protocol buffer module).

Now, creating the code to construct and send the message is done as follows:

The msg struct is used to contain the actual data and message information in an abstracted system data structure, and the stream struct manages the destination buffer for the serialized data.

The pb_encode() function ties it all together by processing the structured data and piping its serialized format into the buffer.

Note the simplicity of the code as well as the readability. Besides variable creation, and actually storing the ADC variables, there are only three protobuf-related lines of code: setting the value of msg.which_msg, and creating the stream as well as encoding the buffer.

Note how the bytes_written member of the stream struct permitted only transmitting exactly what was needed. If, for example, the error code was zero, the serialized data would simply not include that variable in the buffer, making the effective size shorter.

Recall that this exact packet will make its way (via UART, presumably) to a PC, where it must be decoded using Python. The method for decoding in Python is just as simple, if not simpler:

# assumes “packet” variable already contains the received packet

# not necessary in this example, but would be used if there were multiple message types

There we have it — this code enables structured data to be passed effortlessly from an embedded target to a PC, with nicely abstracted code that is easy to read, easy to write, and can be maintained and scaled without much issue.

Things primarily come down to understanding the syntax of the .proto file, and knowing the relevant functions for encoding, decoding, etc. for the implementation of protobuf you wish to use.

Hopefully, you now understand the motivation behind using something like protocol buffers and what it can do to streamline an embedded project’s workflow and code strength.

I encourage you to give protocol buffers a try in your next project!

Written by Brandon Alba

Other content you may like:

• How to Connect an ESP32 WiFi Microcontroller to a Raspberry Pi Using IoT MQTT
• Auto-Generated Firmware Code: Deconstructing an STM32CubeMX Project
• How to Update Embedded Firmware Over-The-Air (OTA)
• The Beginner’s Guide to Designing with the dsPIC33 Microcontroller
• Home Automation with an Arduino: A Basic Tutorial