Firmware Programming – How to Format Serial Communication Data

In this article you will discover how to format data such that it can be efficiently transmitted, received, and processed via a serial communication interface.

Published on

Every embedded systems engineer will have to, at some point, transmit data over a serial communication bus like UART or SPI in which the data is transmitted and received one byte at a time.

The question when approaching such a communication system is how to format the data such that it can be transmitted, received, and processed as intended.

For simple commands that don’t require any supporting data (perhaps you want to switch on/off an LED connected to an Arduino), a popular approach is to simply send over a single character as a command.

Using the Arduino as an example, you would send over ‘a’ to turn the LED on, and ‘b’ to turn the LED off.

The extent of your protocol definition could then be the following two lines on the Arduino:

#define RX_LED_ON ‘a’
#define RX_LEF_OFF ‘b’

As long as the communication requirements stay this simple, this may be all you need. The trouble arises as more complex and variable information needs to be sent over, like what value to set for an analog output pin, or the result from reading an ADC.

The approach many engineers reach for at this point is to simply “pack” the values into a byte array.

As an example, if you wanted to transmit at 16-bit ADC reading using C, you would pack it into a byte buffer like so:

uint16_t adc_read = get_adc_val();
uint8_t buffer_to_send[2];

memcpy(&buffer_to_send[0], &adc_read, sizeof(adc_read)):
transmit_bytes(buffer_to_send);

On the other side, the receiver needs to undo that byte packing and reconstruct the original value. There are several apparent problems with this approach, the most immediate being the inherent complexity, which also makes it difficult to scale.

What if you also need to send over the time the data was captured, or other related values that go with the ADC reading? The following snippet would now look somewhat like this:

uint16_t adc_read = get_adc_val();
uint32_t time = get_tick_ms();
uint32_t err = get_adc_error_code();

uint8_t buffer_to_send[10];

memcpy(&buffer_to_send[0], &adc_read, sizeof(adc_read)):
memcpy(&buffer_to_send[2], &time, sizeof(time));
memcpy(&buffer_to_send[6], &err, sizeof(err));

transmit_bytes(buffer_to_send);

This process that ensures the data is properly encoded and decoded is tedious, manual, and prone to errors. Defines and comments can help alleviate these issues, but ultimately when it comes to transmission of grouped and/or structured data, this approach takes too much time and is too difficult to maintain to be useful.

Another approach is to try using a struct to store the data you want, and packing the struct into a buffer instead:

struct adc_data {
uint16_t reading;
uint32_t read_time;
uint32_t err;
};

Use memcpy to pack the struct into an array. With some caveats (some care needs to be taken to ensure the memory locations work out), this can work and be more useful than trying to pack multiple separate variables into a buffer one at a time.

There is one major flaw. This approach increasingly anchors us to only transmitting between two systems with the same endianness, language, etc. Any other combination requires even more code to be written to do conversion, handle edge cases, etc.

Both the low level approaches discussed above (packing individual variables or an entire struct) have the advantage of potentially being very fast and consuming small amounts of memory, which some applications may require.

However, if the device in use allows it, the best solution is to go with a serializer, which is a library that can take some data structure, convert it to an array of bytes, and convert it back to a data structure.

To circumvent the issues of the two previous approaches, the serializer should have a few other qualities:

  • The serialized data should be platform-independent, meaning the same buffer can be generated from Python or C, and the encoding and decoding functions can extract or encode the same information to/from said buffer regardless of language or platform
    • This means the data framework will define its own types that map to existing types on the target platforms
    • Another consequence of this is that the definition of the data structure is also platform agnostic. Some serializers may use something like JSON to define the structure, while others will come up with their own solution.
  • The serializer should be easy to implement and use, requiring only a few lines of code and defined variables to decode/encode arbitrarily large data streams. The user should not need to understand the format of the serialized buffer in order to employ the encoding/decoding functions.

One of the more popular implementations of such a serializer is Google’s protocol buffers, which has libraries (both from Google and third parties) that support Java, Python, C, C++, Go, and more.

This powerful framework enables simple, straightforward transmission and reception of structured data between various applications potentially written in different languages or running on different targets.

The protocol has a ton of powerful features, but to show how easy it is to configure and use in a simple application, I will demonstrate how to go about transmitting data from a microcontroller (firmware written in C) to a client written in Python, using the same example I built upon earlier.

The first step is to create the definition for the data structure to use. All protocol buffers implementations use a file with the .proto extension to accomplish this.

Since we are transmitting ADC data, this file could be called adc.proto with the following contents:

syntax = "proto3";

message adc_data
{
uint32 reading = 1;
uint32 read_time = 2;
uint32 err = 3;
}

message msg_oneof
{
oneof msg
{
adc_data msg_adc_data = 1;
}
}

To break down the above:

  • At the top of the file we define the version of the protobuf syntax to use. Proto3 is the newest
  • In the message adc_data{} block, we define the data structure for our ADC data. Note that the values are all still present but they are written as an unfamiliar ‘uint32’. This is one of the protobuf-specific data types defined in the documentation. This will map to a uint32_t on C and an int() in Python
  • Finally, in the message msg_oneof block, we define a message that can contain one of many different message types. The benefit of this is it allows a decoder to be able to process one of potentially many message types without knowing which one is being transmitted ahead of time. With only a single message type being employed here, this is not strictly necessary, but I left it in as an example of how protobuf makes scaling to larger and more complex applications much easier.

This file forms the foundation of our protocol buffers specification on both the microcontroller side and the PC side.

The next step is to use this file to generate boilerplate code that each application will include in order to map the fields defined above to language-specific structures, and to provide the definitions to enable encoding/decoding.

The program used to do this can vary with the language used. For Python, Google’s own protocol buffer compiler (protoc) can be used. For embedded firmware that supports C++, protoc can be used as well.

But for those writing code in strictly C, there is a third-part library nanopb that implements protocol buffers in pure C, and is otherwise geared towards embedded systems.

This example will use protoc to generate the Python code and nanopb to generate the microcontroller code. The specifics of how to do this can be found in the documentation for each program, but in short it simply involves running a command-line program that runs on the adc.proto file.

A source and header file is generated from nanopb for inclusion in the embedded project (note that nanopb also specifies some general library files that also need to be included).

For Python a single .py file is created to be imported into the application script (note that this file will automatically import the Python protocol buffer module).

Now, creating the code to construct and send the message is done as follows:

msg_oneof msg;
pb_ostream_t stream;
uint8_t tx_buf[30];

msg.which_msg = msg_oneof_msg_adc_data_tag;

msg.msg.msg_adc_data.reading = get_adc_val();
msg.msg.msg_adc_data.read_time = get_tick_msg();
msg.msg.msg_adc_data.err = get_adc_err();

stream = pb_ostream_from_buffer(tx_buf, 30);
pb_encode(&stream, msg_oneof_fields, &msg);

buffer_transmit(tx_buf, stream.bytes_written);

The msg struct is used to contain the actual data and message information in an abstracted system data structure, and the stream struct manages the destination buffer for the serialized data.

The pb_encode() function ties it all together by processing the structured data and piping its serialized format into the buffer.

Note the simplicity of the code as well as the readability. Besides variable creation, and actually storing the ADC variables, there are only three protobuf-related lines of code: setting the value of msg.which_msg, and creating the stream as well as encoding the buffer.

Note how the bytes_written member of the stream struct permitted only transmitting exactly what was needed. If, for example, the error code was zero, the serialized data would simply not include that variable in the buffer, making the effective size shorter.

Recall that this exact packet will make its way (via UART, presumably) to a PC, where it must be decoded using Python. The method for decoding in Python is just as simple, if not simpler:

# assumes “packet” variable already contains the received packet

msg = adc.msg_oneof()
msg.ParseFromString((packet))
msg_type = msg.WhichOneof(‘msg’)

# not necessary in this example, but would be used if there were multiple message types

if msg_type == ‘msg_adc_data’:
adc_val = msg.msg_adc_data.reading
time = msg.msg_adc_data.read_time
err = msg.msg_adc_data.err

There we have it — this code enables structured data to be passed effortlessly from an embedded target to a PC, with nicely abstracted code that is easy to read, easy to write, and can be maintained and scaled without much issue.

Things primarily come down to understanding the syntax of the .proto file, and knowing the relevant functions for encoding, decoding, etc. for the implementation of protobuf you wish to use.

Hopefully, you now understand the motivation behind using something like protocol buffers and what it can do to streamline an embedded project’s workflow and code strength.

I encourage you to give protocol buffers a try in your next project!

Written by Brandon Alba

Other content you may like:

2.3 3 votes
Article Rating
Subscribe
Notify of
guest
2 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
Martin

Yes, the dark world of application messaging.

Been involved in dozens of projects, and I have to say the key failure point, and root of so much evil is with application messaging. (both internally within an application, and to/from the outside world)

Hundreds of home brew messaging implementations. Mix that in with protobuf, and it can quickly get out of hand.

But lots of promise in protobuf ( and grpc ) land … just has to be deeply understood and managed tightly, especially in resource-constrained embedded C projects.

Lots of gotchas, even on something simple as an accelerometer message, returning an array of readings. Let’s say just a x-axis reading only returning a signed integer, since x_axis could be acceleration(+) or deceleration(-) state.

message accelerometer_data
{
repeated int32 x_reading = 1;
}

Lurking behind this are 2 potential issues. If one looks at nanopb output, it will use a callback within the defined C structure, since they length of the array is unknown/unbound in the .proto definition. The C developer would need to code some memory allocation, depending on how big the array is at that moment in time.

Fortunately, in nanobp, you can get rid of the callback, and have a fixed size array ( pros/cons ), with an options file. Add accelerometer_data.x_reading max_count:12, and nanopb will define a C structure of int32 set x_reading[12]

Second issue is with the array fixed at 12 elements, a C coder may be under the impression that the protobuf encoding would be in the range of 4 bytes * 12 elements ( 48 bytes ) plus some overhead bytes.

But it turns out this maximum byte count for this 12 element array, could end up being as large as 132 bytes, depending on the x_reading values. Further reading shows that if one uses sint32 instead of int32 type, that reduces to 72 bytes.

Long story, short … experiment/understand and define messages carefully, and be very careful with managing your message buffers for overflows, etc …

Copyright 2022 Predictable Designs LLC.  Privacy policy | Terms
  10645 N Oracle Blvd, Ste 121-117, Tucson, Arizona 85737 USA