Serialization for Embedded Systems

This page details a way (one of many!) of serializing messages for embedded systems. The method described here is based on Google’s protobuf serialization format along with additional byte stuffing and framing to allow for the data to be streamed across a communication channel like UART, SPI, a WebSocket or any other “byte stream” style communication.

Schema vs. Schemaless Serialization

Is it better to use a schema-based serialization format or a schemaless one for embedded systems? I generally think formats with schemas are a better fit for embedded systems, because:

1. It allows for the creation of structs or classes in the embedded code that match the schema, which can be used to serialize and deserialize messages. Without these schema created structs it can be hard to represent the data without dynamic memory allocation (which is avoided at runtime in many embedded systems).
2. The schema provides a way of defining and documenting your “API” for the messages that are sent between devices. In typed languages this can improve the developer experience because your message types can be objects containing the message fields. The types, variable names and comments in the .proto file(s) can also serve as the documentation of the API (you can also create proper documentation from .proto files, e.g. see the GitHub repo pseudomuto/protoc-gen-doc).

Schema based serialization formats include protobuf, Cap’n Proto, and FlatBuffers. Schemaless serialization formats include JSON, CBOR, and MessagePack.

protobuf

We are going to use protobuf to convert objects in computer memory into a stream of bytes (this is what serialization is).

protobuf is a schema-based binary serialization format developed by Google. It is one of the most popular serialization formats. The schema (the message definitions) is defined in a .proto file. protobuf is officially supported in C++, Java, Python and other languages. One language that is not officially supported is C! Thankfully, there is a popular 3rd party tool called Nanopb which provides support for C, and is well-suited for use on embedded systems. For each language, protobuf comes in two parts:

1. A compiler that takes the .proto file and generates objects representing each message in the target language.
2. A runtime library that provides functions to serialize and deserialize messages.

Protobuf serializes each variable in a message into a tag (a unique variable ID) and the variable value. Providing a tag for each variable in a message might seem like a waste. Why not just send them in order and get rid of all the bytes that the tags take up? The reason is to allow for both forwards and backwards compatibility when the message format changes. The ids allow newer code to receive an old message and create default values for variables that are not present, and older code to receive a new message and just ignore variables that it doesn’t know about. It also allows for optional variables and for the ability to just not send the variable if it’s set to its default value.

Let’s look at a basic example to see how protobuf works. Here is a simple .proto file:

message myMessage
{
  uint32 myInt = 1;
}

We’ll set myInt to 7. This encodes to (I’m using https://www.protobufpal.com/ to generate the encoded message):

08 07

The first byte 0x08 is the tag shifted left by 3 bits (1 << 3). You can see this by changing the tag to 2 and seeing that the first byte changes to 0x10 (3 = 0x18, …). The second byte 0x07 is the value of the variable. uint32 is a variable-width type (there is no uint8 or uint16). Any uint32 up to 127 uses 2 bytes (1 byte for the tag and other data, 1 byte for the value). 128 uses 3 bytes:

Value Encoded Msg.
7     08 07
127   08 7F
128   08 80 01

protobuf supports a lot of nice types like string which is useful in the embedded world for serializing arbitrary length char *. It also supports bytes which is useful for serializing binary data. It also supports repeated which is useful for serializing arbitrary length arrays of other types.

One of its downsides for embedded systems is the lack of small variable types. It provides uint32 and bytes type (array of bytes), but no uint8, uint16 or bit fields! It does do clever serialization though and will encode small uint32 numbers in less than 4 bytes (as we saw above). However, we still lack the expressiveness of smaller types. And when using Nanopb, by default uint32s will all map to uint32_t in the generated structs, which may be a memory issue for some users. Luckily, Nanopb provides a way of specifying smaller types in the .proto file, so that the resulting structs use a smaller type (presumably an error is thrown during decoding if Nanopb received an encoded number that doesn’t fit in the smaller type).

If the lack of smaller types is a show stopper for you, have a look at bitproto¹ (more on this in the Alternatives section).

protobuf knows how to decode a sequence of received bytes into a message as long as you know:

1. The message type of the message.
2. You provide it with the exact bytes that were used to create the message.

This leaves you with two problems if you are sending the encoded bytes in a stream-like fashion, i.e. across a UART:

1. How do you know the type of the message, if you want to send more than one type of message?
2. How do you know when one message ends and the next one starts?

Protobuf Defaults

protobuf does not send message fields that are set to their default value. In proto3 there is no ability to set a user-defined default value (there were good reasons for this, relating to portability across multiple languages). Instead, protobuf uses the following default values:

• uint32, int32, sint32, fixed32, sfixed32: 0
• bool: false
• string, bytes, repeated: Array with 0 elements
• Enum: First value in the enum (which is always 0 in proto3).
• Message: Language-specific representation of null

Because of this, you have to be careful when considering basing logic around the “presence” of fields in the message. By default, for all field types except Message (i.e. all scalar types), you cannot distinguish between a field that was set to its default value and a field that was not assigned to at all. You can detect the presence of Message because it will be set to null if not assigned to (as per above).

protobuf provides an optional modifier if you do need to detect presence of scalar types. In this case, protobuf will send the field if it is set to its default value. How the encoder works out that the field was set is language specific. On the receiving/decoding side, protobuf provides a language-specific API to check if the field was set or not, as well as its value.

The Message Type

There are a few ways to solve the problem of not knowing the message type. Here are two ways you can solve this problem:

1. Use an enum to define the message types and then send that along with your message.
2. Use a “wrapper” message with a oneof field that contains all the possible message types (I recommend this approach).

Higher-level languages can solve this problem by using a RPC client. gRPC would be the obvious choice if using protobuf, as it’s built on top of protobuf, with the RPC definitions being defined alongside the message definitions in the .proto file. This is not a great solution for low-level embedded systems though, as the gRPC server and client are quite heavy. There are embedded orientated RPC libraries like eRPC but they are not as well supported as protobuf for higher level languages.

The enum Method

One solution is to use a enum to define the message types and then send that along with your message (e.g. in the header before the message, we’ll cover the header in more detail later). Here is an example of this in a .proto file:

enum MessageType
{
  HelloMessage = 1;
  GoodbyeMessage = 2;
}

message HelloMessage
{
  string greeting = 1;
}

message GoodbyeMessage
{
  string greeting = 1;
}

The oneof Method

A different approach is to use protobuf’s oneof functionality, and create a wrapper message which contains oneof all the possible messages that can be sent. oneof allows you specify a field which can be one of a number of different types (think of it like a union). Here is an example of this in a .proto file:

message HelloMsg
{
  string greeting = 1;
}

message GoodbyeMsg
{
  string greeting = 1;
}

message WrapperMsg
{
  oneof innerMsg {
    HelloMsg helloMsg = 1;
    GoodbyeMsg goodbyeMsg = 2;
  }
}

Behind the scenes, protobuf will encode an ID for each possible message in the oneof field (it uses the tag number), and expose a way in each programming language to be able to determine which type it is. In a way, this is very similar to the enum method above, except that protobuf is generating the enum and including it in the message, rather than you adding it to a header.

One further benefit from the wrapped message approach is that you can add additional generic fields to the wrapper message which will then be added to all messages. For example, this could be useful for holding things such as a timestamp or CRC checksum.

Framing and Byte Stuffing

protobuf does not provide a way to determine when one message ends and the next one starts. Again, this is a problem if you want to send messages in a stream-like fashion (e.g. send them through a serial link). The solution to this is to further process the protobuf-encoded message by adding a specific end-of-packet delimiter. But the problem is then you have to make sure the rest of the message contains no occurrences of the chosen delimiter. Removing these occurrences is called byte stuffing. One way to do byte stuffing is with escaping, another is with COBS (Consistent Overhead Byte Stuffing).

If you don’t have any personal preference, I recommend you use COBS. COBS is explained in detail in the COBS page. Below we will illustrate how escaping works:

Escaping

For example, you could pick the end-of-packet delimiter to be 0xFE (it’s a good idea to avoid commonly occurring bytes like 0x00 or 0xFF so that you don’t have to escape as many bytes). We will also pick a start-of-packet delimiter. This is not mandatory, but allows us to throw away bytes easily for example if we connect to the other device half-way through a packet. Let’s use 0xFD. We also need to pick an escape character, let’s choose 0xFC.

Then during encoding:

• Every time we come across the end-of-packet delimiter 0xFE, replace it with 0xFC 0x00.
• Every time we come across the start-of-packet delimiter 0xFD, replace it with 0xFC 0x01.
• Every time we come across the escape character 0xFC, replace it with 0xFC 0x02 (this is escaping the escape character).

Now we have a message which is guaranteed to have no 0xFE nor 0xFD bytes in it, as we can safely prefix and suffix these bytes to the message as unique start-of-packet and end-of-packet identifiers.

For example, let’s say we have the following protobuf encoded message:

0x08 0xFE 0x01 0xFC 0xAA

This would be escaped to:

0x08 0xFC 0x00 0x01 0xFC 0x02 0xAA

And then framed:

|<- SOP                                 |<- EOP
0xFD 0x08 0xFC 0x00 0x01 0xFC 0x02 0xAA 0xFE

This is what the sender sends across the communication channel, e.g. UART, SPI, etc.

What does the receiver do? The receiver throws away bytes until it receives a start-of-packet delimiter. Then it buffers received bytes until it receives an end-of-packet delimiter. Then it performs the reverse of the escaping process to get the original protobuf encoded message. This can then be decoded with protobuf if you used the oneof method above.

If you decided to use the enum method to determine the message type, you would need to add the message type to the start of the encoded protobuf message before the escaping and framing process. You would then unpack this during decoding and calling the correct decode function based on the message type. Another thing you may want to do is add a CRC checksum. This would be added to the header similarly to the message type, and ideally be calculated over the protobuf encoded message and the message type enum (if used).

Tip

If you are worried about how many extra bytes the escaping might add, take a look at COBS (Consistent Overhead Byte Stuffing). It’s a clever byte-stuffing scheme which has a maximum overhead of 1 byte per 254 bytes of data.

Adding an RPC Layer

Everything up to this point gives us a way to reliably send a self-contained message from one device to another. This is fine and is all you need for a large number of applications. But sometimes what you really want is to call a function on the remote device and get a result back, e.g. “read this sensor”, “set this config value”, “start a calibration”. This is what a Remote Procedure Call (RPC) layer provides: it makes calling a function on a remote device look (almost) like calling a local one.

As mentioned in The Message Type section, the heavyweight option here is gRPC, which is built directly on top of protobuf. It’s a poor fit for low-level embedded systems though, as the server and client are quite heavy (it relies on HTTP/2 and dynamic memory allocation). There are embedded-orientated RPC libraries like eRPC which generate client stubs and server shims from an IDL file, but they are not as well supported across higher-level languages as protobuf is.

Tip

If you want a battle-tested framework rather than rolling your own, take a look at Pigweed’s pw_rpc. It’s Google’s low-code-size embedded RPC system, designed to run with static (no dynamic) memory allocation, and it generates directly from your .proto files using either Nanopb or pw_protobuf as the backend. It supports unary and streaming calls, multiplexes calls over transport-agnostic “channels”, and ships clients for C++, Python, TypeScript, Java and JavaScript. The main trade-offs are that the device side is C++ (not C), and it’s happiest inside the broader Pigweed ecosystem and build tooling (i.e. Bazel). However examples for other development environments such as Zephyr do exist.²

The good news is that a lightweight RPC layer can be hand-rolled on top of the oneof wrapper message and framing we’ve already built up, usually with only a modest amount of code. There are three things you need to add:

1. Request/response correlation. Add a request_id field to your wrapper message, and fold the response messages into the same oneof. The caller remembers each request_id it has sent and which are still outstanding (“pending”). When a response frame arrives, the caller matches its request_id against the pending list to work out which call it belongs to. This is what allows you to have more than one call in-flight at once without getting the replies mixed up. This is a natural extension of the “add generic fields to the wrapper message” idea from the oneof Method section.
2. A dispatch table on the callee. When a request message decodes, switch on the oneof case, call the matching handler function, build the response message with the same request_id, and send it back through the same framing pipeline.
3. Caller-side ergonomics and failure handling. A stub function (e.g. add(a, b)) that builds the request, sends it, and waits for (or is called back with) the result. Crucially, this layer must handle the failure modes that don’t exist for local function calls.

Here is what the wrapper message might look like, extended for RPC:

message RpcEnvelope {
  uint32 request_id = 1;
  oneof body {
    // Requests.
    AddRequest        add_request        = 2;
    GetStatusRequest  get_status_request = 3;
    // Responses.
    AddResponse       add_response       = 10;
    GetStatusResponse get_status_response = 11;
    ErrorResponse     error              = 12;
  }
}

The most important things to get right are the failure modes, because these are what distinguish a remote call from a local one:

• Timeouts. The remote device might never answer (it could have reset, or the response frame could have been dropped or failed its CRC check). A local function call can’t time out, but a remote one must — otherwise the caller blocks forever.
• Errors. A dedicated ErrorResponse lets the callee report problems like “unknown method”, “invalid arguments” or “busy” back to the caller.
• Concurrency. The request_id makes it safe to have multiple outstanding requests at once.

Symmetric (Peer-to-Peer) RPC

The terms “client/server” and “caller/callee” make it sound like one device is in charge and the other just answers. That’s a common arrangement (e.g. a PC host driving a device), but it’s not a requirement. Client and server are roles per-call, not per-device. In a lot of embedded device-to-device links neither side is really more important than the other, and both want to call functions on the other. RPC handles this fine — you just let each device be both a caller and a callee at the same time.

That means each device needs:

• A dispatch table (so it can service incoming requests), and
• A pending-request table (so it can match incoming responses to the requests it sent).

The single piece of glue that makes this work is being able to tell, for any incoming envelope, whether it’s a request (and therefore to run it through the dispatch table) or a response (and therefore to match it against the pending table). The oneof case already tells you this — add_request is a request, add_response is a response — so no extra field is strictly needed. If you’d rather not infer it from the message type, add an explicit bool is_request (or a direction enum) to the envelope.

One subtlety with request_id: when both sides allocate ids independently, the two devices can easily pick the same id at the same time. This is not a problem as long as each device only ever matches a response against its own outgoing requests (i.e. the pending table is local to each device, and you never try to share a single id space across the link). Device A’s request #5 and device B’s request #5 are completely unrelated — each is resolved by the device that sent it. If you find the symmetry confusing to reason about, a common trick is to namespace the id with the originator (e.g. top bit = “who sent it”), but it usually isn’t necessary.

RPC on a Shared (One-to-Many) Bus

So far we’ve assumed a point-to-point link, where every byte you send goes to the one device you’re talking to. Many low-level buses aren’t like that, e.g. RS-485, CAN and I²C are all multi-drop, where every node hears every frame. RPC still works on these buses; it just needs one more thing: addressing.

Add a destination address (and usually a source address too) to the wrapper message RpcEnvelope:

message RpcEnvelope {
  uint32 dst_addr   = 1; // Which node this frame is for.
  uint32 src_addr   = 2; // Which node sent it.
  uint32 request_id = 3;
  oneof body {
    // ... requests and responses as before ...
  }
}

With addressing in place, most of the RPC layer is unchanged:

• Unicast call. The caller sets dst_addr to the node it wants. Every node on the bus receives the frame, but each one checks dst_addr against its own address and drops the frame if it isn’t a match. The targeted node dispatches the request and sends a response back with dst_addr set to the original sender’s address. From the RPC layer’s point of view this is identical to a point-to-point call — the bus is just acting as a filter.
• Correlation must be keyed by peer. A request_id is no longer unique on its own, because node 3 and node 5 could each have a request #7 in flight to you at the same time. So the pending-request table has to be keyed by (peer address, request_id), not request_id alone.

The part that is kinda weird in RPC is broadcast (sending a message to all nodes). A request sent to a broadcast address has no single responder, so it does not fit the RPC model of “just like calling a function” (which node’s response is the return value?). There are a few ways to handle this:

1. Make broadcasts fire-and-forget. No response is expected and nothing is added to the pending-request table — it’s a one-way command, e.g. “everyone, sync your clocks now”. You give up the return value in exchange for one-to-many delivery. This is really just the plain one-way messaging from earlier in this page, sent to a broadcast address.

2. Iterate unicast calls. If you actually need an answer from many nodes, poll each node with its own unicast call in turn. The RPC model stays intact (each call returns one value), at the cost of one round-trip per node.

3. Broadcast and collect (scatter-gather). If collisions are handled by a lower layer in the comms stack — either by bus arbitration (CAN), or by giving each node an assigned reply time-slot so the responses don’t talk over each other, then a single broadcast request can gather many responses. This works, but note it bends the “calling a function” mental model: the return value is no longer a single result, it’s a collection of (node → result) pairs (i.e. it would have to return something like an array or map). The RPC layer keeps one pending-request entry for the broadcast and appends each arriving response to it instead of resolving and discarding the entry on the first reply.

   The catch with scatter-gather is knowing when you’re done. With a unicast call, one response means the call is complete. With a broadcast, there’s no inherent “last” response, so you need one of:

   • A known roster of nodes expected to answer, so you can complete the call once all of them have replied (and flag the ones that didn’t as timed-out, a per-node version of the normal RPC timeout).
   • A plain collection timeout: gather whatever responds within a fixed window and return that. Simpler, but you can’t tell “node absent” from “node slow”.

Finally, watch out for who is allowed to talk, because the physical bus often dictates this and it can undo the symmetric peer-to-peer model from the previous section:

• On a half-duplex master/slave bus (classic RS-485, Modbus-style), only the master may initiate a transaction, and a slave may transmit only when addressed and only as a response — exactly one device drives the line at a time. A slave therefore can’t make an unsolicited RPC call; the best it can do is set a flag that the master picks up on its next poll.
• On a bus with built-in arbitration (CAN), any node can initiate at any time, so symmetric RPC works (both sides can initiate calls). This generally gives you more flexibility in the comms layer and reduces the amount of polling you have to do.

Is RPC a Good Idea for Embedded Systems?

It depends on the kind of traffic. RPC is a great fit for one-to-one links where one side needs to control/command the other. For example, a host (PC, phone or gateway) sends commands and queries like “read sensor”, “set config” or “start calibration”. The request/response model maps perfectly onto this, and the host side gets clean stub functions to work with. If you already have the oneof wrapper in place, adding RPC on top is cheap.

It’s a poor fit in a few cases, where plain one-way messaging (as described in the rest of this page) is the better choice:

• High-rate telemetry or streaming (e.g. continuous sensor data or log streams). These are fire-and-forget events, not calls. RPC’s correlation and waiting semantics add overhead and buy you nothing.
• Hard real-time paths. Blocking on a remote call would stall your control loop. On embedded systems you should make RPC asynchronous (callbacks or a state machine) rather than synchronous-blocking, unless the call is on a non-critical task/thread.
• Severely constrained MCUs. The pending-request table, timeout timers and per-call buffers all cost RAM. Usually this is small, but it’s not free.

In practice, many systems run both: a lightweight async RPC layer for commands, and plain one-way messages for telemetry — all sharing the same framing layer.

Nanopb

Nanopb is a popular library for working with protobuf in C/embedded systems.

Nanopb allows you to specify extra information in the .proto file to help it generate structs when compiling for C. This is useful for fields such as string, repeated and bytes, which are all variable length. If you don’t specify anything, Nanopb will expect you to use callbacks instead to handle the data as a stream. This is generally more cumbersome to work with than having fixed size members of message structs, so I strongly recommend specifying the extra options unless you have good reason not to.

First you have to add:

import "nanopb.proto";

Nanopb Options

Nanopb provides a number of special options you can specify in the .proto file which are used to avoid dynamic memory allocation (or the need for callbacks, and to reduce the memory footprint of the structs by allowing sizes smaller than 32-bits) and allow Nanopb to generate fixed sized structs for holding message data. These additional options don’t prevent you from compiling the .proto file for Python etc. with the official compiler, but you do have to make sure nanopb.proto is importable even if these options are going to be ignored for this target language.³

Installing nanopb also provides a version of protoc (the official protobuf compiler) that you can use.

max_size for bytes

You can use max_size to specify the maximum size of a bytes field:

message Image {
    bytes data = 1 [(nanopb).max_size = 256];
}

nanopb will generate a struct with 256 bytes for the data field:

typedef PB_BYTES_ARRAY_T(256) Image_data_t;
typedef struct _Image {
    Image_data_t data;
} Image;

max_count for repeated

You can use max_count to specify the maximum number of elements in a repeated field:

/**
 * Represents a single x,y cartesian point.
 */
message Point {
    float x = 1; // x coordinate, in the range [0, 1].
    float y = 2; // y coordinate, in the range [0, 1].
}

message PointsArray {
    repeated Point points = 1 [(nanopb).max_count = 10];
}

nanopb will generate a struct with 10 elements for the points field, along with a points_count field to keep track of how many elements are in the array:

typedef struct _Point {
    float x; /* x coordinate, in the range [0, 1]. */
    float y; /* y coordinate, in the range [0, 1]. */
} Point;

typedef struct _PointsArray {
    pb_size_t points_count;
    Point points[10];
} PointsArray;

max_length for string

You can use max_length to set the max. length of a string field:

message HelloMsg {
    string text = 1 [(nanopb).max_length = 40]; // The message text.
}

nanopb will generate a struct with a char array of length 41 for the text field (+1 to allow for the null terminator):

typedef struct _HelloMsg {
    char text[41];
} HelloMsg;

No length is needed because the string length is determined by the null character.

int_size

You can use the Nanopb option int_size with IS_8, IS_16, etc. to specify the exact size of integer fields in the struct. The standard .proto syntax doesn’t allow you to specify a smaller integer type than 32-bits (note though that it does compress small integers stored in these fields to a smaller number of bytes when sending them over the wire). This allows you to specify a smaller integer type in the generated struct, which is useful for saving on precious RAM on your MCU, especially for repeated fields.

import "nanopb.proto";

message Frame {
  uint32 my_number = 1 [(nanopb).int_size = IS_8]; // This will generate a struct with a uint8_t.
  int32 my_other_number = 2 [(nanopb).int_size = IS_16]; // This will generate a struct with a int16_t.
}

When using IS_8, IS_16, etc., the “sign-ness” of the integer type is preserved based on the original type (e.g. uint32 vs. int32).

The supported types for the int_size option are:

• IS_DEFAULT: Defaults to 32-bit or 64-bit depending on the protobuf type.
• IS_8: uint8_t or int8_t.
• IS_16: uint16_t or int16_t.
• IS_32: uint32_t or int32_t.
• IS_64: uint64_t or int64_t.

Generating Python Code

The Python code generated by the official protobuf compiler is awful. It embeds a giant string into a Python file which generates code at runtime. This means no nice message classes that you can inspect or be leveraged by Intellisense in an IDE.

Luckily, there are alternatives. danielgtaylor/python-betterproto is a popular one with 1.4k stars as of May 2024.⁴ It generates Python message classes (data classes, to be specific) from your .proto files. It also supports gRPC (not very useful for embedded).

Alternatives

bitproto is a protobuf-like serialization format which allows you to specify variable widths on the bit level, eliminating some of protobuf’s shortcomings when it comes to specifying small data types. It uses similar looking files to protobuf to describe messages.

The bitproto logo.

NordicSemiconductor/zcbor is a C (it’s also C++ compatible) CBOR library that uses CDDL to essentially provide a “schema” for defining the messages. C/C++ code is generated from this schema. As of May 2024, it has 100 stars on GitHub.

More Resources

See the protobuf page for general information on protobuf.

Footnotes

1. bitproto. Homepage [documentation]. Retrieved 2024-04-24, from https://bitproto.readthedocs.io/en/latest/. ↩

2. Pigweed. pw_rpc [documentation]. Retrieved 2026-06-29, from https://pigweed.dev/pw_rpc/. ↩

3. Petteri Aimonen. Nanopb: API reference [documentation]. Retrieved 2024-10-10, from https://jpa.kapsi.fi/nanopb/docs/reference.html. ↩

4. Daniel G. Taylor. danielgtaylor/python-betterproto [GitHub repository]. Retrieved 2024-05-02, from https://github.com/danielgtaylor/python-betterproto. ↩