The schema is the contract; the wire format is how data that honors it actually travels. This section digs into the raw bytes: the physical layer of the specification.
This guide should help explain how the Protobuf encoding works, but if you have questions about edge cases or specifics that we don't cover, refer to the official encoding guide.
Protobuf's encoding leans on three bit-level concepts: byte order, bit shifting, and bitwise OR. If you can already read x << 3 | y, skip ahead. If not, here is the short version.
Varints are the fundamental building block of Protobuf efficiency, allowing integers to occupy only as many bytes as necessary.
Standard integers in memory take 4 or 8 bytes regardless of their value. Varints use Base-128 Serialization to represent smaller numbers with fewer bytes.
Each byte in a varint, except the last byte, has the most significant bit (MSB) set to 1. This acts as a continuation flag, telling the decoder "more bytes are coming."
The lower 7 bits of each byte store the data in groups of 7, least significant group first. This means Protobuf uses a Little-Endian approach even at the bit-group level.
Chunk Data
Split the number into 7-bit groups. Standard bytes are 8 bits, but we reserve the top bit (MSB) as a "continuation bit".
Reverse & Add MSB Flag
The groups are written in Little-Endian order (least significant group first). Set the MSB to 1 for all bytes except the last one.
Standard Varints are great for positive numbers, but they are highly inefficient for negative ones. ZigZag encoding fixes this.
Standard Varint (Two's Complement)
10 Byte(s)As a plain varint, -1 takes all 10 bytes: a negative number has no leading zeros to drop.
ZigZag Varint
1 Byte(s)After ZigZag, size depends on the number's magnitude rather than its sign: -1 encodes in a single byte, and the byte count grows only as the value moves away from zero.
Two's complement stores the sign in the most significant bit, which means every negative number has its high bits set. That is bad news for varints, which save space by dropping leading zeros: a negative number has none to drop. Even -1 takes the full 10 bytes of a 64-bit varint.
ZigZag encoding moves the sign to the least significant bit instead. Positive numbers map to even integers (n << 1) and negative numbers to odd ones.
With the sign at the bottom, small negative numbers have leading zeros again and encode as compactly as small positive ones.
The name comes from how the mapping alternates between positive and negative as the encoded value counts up: 0 maps to 0, -1 to 1, 1 to 2, -2 to 3, 2 to 4, and so on.
ZigZag integers allow for more efficient storage of small negative numbers.
Every field in a Protobuf message is prefixed by a Tag. This tag is the only reason the decoder knows which field it's currently processing and how to interpret the bytes that follow.
Tag Composition
A tag is a single Varint that combines two pieces of information:
- Field Number (bits 3 through N)
- Wire Type (the bottom 3 bits)
The formula for the tag value is (field_number << 3) | wire_type. Small field numbers fit in a single byte; larger ones (16 and above) spill into additional bytes under the same continuation rule varints always follow.
A tag packs two facts into a single number: the field number and the wire type. The lowest three bits hold the wire type, and everything above them holds the field number.
One number, two parts
Encoded as a varint
Every field on the wire is wrapped in an "envelope" that tells the decoder two things: which field number it is, and how to read the payload. These are packed into a single tag value, encoded as a varint.
| Wire type | Protobuf types |
|---|---|
| 0 (Varint) | |
| 1 (I64) | |
| 2 (LEN) | also used for packed repeated scalars |
| 3 (SGROUP) | group start (deprecated) |
| 4 (EGROUP) | group end (deprecated) |
| 5 (I32) |
Varint Fields (Wire Type 0)
Most numeric types use Wire Type 0. The length is implicit because the decoder reads bytes one by one until it finds a byte where the MSB is 0. That's the same continuation-bit rule from the varint section earlier, doing the delimiting on its own.
Fixed-Size Fields (Wire Type 1, 5)
Some types have a known size, so nothing needs to be measured. Wire Type 1 is always 8 bytes, used by double and fixed64. Wire Type 5 is always 4 bytes, used by float and fixed32. The decoder simply reads that many bytes after the tag.
Length-Delimited (Wire Type 2)
string, bytes, and nested message fields use Length-Delimited encoding. These fields include an explicit length prefix (encoded as a varint) immediately after the tag, telling the decoder exactly how many subsequent bytes belong to this field.
Packed Repeated Fields
Repeated fields of primitive types use a specialized encoding to avoid repeating the field tag for every element.
Each element repeats the field tag. High overhead for many small elements.
Elements are concatenated into a single length-delimited record. One tag for the whole set.
Maps are not a native wire-level primitive. Instead, they are syntactic sugar for a repeated message.
map<string, int32> items = 1;
message Entry { string key = 1; int32 value = 2; } repeated Entry items = 1;
One of the biggest space savers in Protobuf is the omission of default values.
Standard fields are omitted from the wire if they hold the default value.
Optional fields track explicit presence. They are serialized even if set to 0.
In Proto3, fields set to their default value (0, empty string, false) are not serialized at all. This makes the wire format compact, but it means you cannot distinguish between "set to 0" and "not set."
The optional keyword (and oneof) reintroduces Explicit Presence. Once explicitly set, these fields are written to the wire even if their value is the default, allowing for has_field() checks.
Note on Editions: With the introduction of Protobuf Editions, the strict boundaries between Proto2 and Proto3 behavior have been removed. You can now explicitly configure whether fields use implicit or explicit presence via features like features.field_presence = EXPLICIT;, giving you granular control over serialization size versus field state tracking.
When multiple fields are sent together, Protobuf concatenates them into one binary stream. The decoder does not need separators between fields; each field tells the decoder how many bytes to consume before moving on.
- 01
Read the tag
The first varint in each field is the tag. Its lower three bits identify the wire type, and the remaining bits identify the field number.
- 02
Choose the payload rule
The wire type tells the decoder how to find the payload boundary: varint continuation bits, a fixed 4 or 8 byte width, or a length-delimited size prefix.
- 03
Consume that field
Once the payload length is known, the decoder consumes exactly those bytes, maps them to the schema field when possible, and advances its cursor.
- 04
Repeat until EOF
The stream has no outer field count. Parsing continues from the next byte and stops only when there are no bytes left to process.
Try modifying the JSON data below or clicking the example buttons to see how the binary stream changes in real-time. Click any segment in the encoded stream to inspect how its tag, length, and payload were parsed.