The schema is the contract; the wire format is how data that honors it travels. This section digs into the raw bytes: the physical layer of the specification.
This guide should help explain how the Protobuf encoding works, but if you have questions about edge cases or specifics that we don't cover, refer to the official encoding guide.
Protobuf's encoding leans on three bit-level concepts: byte order, bit shifting, and bitwise OR. If you can already read x << 3 | y, skip ahead. If not, here is the short version.
Varints are the fundamental building block of Protobuf efficiency, allowing integers to occupy only as many bytes as necessary.
Standard integers in memory take 4 or 8 bytes regardless of their value. Varints use Base-128 Serialization to represent smaller numbers with fewer bytes.
Each byte in a varint, except the last byte, has the most significant bit (MSB) set to 1. This acts as a continuation flag, telling the decoder "more bytes are coming."
The lower 7 bits of each byte store the data in groups of 7, least significant group first. This means Protobuf uses a Little-Endian approach even at the bit-group level.
Chunk Data
Split the number into 7-bit groups. Standard bytes are 8 bits, but we reserve the top bit (MSB) as a "continuation bit".
Reverse & Add MSB Flag
The groups are written in Little-Endian order (least significant group first). Set the MSB to 1 for all bytes except the last one.
Standard Varints are great for positive numbers, but they are highly inefficient for negative ones. ZigZag encoding fixes this.
Standard Varint (Two's Complement)
10 Byte(s)As a plain varint, -1 takes all 10 bytes: a negative number has no leading zeros to drop.
ZigZag Varint
1 Byte(s)After ZigZag, size depends on the number's magnitude rather than its sign: -1 encodes in a single byte, and the byte count grows only as the value moves away from zero.
Two's complement stores the sign in the most significant bit, which means every negative number has its high bits set. That is bad news for varints, which save space by dropping leading zeros: a negative number has none to drop. Even -1 takes the full 10 bytes of a 64-bit varint.
ZigZag encoding moves the sign to the least significant bit instead. Positive numbers map to even integers (n << 1) and negative numbers to odd ones.
With the sign at the bottom, small negative numbers have leading zeros again and encode as compactly as small positive ones.
The name comes from how the mapping alternates between positive and negative as the encoded value counts up: 0 maps to 0, -1 to 1, 1 to 2, -2 to 3, 2 to 4, and so on.
In a schema, ZigZag is what sint32 and sint64 select. The Types reference covers when to reach for those over plain int32 and int64.
Every field in a Protobuf message is prefixed by a Tag. This tag is the only reason the decoder knows which field it's currently processing and how to interpret the bytes that follow.
Tag Composition
A tag is a single Varint that combines two pieces of information:
- Field Number (bits 3 through N)
- Wire Type (the bottom 3 bits)
The formula for the tag value is (field_number << 3) | wire_type. Small field numbers fit in a single byte; larger ones (16 and above) spill into additional bytes under the same continuation rule varints always follow.
A tag packs two facts into a single number: the field number and the wire type. The lowest three bits hold the wire type, and everything above them holds the field number.
One number, two parts
Encoded as a varint
Every field on the wire is wrapped in an "envelope" that tells the decoder two things: which field number it is, and how to read the payload. These are packed into a single tag value, encoded as a varint.
| Wire type | Protobuf types |
|---|---|
| 0 (Varint) | |
| 1 (I64) | |
| 2 (LEN) | also used for packed repeated scalars |
| 3 (SGROUP) | group start (deprecated) |
| 4 (EGROUP) | group end (deprecated) |
| 5 (I32) |
Varint Fields (Wire Type 0)
Most numeric types use Wire Type 0. The length is implicit because the decoder reads bytes one by one until it finds a byte where the MSB is 0. That's the same continuation-bit rule from the varint section earlier, doing the delimiting on its own.
Fixed-Size Fields (Wire Type 1, 5)
Some types have a known size, so nothing needs to be measured. Wire Type 1 is always 8 bytes, used by double and fixed64. Wire Type 5 is always 4 bytes, used by float and fixed32. The decoder reads that many bytes after the tag.
Length-Delimited (Wire Type 2)
string, bytes, and nested message fields use Length-Delimited encoding. These fields include an explicit length prefix (encoded as a varint) immediately after the tag, telling the decoder exactly how many subsequent bytes belong to this field.
Packed Repeated Fields
Repeated fields of primitive types use a specialized encoding to avoid repeating the field tag for every element. Edition 2024 packs them by default; set features.repeated_field_encoding to opt a file, message, or field back out.
Each element repeats the field tag. High overhead for many small elements.
Elements are concatenated into a single length-delimited record. One tag for the whole set.
Maps are not a native wire-level primitive. Instead, they are syntactic sugar for a repeated message.
map<string, int32> items = 1;
message Entry { string key = 1; int32 value = 2; } repeated Entry items = 1;
Whether a field holding its zero value reaches the wire at all is a per-field choice.
Omitted from the wire whenever it holds the zero value, so a reader cannot tell this from never set.
Edition 2024's default. Once set, the field is written even when the value is zero.
A field with implicit presence is not serialized at all when it holds its zero value (0, empty string, false). That is the format's biggest space saver, but the distinction does not survive the trip: a reader cannot tell "set to 0" from "never set."
Explicit presence keeps that distinction. Once set, the field is written even when its value is the zero value, costing a tag and a payload but letting generated code offer a has_field() check. Message fields and members of a oneof always have explicit presence.
Presence is a schema feature. Edition 2024 gives every field EXPLICIT presence by default; set features.field_presence = IMPLICIT; on a file, message, or field to trade the distinction back for bytes.
Older syntaxes: Before editions, presence was fixed by the syntax you picked. Proto2 fields were explicit; proto3 fields were implicit unless marked optional. Editions replace that split with a feature you set per field.
When multiple fields are sent together, Protobuf concatenates them into one binary stream. The decoder does not need separators between fields; each field tells the decoder how many bytes to consume before moving on.
- 01
Read the tag
The first varint in each field is the tag. Its lower three bits identify the wire type, and the remaining bits identify the field number.
- 02
Choose the payload rule
The wire type tells the decoder how to find the payload boundary: varint continuation bits, a fixed 4 or 8 byte width, or a length-delimited size prefix.
- 03
Consume that field
Once the payload length is known, the decoder consumes exactly those bytes, maps them to the schema field when possible, and advances its cursor.
- 04
Repeat until EOF
The stream has no outer field count. Parsing continues from the next byte and stops only when there are no bytes left to process.
Try modifying the JSON data below or clicking the example buttons to see how the binary stream changes in real-time. Click any segment in the encoded stream to inspect how its tag, length, and payload were parsed.
Next
Community
Where Protobuf came from, who runs it at scale, and where to ask questions.