protobuf.com

Binary

Binary 101

Protobuf's encoding uses three bit-level operations: byte order, bit shifting, and bitwise OR. If you can already read x << 3 | y, skip ahead. If not, here's the short version.

Endianness

Multi-byte numbers can be laid out least significant byte first (little-endian) or most significant byte first (big-endian). Protobuf's fixed-width types (fixed32, fixed64, float, double) are little-endian. That's why the low bytes of a number show up first in the raw stream.

00000011111010001110100000000011

Bit Shifting

A left shift (<<) moves every bit left and fills the gap with zeroes. Shifting left by 3 multiplies by 8 and leaves 3 empty bits on the right. Protobuf uses this to build the field tag. It shifts the field number left by 3 to make room for the wire type.

0000000100000001000

Bitwise OR

OR (|) merges two numbers. An output bit is 1 if either input bit is 1. After a shift clears space, OR writes another value into it. As long as the two values occupy non-overlapping bit ranges, neither one corrupts the other.

000110000000101000011010

Further Reading

  • Endianness

    Little- and big-endian byte order, and why it matters when bytes move between machines.

  • Bit shifts

    The shift operations behind the tag formula, including the arithmetic/logical distinction this page glosses over.

  • Bitwise OR

    How OR merges bit patterns, along with the AND and XOR operations decoders also use.

Next

Base-128 Varints

How Protobuf base-128 varints encode integers in as few bytes as possible: continuation bits, byte layout, and an interactive varint encoder.

Previous
Binary
Next
Base-128 Varints