Basics | Protobuf Explained

Messages

Messages are the primary logical structure in Protobuf. They act as containers for your data, analogous to a struct in C/Rust or a class in Java/TypeScript.

A message is a strictly-enforced contract. Once defined in a schema, the Protobuf compiler ensures that every system interacting with this data, regardless of the programming language, agrees on its structure.

One of the best features of Protobuf is that it's designed to be evolvable. You can add new fields to messages without breaking existing code, which lets servers and clients upgrade at their own pace. Many other binary formats don't support this level of compatibility out of the box.

// A simple message definition
message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 results_per_page = 3;
}

Fields

Every field in a message requires a specific type (e.g., string, int32, bool) and a name. The full set of scalar and well-known types is covered on the Types page.

Since Protobuf is strictly typed, it catches many of the data-type errors that would otherwise only show up at runtime with formats like JSON. If a client expects an integer, they will never accidentally receive a string.

While names are used in your code for readability, they are ignored entirely in the binary encoding. This allows you to rename fields in your schema without breaking binary compatibility (though it may break JSON consumers).

message User {
  string username = 1;
  bool is_active = 2;
  uint32 login_count = 3;
}

Field Numbers

Field numbers are the most critical part of a Protobuf message. Instead of sending long string names (like "username") over the wire, Protobuf identifies each field by this integer ID.

On the wire, the field number is packed together with the field's wire type into a single value called the tag (the Binary section shows exactly how). Because these numbers identify fields, they must never be changed once a message type is in use. Reusing a number for a different field will cause silent data corruption.

Optimization Tip: Numbers 1 through 15 take 1 byte to encode (including the field number and wire type). Numbers 16 through 2047 take 2 bytes. Use 1-15 for your most frequently sent fields.

message User {
  // the number, not the name, identifies
  // this field on the wire
  string id = 1;

  // Small numbers (1-15) take 1 byte to encode
  string name = 2;
}

Enums

Enums allow you to define a restricted set of named constants. This is useful for states, roles, or configurations.

In proto3, the first constant must always map to zero. This serves as the default value when the field is not explicitly set in the binary payload.

Naming Convention: To avoid name collisions in languages like C++ or Go (where enum values are often in the parent scope), it is a best practice to prefix values with the enum name. Conventions like this are codified in Buf's style guide and enforced automatically by buf lint.

Open Enums: Modern Protobuf implementations support "open" enums, meaning if a server sends a value that a client doesn't recognize, the client will still preserve that value instead of crashing.

enum Status {
  // Prefixing avoids collisions
  STATUS_UNSPECIFIED = 0;
  STATUS_ACTIVE = 1;
  STATUS_DEFERRED = 2;
}

message User {
  Status current_status = 1;
}

Packages

As your project grows, you'll likely have many messages with similar names. Protobuf uses package declarations to prevent name clashes.

These packages often map directly to namespaces in C++, packages in Go/Java, or modules in TypeScript. They keep large-scale schemas organized, ensuring that an Account in the billing service doesn't conflict with an Account in the identity service.

How packages, file paths, and directory layout relate is covered in depth in Buf's files and packages reference.

syntax = "proto3";

// Defines the namespace
package demo.identity.v1;

message Account {
  string id = 1;
}

Composition

Protobuf supports complex, hierarchical data structures. You can define messages within other messages, or use previously defined messages as field types.

This Composition allows you to build reusable data models shared across services. For example, a Location message can be used across User, Event, and Office messages.

On the wire, embedded messages are "length-delimited", allowing decoders to skip the entire sub-message if they don't have the schema for it.

message Result {
  string url = 1;
  string title = 2;
}

message SearchResponse {
  // Result is embedded here
  Result top_result = 1;
}

Collections

To represent an array or list of items, use the repeated keyword. These fields can contain zero or more elements of the specified type.

In modern Protobuf, repeated scalar numeric fields (like int32, float, etc.) are "packed" by default. Instead of repeating the field tag for every element, they are stored as one single block with a length prefix. This is significantly more efficient for large arrays.

message SearchResponse {
  // A list of strings
  repeated string related_queries = 1;
  
  // A list of messages
  repeated Result results = 2;
}

Maps

Protobuf provides native support for associative maps (dictionaries). However, there are strict rules for map keys and values:

Keys: Can be any integral or string type. Messages, enums, floats, and bytes cannot be keys.
Values: Can be any type, including another message, but cannot be another map or a repeated field.

Behind the scenes, maps are actually just repeated messages with key and value fields, ensuring backward compatibility with older decoders.

message Project {
  string name = 1;

  // String keys to string values
  map<string, string> labels = 2;

  // Integer keys work too
  map<uint32, string> port_names = 3;

  // Values can be full messages
  map<string, Contributor> contributors = 4;
}

message Contributor {
  string name = 1;
  uint32 commit_count = 2;
}

Oneof

If you have a message with multiple fields where only one can be set at a time, you can enforce this behavior and save memory using the oneof keyword.

Setting any field within the oneof automatically clears all other fields in that same oneof. This is Protobuf's equivalent to a tagged union or variant.

A common use is polymorphism: an Event that could be a ClickEvent, HoverEvent, or ScrollEvent.

message ErrorStatus {
  string message = 1;
  
  oneof details {
    string stack_trace = 2;
    int32 error_code = 3;
  }
}

Generating Code

In practice, virtually nobody hand-writes serializers. Teams generate code from .proto files. The generated code provides typed message constructors, binary serialization, JSON mapping, and service bindings depending on the plugin.

proto/intro/v1/user.proto

edition = "2023";

package intro.v1;

message User {
  int64 id = 1;
  string name = 2;
  uint32 age = 3;
  float height_cm = 4;
  double weight_kg = 5;
  bool verified = 6;
}

From Contract to Runtime API

This schema defines a User message with six fields. Each field has a type, a generated-code name, and a stable field number used by the binary format.

Once code is generated from this schema, you can:

Instantiate: Create User objects in your language with type checking and editor support.
Serialize: Convert objects into compact binary buffers for transmission or storage.
Validate: Reject data that doesn't match the schema's structure before it reaches application logic.

Generating Code with buf

This guide uses the Buf CLI, which keeps generation declarative with a buf.gen.yaml file: check it into your repo and everyone on the team generates the same output with one command. The remote plugins in this example run on the Buf Schema Registry, so there are no plugin binaries to install locally.

buf.gen.yaml

version: v2
plugins:
  - remote: buf.build/bufbuild/es
    out: web/src/gen
    opt: target=ts
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    opt: paths=source_relative

GENERATE CODE

$ buf generate

Generating with protoc

protoc is the reference compiler for Protobuf. Plugins (like protoc-gen-es) are installed as executables on your system's PATH and selected with command-line flags.

Install plugins

$ npm install --save-dev @bufbuild/protoc-gen-es
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

Generate code

$ protoc --es_out=web/src/gen --es_opt=target=ts \
    --go_out=gen/go --go_opt=paths=source_relative \
    proto/intro/v1/user.proto

Using the Generated Code

With code generated by protobuf-es, the schema becomes a native TypeScript API.

src/main.ts

import { create, toBinary, toJsonString } from "@bufbuild/protobuf";
import { UserSchema } from "./gen/intro/v1/user_pb";

const user = create(UserSchema, {
  id: 1042n,
  name: "cyber_ninja",
  age: 28,
  verified: true,
});

const bytes = toBinary(UserSchema, user);
const json = toJsonString(UserSchema, user);

console.log("JSON Output:", json);
console.log("Binary Output:", bytes);

Different languages and runtimes

The same schema-first workflow applies across supported languages, but import paths, package names, generated types, and runtime APIs differ by ecosystem.

Getting Started Tutorials:

Go Python Java Kotlin C++C#Dart TypeScript

Home

Types

Build

Messages

Fields

Types

Field Numbers

Enums

Packages

Composition

Collections

Maps

Oneof

Generating Code

Anatomy of a .proto file

Messages

Fields

Field Numbers

Enums

Packages

Composition

Collections

Maps

Oneof

Generating Code

From Contract to Runtime API

Generating Code with buf

Generating with protoc

Using the Generated Code