Protobuf offers advanced features for modeling richer APIs and evolving schemas safely over time. Descriptor-based extension, reflection, and validation workflows now have dedicated pages.
Anatomy of an evolving schema
This diagram breaks down the features this page covers. Each label links to the section that covers it in detail.
edition = "2024";
package shop.v1;
import "google/protobuf/any.proto";
import "google/protobuf/struct.proto";
option features.enum_type = CLOSED;
message Order {
reserved 2, "legacy_status";
string id = 1;
string note = 3 [deprecated = true];
google.protobuf.Any payload = 4;
google.protobuf.Value metadata = 5;
}
message OrderRequest {
string order_id = 1;
}
service Orders {
rpc GetOrder(OrderRequest) returns (Order);
}
The import statement brings definitions from another .proto file into scope. Paths resolve from your module root rather than relative to the file doing the importing. Always write the full path from that root, so the same import string refers to the same file no matter which .proto it appears in.
Often the definitions you want already live in another project. You can vendor them: copy that team's .proto files into your repository, then copy them again whenever they change. Nothing tracks which version you copied.
A schema registry makes it a declared dependency instead. The Buf Schema Registry pins each one to an exact version in a lockfile, so every checkout resolves imports identically. It's the model npm and Cargo use, applied to schemas. The buf.yaml tab in the example declares buf.build/bufbuild/protovalidate as a dependency, which is what lets service.proto import buf/validate/validate.proto.
protoc include paths (-I / --proto_path)
The protoc compiler requires manually specifying every include directory via -I (or --proto_path) flags. If import paths are inconsistent across your project (e.g. import "proto/user.proto" vs import "user.proto"), protoc treats them as different types.
edition = "2024"; package auth.v1; import "buf/validate/validate.proto"; import "common/v1/user.proto"; import "google/type/datetime.proto"; message LoginRequest { string email = 1 [(buf.validate.field).string.email = true]; } message LoginResponse { common.v1.User user = 1; string session_token = 2; google.type.DateTime expires_at = 3; }
buf dep update buf build
The Any type allows you to include messages where the schema isn't known at compile time.
google.protobuf.Any embeds an arbitrary serialized Protobuf message along with a URL that identifies its type (e.g., type.googleapis.com/mypackage.MyMessage).
When serialized to ProtoJSON, this type identifier is rendered as a special @type property alongside the standard JSON fields of the embedded message, allowing parsers to route the payload correctly.
import "google/protobuf/any.proto"; message Event { google.protobuf.Any payload = 1; }
{
"payload": {
"@type": "type.googleapis.com/demo.User",
"name": "Hiro"
}
}If you are working with dynamic Protobuf messages, use Any. However, for arbitrary structured JSON data you don't want to model, or data that is completely dynamic (like a schema-less JSON object), use google.protobuf.Value or google.protobuf.Struct.
A Value represents a dynamically typed value which can be either a null, a number, a string, a boolean, a recursive struct (object), or a list of values. It maps directly to any valid JSON structure.
Use this sparingly, as it defeats the purpose of Protobuf's strong typing, but it's useful for integrating with schemaless NoSQL databases or passing untyped metadata blocks.
Any, Value, and Struct are Well-Known Types. The Types reference lists the rest, including Timestamp and Duration.
import "google/protobuf/struct.proto"; message Event { // Any arbitrary JSON value google.protobuf.Value metadata = 1; // Specifically a JSON object google.protobuf.Struct custom_attributes = 2; }
{
"metadata": "simple string or object",
"custom_attributes": {
"dynamic_key": [1, 2, 3],
"enabled": true
}
}The service keyword is used to define RPC (Remote Procedure Call) interfaces. Frameworks like gRPC or ConnectRPC use these definitions to generate client and server code.
Services support four types of communication:
- Unary: Simple request-response.
- Streaming: Send or receive sequences of messages in a single call (Client, Server, or Bidirectional).
Note: While Protobuf provides the language to define these interfaces, the underlying networking protocols and implementation frameworks (like gRPC or ConnectRPC) are a broad topic and are out of scope for this guide.
service UserService { // Unary: One request, one response rpc GetUser(GetUserRequest) returns (User); // Server Stream: One request, many responses rpc ListUsers(ListUsersRequest) returns (stream User); // Bidirectional Stream: Real-time chat rpc Chat(stream Message) returns (stream Message); }
Protobuf options control how code is generated and how data is mapped. They are categorized by scope: File, Message, Field, or Service. The full set of standard options is defined in descriptor.proto.
option go_package: Defines the Go import path.option java_package: Defines the Java package.option optimize_for = SPEED;: Generates highly optimized (but larger) code. Alternatives:CODE_SIZE,LITE_RUNTIME.[deprecated = true]: Marks a field as deprecated.[json_name = "custom"]: Sets a custom JSON key.
edition = "2024"; option go_package = "github.com/example/v1"; option java_package = "com.example.v1"; option optimize_for = SPEED; message User { string user_id = 1 [json_name = "uid"]; string old_field = 2 [deprecated = true]; }
{
"uid": "01H8XGJWBWBAQ4",
"oldField": "still serialized"
}What happens when a parser meets a field number its schema doesn't define? Nothing dramatic. The wire format tells it how many bytes the value occupies, so it reads them, keeps them as an unknown field, and moves on. When the message is serialized again, those bytes are written back out.
This round-tripping is the machinery behind Protobuf's compatibility guarantees. An old client can accept a message from a new server, touch the fields it knows about, and pass the message along without stripping the fields added since it was compiled. The same goes for proxies and queues sitting between services that upgrade on different schedules.
Be careful with this at the edge of your system, though. A payload from outside can carry unknown fields your validation never inspected, and preservation will happily ferry them to internal services that do understand them. For this reason, many teams that expose Protobuf APIs to the outside world drop unknown fields at the boundary as a precaution; most runtimes provide a discard option when parsing.
message User {
string id = 1;
string email = 2;
}
{ "id": "u_1", "email": "a@b.c" }message User {
string id = 1;
}
id = "u_1"Re-serializing writes those bytes back out unchanged, so a field the reader has never heard of survives the round trip.
This is one of the superpowers of schema-driven APIs: because the contract is written down, you can detect a change that would break clients before it ships. With a schemaless JSON API, you find out in production. With Protobuf, a CI check like buf breaking compares your schema against the previous version and flags the exact line.
Not every change breaks in the same way. Renaming a field, for example, is invisible on the binary wire but breaks JSON clients. Breaking changes fall into four categories:
- WIRE: The most severe level. This includes changing a field number or using an incompatible type (e.g.,
stringtoint32). This causes data corruption when old and new endpoints decode each other's messages; you should never do this. - WIRE_JSON: Breakage in JSON representation. Renaming a field is safe on the binary wire, but clients expecting the old JSON key will fail. You can mitigate this using the
[json_name="old_name"]annotation. - PACKAGE: Source code breakage at the package level. Changing a type in a wire-compatible way (e.g.,
int32toint64) transmits safely, but when developers update their generated code, their builds will fail until they update their types. - FILE: The strictest level. This ensures source code compatibility down to the individual file level. Moving a message to another file might break code generation that relies on specific file imports.
edition = "2024"; package api.v1; message User { string id = 1; int32 age = 2; string display_name = 3; }
edition = "2024"; package api.v1; message User { // [WIRE] breakage: type changed from string int32 id = 1; // [PACKAGE] breakage: source code type change int64 age = 2; // [WIRE_JSON] breakage: JSON key changed string full_name = 3; }
Protobuf Editions unifies proto2 and proto3, allowing features to be toggled individually rather than through major syntax version upgrades.
Editions allows for smooth migrations and fine-grained control over behaviors:
- Field Presence: whether a reader can tell a field that was never set apart from one set to its default value. Choose
IMPLICIT(the proto3 behavior) orEXPLICIT(the proto2 behavior); the Field Presence section below covers the tradeoff. - Enum Type: where a value the enum does not declare ends up.
OPENkeeps it in the field;CLOSEDmoves it to the unknown fields. - Repeated Encoding: Standardize on
PACKED(for efficiency) orEXPANDED(for compatibility).
That granularity is the whole point. Under proto2 and proto3, these behaviors were welded to the single keyword at the top of the file, so migration was all or nothing: moving a file to proto3 opened its closed enums, dropped its custom field defaults, and removed explicit presence from its singular scalars, all at once.
Editions makes each of those a separate feature with a scope of its own, set per file and then narrowed on a message, an enum, or a single field, so a schema can adopt one new behavior without taking the others. New behavior then ships as a new feature with a per-edition default, rather than as a "proto4" that would force the same all-or-nothing migration again.
edition = "2024"; package demo.v1; // Set for the whole file... option features.enum_type = CLOSED; enum Unit { UNIT_UNSPECIFIED = 0; UNIT_CELSIUS = 1; } enum Status { // ...and overridden on one enum. option features.enum_type = OPEN; STATUS_UNSPECIFIED = 0; STATUS_ACTIVE = 1; }
Presence is whether a reader can tell a field that was never set apart from one that was set to its zero value, like 0 or "". Edition 2024 gives every field explicit presence, so the two are different states. IMPLICIT collapses them into one: a zero stops being written, and a reader cannot tell it from absent.
These two schemas differ by one word. Both messages below have every field set to its zero value, which is the only point where the difference shows.
edition = "2024"; package demo.v1; option features.field_presence = EXPLICIT; message Settings { string label = 1; int32 retries = 2; bool verbose = 3; }
{
"label": "",
"retries": 0,
"verbose": false
}Every field tracks presence, so a zero is a value it holds and gets written out.
edition = "2024"; package demo.v1; option features.field_presence = IMPLICIT; message Settings { string label = 1; int32 retries = 2; bool verbose = 3; }
{}No field tracks presence, so every zero means the same thing as absent and there is nothing left to write.
The Evolution of Required
The required keyword was removed in proto3. This was a deliberate architectural decision to ensure that schemas could evolve safely without breaking backward compatibility.
Why was it removed?
If a field is marked required, it must be present in every message. If you later decide to stop sending that field, every older client in the world will crash when they try to decode the new message. Required fields are considered harmful for long-term schema evolution.
What to do instead
Application Validation: Use generated getters that return zero values if the field is missing (e.g., Go's
GetField()) and perform null checks in your business logic.Metadata Validation: Declare constraints in the schema as custom options and enforce them when a message is validated rather than when it is decoded, so the field stays wire compatible.
protovalidateis the usual implementation; since the mechanism is just options, you can define and check your own instead.
import "buf/validate/validate.proto"; message CreateUserRequest { // Required at the validation layer // but optional at the wire layer. string email = 1 [ (buf.validate.field).string.email = true, (buf.validate.field).required = true ]; }
// Safe access even if req is nil if req.GetEmail() == "" { return status.Error(InvalidArgument, "email is required") }
Maximum size
The absolute maximum size of a serialized Protobuf message is 2 GiB. This is a hard architectural limit because the protocol relies on 32-bit signed integers to encode byte lengths and offsets. If a payload exceeds this size, standard parsers will throw an overflow error and refuse to read it.
Practical sizes
Protobuf is optimized for small, fast payloads. The official recommendation is to keep messages under a few megabytes. In practice, the ideal size is typically under 1 MB.
Once a message grows beyond 10 MB, the CPU and memory costs of parsing become highly noticeable. For moving large datasets, the standard pattern is to chunk the data into a stream of smaller messages.
Whole-message parsing
Protobuf is fundamentally designed around the expectation that you will load the entire message into memory at once. When you deserialize a payload, the parser reads the entire binary stream and instantiates a complete object graph.
As with most serialization formats, the resulting in-memory representation is significantly larger than the serialized binary. Pointers, object overhead, and data structure padding can cause memory usage to be several times the size of the original payload.
Next
Descriptors
A schema compiled into Protobuf data. Descriptors are what reflection, dynamic decoding, and breaking-change detection read.