Protobuf offers advanced features for modeling richer APIs and evolving schemas safely over time. Descriptor-based extension, reflection, and validation workflows now have dedicated pages.
Anatomy of an evolving schema
This diagram breaks down the features this page covers. Each label links to the section that covers it in detail.
You can use definitions from other .proto files using the import statement. Imports from another team or project are the awkward part: the traditional answer is to copy their files into your repo and re-copy them whenever they change.
The Buf Schema Registry treats them as versioned dependencies instead, the way NPM or Cargo would. You name a module under deps in buf.yaml, buf dep update pins exact versions in buf.lock, and the imports resolve identically for everyone who checks out the repo. The schema you are editing on this site does exactly that with buf.build/bufbuild/protovalidate.
Always import using fully qualified paths from your module root to avoid confusing "Duplicate Symbol" errors.
protoc include paths (-I / --proto_path)
The protoc compiler requires manually specifying every include directory via -I (or --proto_path) flags. If import paths are inconsistent across your project (e.g. import "proto/user.proto" vs import "user.proto"), protoc treats them as different types.
edition = "2023"; package auth.v1; import "buf/validate/validate.proto"; import "common/v1/user.proto"; import "google/type/datetime.proto"; message LoginRequest { string email = 1 [(buf.validate.field).string.email = true]; } message LoginResponse { common.v1.User user = 1; string session_token = 2; google.type.DateTime expires_at = 3; }
buf dep update buf build
The Any type allows you to include messages where the schema isn't known at compile time.
google.protobuf.Any embeds an arbitrary serialized Protobuf message along with a URL that identifies its type (e.g., type.googleapis.com/mypackage.MyMessage).
When serialized to ProtoJSON, this type identifier is rendered as a special @type property alongside the standard JSON fields of the embedded message, allowing parsers to route the payload correctly.
import "google/protobuf/any.proto"; message Event { google.protobuf.Any payload = 1; }
{
"payload": {
"@type": "type.googleapis.com/demo.User",
"name": "Hiro"
}
}If you are working with dynamic Protobuf messages, use Any. However, for arbitrary structured JSON data you don't want to model, or data that is completely dynamic (like a schema-less JSON object), use google.protobuf.Value or google.protobuf.Struct.
A Value represents a dynamically typed value which can be either a null, a number, a string, a boolean, a recursive struct (object), or a list of values. It maps directly to any valid JSON structure.
Use this sparingly, as it defeats the purpose of Protobuf's strong typing, but it's useful for integrating with schemaless NoSQL databases or passing untyped metadata blocks.
import "google/protobuf/struct.proto"; message Event { // Any arbitrary JSON value google.protobuf.Value metadata = 1; // Specifically a JSON object google.protobuf.Struct custom_attributes = 2; }
{
"metadata": "simple string or object",
"custom_attributes": {
"dynamic_key": [1, 2, 3],
"enabled": true
}
}The service keyword is used to define RPC (Remote Procedure Call) interfaces. Frameworks like gRPC or ConnectRPC use these definitions to generate client and server code.
Services support four types of communication:
- Unary: Simple request-response.
- Streaming: Send or receive sequences of messages in a single call (Client, Server, or Bidirectional).
Note: While Protobuf provides the language to define these interfaces, the underlying networking protocols and implementation frameworks (like gRPC or ConnectRPC) are a broad topic and are out of scope for this guide.
service UserService { // Unary: One request, one response rpc GetUser(GetUserRequest) returns (User); // Server Stream: One request, many responses rpc ListUsers(ListUsersRequest) returns (stream User); // Bidirectional Stream: Real-time chat rpc Chat(stream Message) returns (stream Message); }
Protobuf options control how code is generated and how data is mapped. They are categorized by scope: File, Message, Field, or Service. The full set of standard options is defined in descriptor.proto.
option go_package: Defines the Go import path.option java_package: Defines the Java package.option optimize_for = SPEED;: Generates highly optimized (but larger) code. Alternatives:CODE_SIZE,LITE_RUNTIME.[deprecated = true]: Marks a field as deprecated.[json_name = "custom"]: Sets a custom JSON key.
edition = "2023"; option go_package = "github.com/example/v1"; option java_multiple_files = true; option optimize_for = SPEED; message User { string user_id = 1 [json_name = "uid"]; string old_field = 2 [deprecated = true]; }
What happens when a parser meets a field number its schema doesn't define? Nothing dramatic. The wire format tells it how many bytes the value occupies, so it reads them, keeps them as an unknown field, and moves on. When the message is serialized again, those bytes are written back out.
This round-tripping is the machinery behind Protobuf's compatibility guarantees. An old client can accept a message from a new server, touch the fields it knows about, and pass the message along without stripping the fields added since it was compiled. The same goes for proxies and queues sitting between services that upgrade on different schedules.
Be careful with this at the edge of your system, though. A payload from outside can carry unknown fields your validation never inspected, and preservation will happily ferry them to internal services that do understand them. For this reason, many teams that expose Protobuf APIs to the outside world drop unknown fields at the boundary as a precaution; most runtimes provide a discard option when parsing.
message User { string id = 1; string email = 2; // added in v2 }
message User { string id = 1; // No email field here. A parsed message // still carries field 2's bytes as an // unknown field, and re-serializing // writes them back out intact. }
This is one of the superpowers of schema-driven APIs: because the contract is written down, you can detect a change that would break clients before it ships. With a schemaless JSON API, you find out in production. With Protobuf, a CI check like buf breaking compares your schema against the previous version and flags the exact line.
Not every change breaks in the same way. Renaming a field, for example, is invisible on the binary wire but breaks JSON clients. Breaking changes fall into four categories:
- WIRE: The most severe level. This includes changing a field number or using an incompatible type (e.g.,
stringtoint32). This causes data corruption when old and new endpoints decode each other's messages; you should never do this. - WIRE_JSON: Breakage in JSON representation. Renaming a field is safe on the binary wire, but clients expecting the old JSON key will fail. You can mitigate this using the
[json_name="old_name"]annotation. - PACKAGE: Source code breakage at the package level. Changing a type in a wire-compatible way (e.g.,
int32toint64) transmits safely, but when developers update their generated code, their builds will fail until they update their types. - FILE: The strictest level. This ensures source code compatibility down to the individual file level. Moving a message to another file might break code generation that relies on specific file imports.
edition = "2023"; package api.v1; message User { string id = 1; int32 age = 2; string display_name = 3; }
edition = "2023"; package api.v1; message User { // [WIRE] breakage: type changed from string int32 id = 1; // [PACKAGE] breakage: source code type change int64 age = 2; // [WIRE_JSON] breakage: JSON key changed string full_name = 3; }
Protobuf Editions unifies proto2 and proto3, allowing features to be toggled individually rather than through major syntax version upgrades.
Editions allows for smooth migrations and fine-grained control over behaviors:
- Field Presence: whether a reader can tell a field that was never set apart from one set to its default value. Choose
IMPLICIT(the proto3 behavior) orEXPLICIT(the proto2 behavior); the Field Presence section below covers the tradeoff. - Enum Type:
OPENenums keep values added to the enum after your code was generated;CLOSEDenums treat them as invalid. - Repeated Encoding: Standardize on
PACKED(for efficiency) orEXPANDED(for compatibility).
That granularity is the whole point. Under proto2 and proto3, these behaviors were welded to the single keyword at the top of the file, so migration was all or nothing: moving a file to proto3 opened its closed enums, dropped its custom field defaults, and removed explicit presence from its singular scalars, all at once.
Editions makes each of those a separate feature you can set per file, message, or field, so a schema can adopt one new behavior without taking the others. New behavior then ships as a new feature with a per-edition default, rather than as a "proto4" that would force the same all-or-nothing migration again.
edition = "2023"; // Globally enforce field presence option features.field_presence = EXPLICIT; message User { // Optional fields are back string name = 1; // Mixed behavior in one file int32 age = 2 [features.enum_type = OPEN]; }
Implicit vs. Explicit
Field presence determines whether a receiver can distinguish between a field that was never set and one that was set to its default value (like 0 or ""). In short, implicit presence saves space by never sending default values, while explicit presence includes extra tracking to definitively tell you if a field was populated.
proto2 and proto3
In proto2, all fields were explicit. In proto3, the optional keyword was initially removed for scalar fields to simplify the wire format and generated code. This meant all scalars had implicit presence: if you didn't send a value, the receiver saw the default.
Presence Today
Due to widespread demand, the optional keyword was re-introduced in later versions of proto3 (v3.15+). Today, Protobuf Editions lets you globally or locally toggle field_presence between IMPLICIT and EXPLICIT.
File-Level Default
edition = "2023"; // Set EXPLICIT presence for the entire file option features.field_presence = EXPLICIT; message Profile { string bio = 1; // Explicit (tracked) int32 views = 2; // Explicit (tracked) }
Field-Level Overrides
message LegacyData { // Override to IMPLICIT for specific fields int32 raw_id = 1 [features.field_presence = IMPLICIT]; // Follows file-level default (EXPLICIT) string note = 2; }
The Evolution of Required
The required keyword was removed in proto3. This was a deliberate architectural decision to ensure that schemas could evolve safely without breaking backward compatibility.
Why was it removed?
If a field is marked required, it must be present in every message. If you later decide to stop sending that field, every older client in the world will crash when they try to decode the new message. Required fields are considered harmful for long-term schema evolution.
Modern Best Practices
Application Validation: Use generated getters that return zero values if the field is missing (e.g., Go's
GetField()) and perform null checks in your business logic.Metadata Validation: Use
protovalidateto declare constraints (includingrequired) in the IDL without breaking wire compatibility.
import "buf/validate/validate.proto"; message CreateUserRequest { // Required at the validation layer // but optional at the wire layer. string email = 1 [ (buf.validate.field).string.email = true, (buf.validate.field).required = true ]; }
// Safe access even if req is nil email := req.GetEmail() if email == "" { return status.Error(InvalidArgument, "email is required") }
Maximum size
The absolute maximum size of a serialized Protobuf message is 2 GiB. This is a hard architectural limit because the protocol relies on 32-bit signed integers to encode byte lengths and offsets. If a payload exceeds this size, standard parsers will throw an overflow error and refuse to read it.
Practical sizes
Protobuf is optimized for small, fast payloads. The official recommendation is to keep messages under a few megabytes. In practice, the ideal size is typically under 1 MB.
Once a message grows beyond 10 MB, the CPU and memory costs of parsing become highly noticeable. For moving large datasets, the standard pattern is to chunk the data into a stream of smaller messages.
Whole-message parsing
Protobuf is fundamentally designed around the expectation that you will load the entire message into memory at once. When you deserialize a payload, the parser reads the entire binary stream and instantiates a complete object graph.
As with most serialization formats, the resulting in-memory representation is significantly larger than the serialized binary. Pointers, object overhead, and data structure padding can cause memory usage to be several times the size of the original payload.