vtprotobuf

vtprotobuf is a protoc/buf compiler plug-in, protoc-gen-go-vtproto, built by PlanetScale for Vitess. It runs alongside the standard protoc-gen-go generator and emits additional helper methods on your existing protobuf-generated Go structs — SizeVT, MarshalVT, UnmarshalVT, EqualVT, CloneVT, and pool-based ResetVT/ReturnToVTPool — that perform the same work as the standard google.golang.org/protobuf API but without reflection or (in most cases) extra allocations.

Because it only adds opt-in methods rather than replacing the generated types or the wire format, teams can adopt it incrementally on hot paths — gRPC servers, high-throughput RPC codecs, memory-pooled message handling — while keeping full compatibility with the rest of the protobuf ecosystem, including protojson, reflection-based tooling, and existing generated code.

What You Get

• A protoc-gen-go-vtproto binary installable via go install and pluggable into any protoc or buf.gen.yaml pipeline alongside the standard Go protobuf generator
• Per-feature code generation flags (size, marshal, marshal_strict, unmarshal, unmarshal_unsafe, equal, clone, pool, grpc) so you only generate the helper methods you actually need
• A ready-made gRPC codec (codec/grpc) and dRPC codec (codec/drpc) that automatically use the generated MarshalVT/UnmarshalVT methods for RPC traffic
• Glob-based -pool / -pool-exclude / -ignoreUnknownFields rules to control memory-pooling and unknown-field behavior on a per-message basis
• An unmarshal_unsafe mode that aliases incoming byte slices instead of copying them, trading a lifetime constraint for materially fewer allocations
• A conformance test suite that runs the upstream protobuf conformance tests against vtprotobuf-generated code to validate wire-format correctness

Common Use Cases

• Speeding up marshal/unmarshal on gRPC or dRPC hot paths in high-QPS Go services without changing the wire protocol
• Reducing GC pressure in latency-sensitive systems by reusing protobuf messages through the generated pool methods
• Replacing reflection-based proto.Equal/proto.Clone calls with generated, allocation-free equivalents in comparison-heavy code
• Selectively enabling only the size+marshal+unmarshal features for specific message types via glob patterns, leaving the rest on the standard reflection-based path
• Adopting a Vitess-proven optimization inside another gRPC-heavy Go codebase (e.g. a distributed database or proxy) that already generates protobuf code with protoc or buf

Under The Hood

Architecture The plugin is organized around a small Feature registry: generator/generator.go resolves the -features flag into a list of Feature constructors (via generator/features.go), and each concrete feature package (features/size, features/marshal, features/unmarshal, features/equal, features/clone, features/pool, features/grpc) self-registers through a blank import in cmd/protoc-gen-go-vtproto/main.go. Generate() walks every requested protogen.File, opens one GeneratedFile per source file, and calls each enabled feature’s GenerateFile against a shared generatedfile.go wrapper that exposes helpers like p.P(), pool/ignore-unknown-fields glob lookups (generator.ObjectSet), and wrapper-type support. Because every feature package is implemented against that one shared wrapper and Config struct, a breaking change to GeneratedFile or the Feature interface would ripple through all seven feature packages simultaneously — the tradeoff for keeping each feature’s own logic orthogonal and independently testable.

Tech Stack The module targets Go 1.26 and builds directly on google.golang.org/protobuf’s compiler/protogen API for AST access and code emission, with google.golang.org/grpc pulled in for the bundled gRPC codec and conformance testing, and stretchr/testify for test assertions. There’s no runtime web or ORM layer — the deliverable is a single Go binary (protoc-gen-go-vtproto) invoked by protoc or buf generate; a Makefile and protobuf.sh script vendor and build a pinned protobuf C++ toolchain (21.12) in CI so the generator can be exercised end-to-end and its output diffed against checked-in fixtures.