Skip to content
Tutorials Center
My learning

Topics

Lesson 112 of 147 ยท Web and Networking

RPC, gRPC and Protocol Buffers

Learn remote procedure calls in Go with net/rpc and JSON-RPC, see how protobuf encodes data on the wire, and get a practical tour of gRPC, streaming, Connect and REST versus gRPC.

When two programs need to cooperate, the most natural way to express it in code is a function call: reply, err := service.Greet(name). A remote procedure call (RPC) makes that call travel over the network. The client has a local object with methods, the framework turns each call into bytes, the server runs the real function and the result travels back. The idea is old, and it is the heart of most service-to-service communication in modern backends.

Go supports RPC in three layers. The standard library has net/rpc (simple, Go-only, and frozen) and net/rpc/jsonrpc. The industry standard is gRPC, which uses Protocol Buffers (protobuf) to describe services and encode messages. A third option, Connect, speaks gRPC but also works with plain HTTP and browsers. In this lesson you will run net/rpc and JSON-RPC on loopback, build the client-stub idea by hand, and decode a protobuf message byte by byte. gRPC itself needs third-party modules and code generation, so those parts are shown as read-only code.

net/rpc: calling methods across a network

net/rpc exposes the methods of a Go value to remote callers. A method is available if it follows this exact shape:

Go
func (t *T) MethodName(args ArgType, reply *ReplyType) error

The method and its type must be exported, both parameters must be exported (or built-in) types, and the second parameter must be a pointer, because it is where the result goes. A non-nil error is sent back to the caller as the failure of the call. Data travels in Go's encoding/gob format by default, so both sides must be written in Go.

Go A net/rpc service and client

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

client.Call is synchronous: it blocks until the reply arrives. client.Go starts the call and returns a *rpc.Call whose Done channel receives the finished call, which lets you run many calls in parallel over one connection. An error from the server arrives as an rpc.ServerError, which is just a string type: the original error type is lost on the way, which is one of the limitations of a framework that only has strings to work with. Unknown methods and transport failures such as a closed connection are reported through the same error result, so always check it.

The net/rpc package is documented as frozen. It works and is stable, but it receives no new features, and it has no support for contexts, deadlines or streaming. Use it for small Go-to-Go tools and learning; for anything that crosses language or team boundaries, use gRPC or another schema-based protocol.

JSON-RPC: a language-neutral wire format

net/rpc/jsonrpc keeps the same server code but replaces gob with JSON, using the (version 1.0) JSON-RPC message format. The service from the previous section works unchanged: you only wrap each connection in a different codec. Because the wire format is plain JSON, clients in any language can talk to it, and you can even debug it by typing into a socket.

Go Talking JSON-RPC by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The request names the method, carries its parameters as a one-element array and has an id that the response repeats, so a client can match replies to calls even when several are in flight. The response holds either a result or an error. A newer JSON-RPC 2.0 exists (with a "jsonrpc": "2.0" field and error codes), but the standard library implements the 1.0 shape.

What an RPC framework really does

Strip away the details and every RPC system, including gRPC, consists of the same parts. Seeing them as ordinary Go code makes the generated code of gRPC much less mysterious:

  1. A service interface: the contract, as methods with typed arguments and results.
  2. A server adapter that decodes a request, calls the real implementation and encodes the answer.
  3. A client stub: an object that implements the same interface, but whose methods send a request and wait for a reply.

If the client stub implements the interface, calling code does not know whether it talks to a local object or a remote one. That is the point (and also the danger: a remote call can fail, time out or be slow, so the interface should include context.Context and an error). The program below builds all three parts on top of net/rpc, and adds what net/rpc itself lacks: a per-call deadline.

Go A client stub that implements the same interface

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The same welcome function works with both implementations. The remote stub fails in ways the local object cannot: the last call asks for a name that makes the server sleep for 300 ms, so after the 100 ms deadline the stub returns context deadline exceeded. Notice what that error does not tell you: whether the server finished the work anyway. That uncertainty is fundamental to remote calls, and it is why operations that may be retried should be idempotent. gRPC improves on this by sending the deadline to the server so it can stop early, but it cannot remove the uncertainty.

Protocol Buffers: the schema and the bytes

gRPC does not send gob or JSON. It sends Protocol Buffers, a binary format driven by a schema file. You write the contract once in a .proto file, and a code generator produces the Go types and the client and server code (and the same for Java, Python, C++, Rust and many more).

TEXT
syntax = "proto3";

package greeter.v1;
option go_package = "example.com/app/gen/greeterv1;greeterv1";

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
  repeated string tags = 2;
}

service Greeter {
  rpc SayHello(HelloRequest) returns (HelloReply);
  rpc WatchGreetings(HelloRequest) returns (stream HelloReply);
}

The numbers (= 1, = 2) are field numbers. They, not the names, identify fields on the wire. This is the reason protobuf messages are small and why schemas can evolve: you may add new fields at any time (old readers skip them), and you must never change the number or type of an existing field or reuse a deleted number (use reserved to retire it).

You do not need a library to understand the wire format; it is simple enough to write in a few lines. Each field starts with a tag, (field_number << 3) | wire_type, followed by the value. Wire type 0 is a varint (a variable-length integer: seven bits per byte, the high bit says "more bytes follow"), and wire type 2 is length-delimited (a varint length, then that many bytes), used for strings, bytes, nested messages and repeated packed data.

Go Encoding and decoding protobuf by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Read the hex dump: 0a 03 41 64 61 is field 1 (0a = field 1, wire type 2), length 3, then the bytes of "Ada". 10 96 01 is field 2 (10 = field 2, wire type 0) with the value 150, which needs two varint bytes: 96 01 (low seven bits first: 0x16 + (1 << 7) = 150). The emails are two more length-delimited fields with tag 1a. No field names, no quotes, no braces: that is why the binary form is much smaller than the JSON, and why a parser can be so fast.

The decoder never looks at names, only numbers, so it silently skips the two unknown fields, which is exactly how protobuf achieves forward compatibility. Real protobuf has more details (zigzag encoding for negative sint values, packed repeated numbers, default values that are not sent at all, a 10-byte encoding for negative int32), but this is the core idea. In real code you never write this by hand; the generated types provide proto.Marshal and proto.Unmarshal.

gRPC in Go

gRPC combines protobuf with HTTP/2. HTTP/2 supplies multiplexed streams over one connection, header compression and flow control, so gRPC gets request/response calls and streaming nearly for free. The workflow is always the same:

  1. Write the .proto file.
  2. Generate Go code with protoc and the plugins protoc-gen-go and protoc-gen-go-grpc, or with the buf tool (buf generate). The output contains the message structs, a GreeterServer interface, a GreeterClient interface and a registration function.
  3. Implement the server interface and register it.
  4. Create a client connection and call methods on the generated client.

A server for the Greeter service defined above looks like this (read-only: it needs google.golang.org/grpc):

Go
type server struct {
	pb.UnimplementedGreeterServer // lets you add RPCs to the .proto without breaking builds
}

func (s *server) SayHello(ctx context.Context, req *pb.HelloRequest) (*pb.HelloReply, error) {
	if req.GetName() == "" {
		return nil, status.Error(codes.InvalidArgument, "name is required")
	}
	return &pb.HelloReply{Message: "Hello, " + req.GetName()}, nil
}

func main() {
	lis, err := net.Listen("tcp", ":50051")
	if err != nil {
		log.Fatal(err)
	}
	s := grpc.NewServer()
	pb.RegisterGreeterServer(s, &server{})
	log.Fatal(s.Serve(lis))
}

and the client:

Go
conn, err := grpc.NewClient("localhost:50051",
	grpc.WithTransportCredentials(insecure.NewCredentials())) // use real TLS in production
if err != nil {
	log.Fatal(err)
}
defer conn.Close()

client := pb.NewGreeterClient(conn)

ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()

reply, err := client.SayHello(ctx, &pb.HelloRequest{Name: "Ada"})
if err != nil {
	if st, ok := status.FromError(err); ok && st.Code() == codes.InvalidArgument {
		log.Println("bad request:", st.Message())
	}
	log.Fatal(err)
}
fmt.Println(reply.GetMessage())

Compare with the hand-built stub from earlier: the generated GreeterClient is exactly that, with protobuf instead of gob and HTTP/2 instead of a raw socket. Things gRPC adds on top:

  • Deadlines that propagate. The ctx deadline is sent to the server, where the handler's ctx expires too, and any further downstream call inherits what is left. Always set one; The context Package explains the pattern.
  • Status codes in a standard set (OK, InvalidArgument, NotFound, AlreadyExists, PermissionDenied, Unauthenticated, ResourceExhausted, Unavailable, DeadlineExceeded and more), so errors keep their meaning across languages, unlike rpc.ServerError strings.
  • Metadata (headers and trailers) for authentication tokens and request IDs.
  • Interceptors: middleware for logging, metrics, authentication and retries, on both client and server (the counterpart of the HTTP middleware in Handlers, Routing and Middleware).
  • Four call shapes, chosen by the stream keyword in the .proto:
Kind Declaration Typical use
Unary rpc Get(Req) returns (Res) Ordinary request and response
Server streaming rpc Watch(Req) returns (stream Res) Live updates, large result sets
Client streaming rpc Upload(stream Req) returns (Res) Uploads, batching
Bidirectional rpc Chat(stream Req) returns (stream Res) Chat, interactive sessions

In Go a streaming handler gets a stream object with Send and Recv methods and returns when the stream is finished. The mechanics are similar to the loops in Streaming, SSE and WebSockets, with typed messages instead of text lines.

Connect: gRPC that browsers and curl can speak

Plain gRPC is awkward for browsers (they cannot control HTTP/2 trailers) and for debugging with curl. Connect (connectrpc.com/connect) is a Go library from Buf that generates handlers compatible with net/http. One service can answer three protocols on the same port: gRPC, gRPC-Web and Connect's own simple protocol, which is plain HTTP POST with protobuf or JSON bodies. You get the schema, generated types and streaming of gRPC, and a request you can send with curl --json '{"name":"Ada"}'. If you need browser clients or want to keep ordinary HTTP tooling, it is worth a look.

gRPC or REST?

REST with JSON gRPC
Contract Optional (OpenAPI) Required (.proto), always in sync with generated code
Wire format Text, larger, easy to read Binary, compact, fast to parse
Streaming SSE or WebSockets, added on Built in, four shapes
Browsers Native Needs gRPC-Web or Connect
Debugging curl, any browser grpcurl, reflection, special tooling
Best for Public APIs, simple CRUD, wide audience Internal service-to-service calls, polyglot teams, streaming

A common architecture is gRPC between internal services and a REST (or Connect) API for the outside world. Do not choose gRPC just because it is faster; choose it when the typed contract and streaming matter. A JSON REST API is easier to explore and cache, and for most public-facing products it is still the right default.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which method signature can net/rpc expose?

    An RPC method must be exported, take an argument and a pointer to the reply, and return an error. The reply is how the result is carried back.
  2. 2. A server method returns errors.New("divide by zero"). What does the net/rpc client receive?

    Only the error string crosses the wire, wrapped in rpc.ServerError. The original error type is lost, which is one reason typed status codes (as in gRPC) are better.
  3. 3. In protobuf, what identifies a field on the wire?

    The tag byte is (field number << 3) | wire type. Names exist only in the schema, which is why messages are compact and why field numbers must never be reused or changed.
  4. 4. Why does greeterStub.Greet return context deadline exceeded for a slow call, and what does that not tell you?

    net/rpc cannot cancel the server-side call. A timeout means the client gave up waiting, and the server may have completed the operation, so retried operations should be idempotent.
  5. 5. Which situation is the best fit for gRPC over plain REST?

    gRPC shines for internal, polyglot service-to-service calls where a generated, typed contract and streaming help. Public and browser-facing APIs usually favor REST or Connect.

Summary

  • RPC makes a remote call look like a local one: a service interface, a server adapter and a client stub. Remote calls can fail, time out or leave the outcome unknown, so include context.Context and error.
  • net/rpc exposes methods of the form func (t *T) Name(args A, reply *R) error using gob; it supports Call and asynchronous Go, but is frozen, Go-only and has no contexts or streaming. net/rpc/jsonrpc speaks JSON-RPC 1.0 over the same API.
  • Protocol Buffers describe messages in .proto files and encode fields as (number << 3 | wiretype) tags followed by varints or length-delimited data. Unknown fields are skipped, so adding fields is safe; never reuse or retype field numbers.
  • gRPC = protobuf + HTTP/2. Generated code gives typed clients and server interfaces, deadlines that propagate, standard status codes, metadata, interceptors and four call shapes (unary, server, client and bidirectional streaming).
  • Connect offers gRPC-compatible services over plain net/http, usable from browsers and curl.
  • Choose gRPC for internal typed service calls and streaming; choose REST or Connect for public APIs and simple tooling.

Next: Databases with database/sql