Building High Performance APIs In Go Using gRPC And Protocol Buffers

8 min readOct 11, 2016

APIs are backbone of modern applications. APIs are powering the backend for web client and mobile client applications, and also used for communicate between applications regardless of technology and platform. When you think about building web based APIs, you typically choose RESTful APIs along with JSON as the standard for interchanging data between applications. This approach is fine and mobile client applications can easily consume these JSON based RESTful APIs. Now we’re building applications for the era of cloud-native applications where our Microservices should be able for massive scale and performance is very critical. And we definitely need a high performance communication mechanism for communicate between various Microservices. Then the big question is whether JSON based APIs provide high performance and scalability power which is required for modern applications. Is JSON really a fast data format for exchanging data between applications?. Is RESTful architecture capable of building complex APIs?. Can we easily build a bidirectional stream APIs with RESTful architecture?. The HTTP/2 protocol provides lot of capability than its previous version, thus we need to leverage those capabilities when building next-generation APIs. Then enter to gRPC and protocol buffers.

Introduction to Protocol Buffers

Protocol Buffers, also referred as protobuf, is Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. Protocol Buffers are smaller, faster, and simpler that provides high performance than other standards such as XML and JSON.
By using protocol buffers, you can define your structured data, then you generate source code for your choice of programming language using the protocol buffer compiler named protoc, to write and read your structured data using it. The current version of protocol buffers is proto3. The proto3 version currently supports generated code in variety of languages including C++, Go, Java, Python, Ruby, and C#.

In order to generated code from protocol buffers definition files, do the following:

Download and install protoc compiler from here: https://github.com/google/protobuf. Add the location of protoc binary file into PATH environment variable so that you can invoke protoc compiler from any location.
Install the protoc plugin for your language. For Go, run the go get command to install the protoc plugin for Go:

go get -u github.com/golang/protobuf/protogo get -u github.com/golang/protobuf/protoc-gen-go

Introduction to gRPC

gRPC is a high performance, open-source remote procedure call (RPC) framework that can run anywhere. It enables client and server applications to communicate transparently, and makes it easier to build connected systems. The gRPC framework is developed and open-sourced by Google. Google has been using a lot of the underlying technologies and concepts in gRPC for a long time for their many products including several of Google’s cloud products. The motivation and design principle of gRPC is available from here: http://www.grpc.io/blog/principles.

gRPC follows HTTP semantics over HTTP/2. It allows you to build services with both synchronous and asynchronous communication model. It supports traditional Request/Response model and bidirectional streams. Its capability for building full-duplex streaming lets you use it for advanced scenarios where both client and server applications can send stream of data asynchronously. gRPC is built with mobile clients in mind that will give you lot of performance advantages and easiness for consuming APIs . When compared to RESTful approach, gRPC has many advantages including performance gain.

By default, gRPC uses Protocol Buffers as the Interface Definition Language (IDL) and as its underlying message interchange format. Unlike JSON and XML, Protocol Buffers are not just message interchange format, it’s also used for describing the service interfaces (service endpoints). Thus Protocol Buffers are used for both the service interface and the structure of the payload messages. In gRPC, you define services and its methods along with payload messages. Like a typical communication between a client application and a RPC system, a gRPC client application can directly call methods on a remote server as if it was a local object in your client application.

Here is an image (taken from gRPC web site: http://www.grpc.io/) that illustrates the communication between a gRPC server and a client application.

The gRPC server implements the service interface and runs an RPC server to handle client calls to its service methods. On the client side, the client has a stub (referred to as just a client in some languages) that provides the same methods as the server.

An Example API in Go with gRPC and Protocol Buffers

Let’s build an API example in Go using gRPC and Protocol Buffers. Here is the directory structure used for the application:

In order to working with gRPC in Go, you must install Go implementation of gRPC:

go get google.golang.org/grpc

Defining Message Types and Service Definition in Protocol Buffer

Let’s define the service interface and the structure of the payload messages in a Protocol Buffer file. File extension, .proto is used for creating Protocol Buffer file. Here is the source of customer.proto file in customer directory:

The .proto file starts with the version of Protocol Buffer and a package declaration. We use the latest proto3 version of the Protocol Buffers language. The package is declared with a name “customer”. When you generate Go source code from proto file, it will add Go package name as “customer”.

Inside a .proto file, you define message types and service interface. Standard data types such as int32, float, double, and string are available as field types for declaring elements in message types. The user defined message types can also be used as the field types. The “ = 1”, “ = 2” markers on each element of the message types specifies the unique “tag” that field uses in the binary encoding. A default value is used if element value is not specified: zero for numeric types, the empty string for strings, false for bools. The Protocol Buffers language guide is available from here: https://developers.google.com/protocol-buffers/docs/proto3.

A named service “Customer” is used to define a service, which has two RPC methods.

// The Customer service definition.
 service Customer { 
 // Get all Customers with a filter — A server-to-client streaming RPC.
 rpc GetCustomers(CustomerFilter) returns (stream CustomerRequest) {}
 // Create a new Customer — A simple RPC 
 rpc CreateCustomer (CustomerRequest) returns (CustomerResponse) {}
 }

Here are the different kind of RPC methods you can define with gRPC:

A simple RPC method that works with a typical Request/Response model where the client sends a request to the RPC server using the stub and waits for a response.
A server-side streaming RPC where the client sends a request to the server and gets a stream to read a sequence of messages back. The stream keyword is specified before the response type to make it as server-side streaming RPC method.
A client-side streaming RPC where the client writes a sequence of messages and sends them to the server using a provided stream. The stream keyword is specified before the request type to make it as client-side streaming RPC method.
A bidirectional streaming RPC where both client and server send a sequence of messages using a read-write stream. The stream keyword is specified before both the request type and response type to make it as bidirectional streaming RPC method.

The Customer service provides two kind of RPC methods: A simple RPC method named CreateCustomer and a A server-side streaming RPC method named GetCustomers. The CreateCustomer creates a new customer and it executes as a Request/Response paradigm. The GetCustomers provides list of Customers where server provides Customer data as stream.

Generating Go Code for Client and Server

Once you’ve defined the proto file, the next step is to generate source code for the gRPC client and server interfaces to write your server implementation and making client calls based on the messages types and service interface defined in the proto file. The protocol buffer compiler protoc is used with a gRPC Go plugin to generate the client and server code. From the root directory of the application, run the protoc compiler with gRPC Go plugin:

protoc -I customer/ customer/customer.proto --go_out=plugins=grpc:customer

This will generate a Go source file named customer.pb.go in the customer directory. The generated source file provides all essential code to create the server and make client calls through RPC.

Creating the gRPC Server

The below main.go file in the server directory shows the source for creating the gRPC server by providing implementation for the RPC methods defined in the service definition:

The struct type server is used to implement CustomerServer defined in the customer.pb.go file. It provides implementation for two RPC methods: CreateCustomer and GetCustomers. In method CreateCustomer, a context object is passed to the RPC and the client’s Protocol Buffer request of type CustomerRequest, and it returns Protocol Buffer object of type CustomerResponse. The GetCustomers method is a server-side streaming RPC. Unlike CreateCustome RPC, which is a simple RPC method in which a request object is specified as input parameter and a response object is specified as the return type, in GetCustomers RPC, a request object of type CustomerFilter to filter the Customer data, and a special type Customer_GetCustomersServer are specified for defining the RPC method. The Customer_GetCustomersServer object is used to write responses of RPC as stream using the method Send of Customer_GetCustomersServer. The type Customer_GetCustomersServer is defined in the generated code file customer.pb.go.

Function grpc.NewServer creates a new gRPC server, to which we register the instance of CustomerServer. The method Serve accepts incoming connections on the given listener, creating a new ServerTransport and service goroutine for each. The service goroutines read gRPC requests and then call the registered handlers to reply to them.

Creating the gRPC Client

The generated customer.pb.go file provides the essential code to make RPC calls from client applications. The below main.go file in the client directory shows the source for creating the gRPC client by providing implementation for the RPC methods defined in the service definition:

A gRPC channel is created to communicate with the server in order to call RPC methods. Function grpc.Dial is used to communicate with the RPC server. When you call grpc.Dial, you can pass a DialOptions to set the authentication credentials such as TLS and JWT. In this example, we pass grpc.WithInsecure as DialOptions which disables transport security for the client connection. In order to call RPC methods, we need to create a client stub which is created using function NewCustomerClient that returns an object of CustomerClient. Function NewCustomerClient and type CustomerClient are defined in the generated code file customer.pb.go.

For the sake of example, we create two Customer using RPC method CreateCustomer and getting the Customer data using RPC method GetCustomers.

Let’s run the gRPC server by running the main.go file from directory server:

go run main.go

To test the RPC methods by calling from the client application, run the main.go file from directory client:

go run main.go

You should see the output similar to the following:

2016/10/11 16:02:47 A new Customer has been added with id: 101
2016/10/11 16:02:47 A new Customer has been added with id: 102
2016/10/11 16:02:47 Customer: id:101 name:”Shiju Varghese” email:”shiju@xyz.com” phone:”732–757–2923" addresses:<street:”1 Mission Street” city:”San Francisco” state:”CA” zip:”94105" > addresses:<street:”Greenfield” city:”Kochi” state:”KL” zip:”68356" isShippingAddress:true >
2016/10/11 16:02:47 Customer: id:102 name:”Irene Rose” email:”irene@xyz.com” phone:”732–757–2924" addresses:<street:”1 Mission Street” city:”San Francisco” state:”CA” zip:”94105" isShippingAddress:true >

Source Code

The source code of the example is available from here.

You can follow me on twitter at @shijucv. I do provide training and consulting on Go programming language (Golang) and distributed systems architectures, in India.