Go Microservices with Go kit: Introduction

11 min readNov 17, 2018

In this post, I will give an introduction to Go kit, a toolkit for building microservices in Go. This post is an introductory post on Go kit as part of a series of blog posts on it, and the source code of examples are available from here: https://github.com/shijuvar/gokit-examples

Go has gradually been becoming a language of choice for building modern distributed systems. When you build cloud-native distributed systems, you may need specialized support for various capabilities such as support for different transports and message encoding formats, RPC safety, logging, tracing, metrics and instrumentation, circuit breaking, rate limiting, infrastructure integration and even guidance on architecture. Go is very popular for its simplicity and “no magic” approach, thus a collection of Go packages like standard library packages, for distributed programming would be more appropriate than using a full-fledged framework with lot of magic and opinions on underlying technologies. Personally I don’t like to use full-fledged frameworks that come with lot of opinions, but I do prefer libraries that come with less opinions and more freedom to developers. Go kit fills this gap in the Go ecosystem by providing a distributed programming toolkit for building microservices, which also encourages you embrace good design principles for individual services in your distributed systems.

Introduction to Go kit

Go kit (http://gokit.io) is a collection of Go packages that help you build robust, reliable, maintainable microservices. Go kit provides libraries for implementing components for system observability and resiliency patterns such as logging, metrics, tracing, rate-limiting and circuit breaking, which are essential requirements for running microservices in production. The good thing about Go kit is that it’s a lightly opinionated, and was designed for interoperability that works with different infrastructures, message encoding formats and transport layers.

Besides a toolkit for building microservices, it encourages good design principles for your application architecture within a service. Go kit helps you embrace SOLID design principles, domain-driven design (DDD), and “hexagonal architecture” proposed by Alistair Cockburn, or one of the similar kind of architecture approaches known as “onion architecture” by Jeffrey Palermo and “clean architecture” by Robert C. Martin. Although Go kit is designed as a microservices toolkit, it is also well-suited for building elegant monoliths.

Go kit Architecture

There are three major components in the Go kit based application architecture:

Transport layer
Endpoint layer
Service layer

Transports

When you build microservices based distributed systems, services often communicate to each other using concrete transports like HTTP or gRPC, or using a pub/sub systems like NATS. The transport layer in Go kit is bound to concrete transports. Go kit supports various transports for serving services using HTTP, gRPC, NATS, AMQP and Thrift. Because Go kit services are just focused on implementing business logic and don’t have any knowledge about concrete transports, you can provide multiple transports for a same service. For example, a single Go kit service can be exposed by using both HTTP and gRPC.

Endpoints

Endpoint is the fundamental building block of servers and clients. In Go kit, the primary messaging pattern is RPC. An endpoint represents a single RPC method. Each service method in a Go kit service converts to an endpoint to make RPC style communication between servers and clients. Each endpoint exposes the service method to outside world using Transport layer by using concrete transports like HTTP or gRPC. A single endpoint can be exposed by using multiple transports.

Services

The business logic is implemented in Services. Go kit services are modelled as interfaces. The business logic in the services contain core business logic, which should not have any knowledge of endpoint or concrete transports like HTTP or gRPC, or encoding and decoding of request and response message types. This will encourage you follow a clean architecture for the Go kit based services. Each service method converts as an endpoint by using an adapter and exposed by using concrete transports. Because of the clean architecture, a single Go kit service can be exposed by using multiple transports.

Introduction to Go kit with an example

Let’s dive into the basics of Go kit by using an example program.

Business logic in services

The business logic is implemented in services that modelled as interface. Here we model our service on an e-commerce domain Order aggregate:

Listing 1. Order service interface

// Service describes the Order service.
type Service interface {
   Create(ctx context.Context, order Order) (string, error)
   GetByID(ctx context.Context, id string) (Order, error)
   ChangeStatus(ctx context.Context, id string, status string) error
}

The Order service interface uses the domain entity Order:

Listing 2. Order domain entity

// Order represents an order
type Order struct {
   ID           string      `json:"id,omitempty"`
   CustomerID   string      `json:"customer_id"`
   Status       string      `json:"status"`
   CreatedOn    int64       `json:"created_on,omitempty"`
   RestaurantId string      `json:"restaurant_id"`
   OrderItems   []OrderItem `json:"order_items,omitempty"`
}

// OrderItem represents items in an order
type OrderItem struct {
   ProductCode string  `json:"product_code"`
   Name        string  `json:"name"`
   UnitPrice   float32 `json:"unit_price"`
   Quantity    int32   `json:"quantity"`
}

// Repository describes the persistence on order model
type Repository interface {
   CreateOrder(ctx context.Context, order Order) error
   GetOrderByID(ctx context.Context, id string) (Order, error)
   ChangeOrderStatus(ctx context.Context, id string, status string) error
}

Here’s the implementation of the service interface:

Listing 3. Implementation of business logic for Order service interface

package implementation

import (
   "context"
   "database/sql"
   "time"

   "github.com/go-kit/kit/log"
   "github.com/go-kit/kit/log/level"
   "github.com/gofrs/uuid"

   ordersvc "github.com/shijuvar/gokit-examples/services/order"
)

// service implements the Order Service
type service struct {
   repository ordersvc.Repository
   logger     log.Logger
}

// NewService creates and returns a new Order service instance
func NewService(rep ordersvc.Repository, logger log.Logger) ordersvc.Service {
   return &service{
      repository: rep,
      logger:     logger,
   }
}

// Create makes an order
func (s *service) Create(ctx context.Context, order ordersvc.Order) (string, error) {
   logger := log.With(s.logger, "method", "Create")
   uuid, _ := uuid.NewV4()
   id := uuid.String()
   order.ID = id
   order.Status = "Pending"
   order.CreatedOn = time.Now().Unix()

   if err := s.repository.CreateOrder(ctx, order); err != nil {
      level.Error(logger).Log("err", err)
      return "", ordersvc.ErrCmdRepository
   }
   return id, nil
}

// GetByID returns an order given by id
func (s *service) GetByID(ctx context.Context, id string) (ordersvc.Order, error) {
   logger := log.With(s.logger, "method", "GetByID")
   order, err := s.repository.GetOrderByID(ctx, id)
   if err != nil {
      level.Error(logger).Log("err", err)
      if err == sql.ErrNoRows {
         return order, ordersvc.ErrOrderNotFound
      }
      return order, ordersvc.ErrQueryRepository
   }
   return order, nil
}

// ChangeStatus changes the status of an order
func (s *service) ChangeStatus(ctx context.Context, id string, status string) error {
   logger := log.With(s.logger, "method", "ChangeStatus")
   if err := s.repository.ChangeOrderStatus(ctx, id, status); err != nil {
      level.Error(logger).Log("err", err)
      return ordersvc.ErrCmdRepository
   }
   return nil
}

Requests and Responses for RPC endpoints

The service methods expose as RPC endpoints. So we need to define message types to be used for send and receive messages over RPC endpoints. Let’s define structs for request and response types to be used for RPC endpoints on Order service:

Listing 4. Message types for Request and Response for RPC endpoints

// CreateRequest holds the request parameters for the Create method.
type CreateRequest struct {
   Order order.Order
}

// CreateResponse holds the response values for the Create method.
type CreateResponse struct {
   ID  string `json:"id"`
   Err error `json:"error,omitempty"`
}

// GetByIDRequest holds the request parameters for the GetByID method.
type GetByIDRequest struct {
   ID  string
}

// GetByIDResponse holds the response values for the GetByID method.
type GetByIDResponse struct {
   Order order.Order `json:"order"`
   Err error `json:"error,omitempty"`
}

// ChangeStatusRequest holds the request parameters for the ChangeStatus method.
type ChangeStatusRequest struct {
   ID  string `json:"id"`
   Status string `json:"status"`
}

// ChangeStatusResponse holds the response values for the ChangeStatus method.
type ChangeStatusResponse struct {
   Err error `json:"error,omitempty"`
}

Go kit endpoints for service methods as RPC endpoints

Our core business logic resides in services, which will be exposed as RPC endpoints using a Go kit abstraction called endpoint.

Here’s the endpoint type in Go kit:

Listing 5. endpoint.Endpoint type in Go kit

type Endpoint func(ctx context.Context, request interface{}) (response interface{}, err error)

As we discussed earlier, an endpoint represents a single RPC method. Each service method converts as an endpoint (type endpoint.Endpoint) by using an adapter. Let’s make Go kit endpoints on Order service methods:

Listing 6. Adapters that convert service methods to endpoint.Endpoint

import (
   "context"

   "github.com/go-kit/kit/endpoint"

   "github.com/shijuvar/gokit-examples/services/order"
)

// Endpoints holds all Go kit endpoints for the Order service.
type Endpoints struct {
   Create       endpoint.Endpoint
   GetByID      endpoint.Endpoint
   ChangeStatus endpoint.Endpoint
}

// MakeEndpoints initializes all Go kit endpoints for the Order service.
func MakeEndpoints(s order.Service) Endpoints {
   return Endpoints{
      Create:       makeCreateEndpoint(s),
      GetByID:      makeGetByIDEndpoint(s),
      ChangeStatus: makeChangeStatusEndpoint(s),
   }
}

func makeCreateEndpoint(s order.Service) endpoint.Endpoint {
   return func(ctx context.Context, request interface{}) (interface{}, error) {
      req := request.(CreateRequest)
      id, err := s.Create(ctx, req.Order)
      return CreateResponse{ID: id, Err: err}, nil
   }
}

func makeGetByIDEndpoint(s order.Service) endpoint.Endpoint {
   return func(ctx context.Context, request interface{}) (interface{}, error) {
      req := request.(GetByIDRequest)
      orderRes, err := s.GetByID(ctx, req.ID)
      return GetByIDResponse{Order: orderRes, Err: err}, nil
   }
}

func makeChangeStatusEndpoint(s order.Service) endpoint.Endpoint {
   return func(ctx context.Context, request interface{}) (interface{}, error) {
      req := request.(ChangeStatusRequest)
      err := s.ChangeStatus(ctx, req.ID, req.Status)
      return ChangeStatusResponse{Err: err}, nil
   }
}

The endpoint adapters take a service interface as a parameter and then converts into Go kit abstraction endpoint.Endpoint to make it individual service methods as endpoints. These adapter functions make type assertion to appropriate request type and then call the service methods and return the response messages.

func makeCreateEndpoint(s order.Service) endpoint.Endpoint {
   return func(ctx context.Context, request interface{}) (interface{}, error) {
      req := request.(CreateRequest)
      id, err := s.Create(ctx, req.Order)
      return CreateResponse{ID: id, Err: err}, nil
   }
}

Exposing service with HTTP transport

We have created our service, and then we have created endpoints to expose service methods as RPC endpoints. Now we need to expose our service to the outside world so that other services can make RPC calls on it. In order to expose our services, we need a concrete transport to serve the services. Go kit supports various transports such as HTTP, gRPC, NATS, AMQP and Thrift, out of the box.

Just for the sake of example, we use HTTP transport for serving the service. The Go kit package github.com/go-kit/kit/transport/http provides the support for serving services with HTTP transport. The function NewServer of transport/http package constructs a new server, which implements http.Handler and wraps the provided endpoint.

Here’s the code block the wires the Go kit endpoints to HTTP transport to serve the service over HTTP transport:

Listing 7. Wires Go kit endpoints to HTTP transport

package http

import (
   "context"
   "encoding/json"
   "errors"
   "github.com/shijuvar/gokit-examples/services/order"
   "net/http"

   "github.com/go-kit/kit/log"
   kithttp "github.com/go-kit/kit/transport/http"
   "github.com/gorilla/mux"

   "github.com/shijuvar/gokit-examples/services/order/transport"
)

var (
   ErrBadRouting = errors.New("bad routing")
)

// NewService wires Go kit endpoints to the HTTP transport.
func NewService(
   svcEndpoints transport.Endpoints, logger log.Logger,
) http.Handler {
   // set-up router and initialize http endpoints
   r := mux.NewRouter()
   options := []kithttp.ServerOption{
      kithttp.ServerErrorLogger(logger),
      kithttp.ServerErrorEncoder(encodeError),
   }
   // HTTP Post - /orders
   r.Methods("POST").Path("/orders").Handler(kithttp.NewServer(
      svcEndpoints.Create,
      decodeCreateRequest,
      encodeResponse,
      options...,
   ))

   // HTTP Post - /orders/{id}
   r.Methods("GET").Path("/orders/{id}").Handler(kithttp.NewServer(
      svcEndpoints.GetByID,
      decodeGetByIDRequest,
      encodeResponse,
      options...,
   ))

   // HTTP Post - /orders/status
   r.Methods("POST").Path("/orders/status").Handler(kithttp.NewServer(
      svcEndpoints.ChangeStatus,
      decodeChangeStausRequest,
      encodeResponse,
      options...,
   ))
   return r
}


func decodeCreateRequest(_ context.Context, r *http.Request) (request interface{}, err error) {
   var req transport.CreateRequest
   if e := json.NewDecoder(r.Body).Decode(&req.Order); e != nil {
      return nil, e
   }
   return req, nil
}

func decodeGetByIDRequest(_ context.Context, r *http.Request) (request interface{}, err error) {
   vars := mux.Vars(r)
   id, ok := vars["id"]
   if !ok {
      return nil, ErrBadRouting
   }
   return transport.GetByIDRequest{ID: id}, nil
}

func decodeChangeStausRequest(_ context.Context, r *http.Request) (request interface{}, err error) {
   var req transport.ChangeStatusRequest
   if e := json.NewDecoder(r.Body).Decode(&req); e != nil {
      return nil, e
   }
   return req, nil
}

func encodeResponse(ctx context.Context, w http.ResponseWriter, response interface{}) error {
   if e, ok := response.(errorer); ok && e.error() != nil {
      // Not a Go kit transport error, but a business-logic error.
      // Provide those as HTTP errors.
      encodeError(ctx, e.error(), w)
      return nil
   }
   w.Header().Set("Content-Type", "application/json; charset=utf-8")
   return json.NewEncoder(w).Encode(response)
}

We create http.Handler values by calling the function NewServer of transport/http package, to which we provide an endpoint and function values for decoding the request (value of DecodeRequestFunc func type) and encoding the response values (value of EncodeResponseFunc func type).

Here’s the definition of DecodeRequestFunc and EncodeResponseFunc func types in Go kit:

Listing 8. DecodeRequestFunc and EncodeResponseFunc func types

// For decoding request
type DecodeRequestFunc func(context.Context, *http.Request) (request interface{}, err error)// For encoding response
type EncodeResponseFunc func(context.Context, http.ResponseWriter, interface{}) error

Running the HTTP server

Finally we run our HTTP server. The NewService function of Listing 7 gives an instance of http.Handler, which is being used to run our HTTP server.

Listing 9. Run HTTP server

func main() {
   var (
      httpAddr = flag.String("http.addr", ":8080", "HTTP listen address")
   )
   flag.Parse()

   var logger log.Logger
   {
      logger = log.NewLogfmtLogger(os.Stderr)
      logger = log.NewSyncLogger(logger)
      logger = level.NewFilter(logger, level.AllowDebug())
      logger = log.With(logger,
         "svc", "order",
         "ts", log.DefaultTimestampUTC,
         "caller", log.DefaultCaller,
      )
   }

   level.Info(logger).Log("msg", "service started")
   defer level.Info(logger).Log("msg", "service ended")

   var db *sql.DB
   {
      var err error
      // Connect to the "ordersdb" database
      db, err = sql.Open("postgres", 
         "postgresql://shijuvar@localhost:26257/ordersdb?sslmode=disable")
      if err != nil {
         level.Error(logger).Log("exit", err)
         os.Exit(-1)
      }
   }

   // Create Order Service
   var svc order.Service
   {
      repository, err := cockroachdb.New(db, logger)
      if err != nil {
         level.Error(logger).Log("exit", err)
         os.Exit(-1)
      }
      svc = ordersvc.NewService(repository, logger)
   }

   var h http.Handler
   {
      endpoints := transport.MakeEndpoints(svc)
      h = httptransport.NewService(endpoints, logger)
   }

   errs := make(chan error)
   go func() {
      c := make(chan os.Signal)
      signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
      errs <- fmt.Errorf("%s", <-c)
   }()

   go func() {
      level.Info(logger).Log("transport", "HTTP", "addr", *httpAddr)
      server := &http.Server{
         Addr:    *httpAddr,
         Handler: h,
      }
      errs <- server.ListenAndServe()
   }()

   level.Error(logger).Log("exit", <-errs)
}

Now our service runs using HTTP transport. The same service can be exposed using other transports. For example, the same service can be exposed using gRPC or Apache Thrift.

As an introductory post, we just used the basic primitives provided by Go kit, but it provides a lot of functionalities for system observability, resiliency patterns, service discovery and load balancing, etc. We will discuss more things on Go kit in later posts.

Source Code

The source code of the examples on Go kit are available from here: https://github.com/shijuvar/gokit-examples

Middlewares in Go kit

Go kit encourages good design principle by enforcing separation of concerns. Cross-cutting components for the services and endpoints are implemented by using Middlewares. Middlewares in Go kit is a powerful mechanism that can wrap services and endpoints to add functionality (cross-cutting components), such as logging, circuit breakers, rate limiting, load balancing, or distributed tracing.

Here’s an image taken from Go kit web site (http://gokit.io) that depicts an onion of layers of a typical Go kit based design with middlewares.

Beware about Spring Boot Microservices syndrome

Like Go kit, Spring Boot is a toolkit for building microservices in Java ecosystem. But, unlike Go kit, Spring Boot is more of a full-fledged framework. Although a lot of Java developers are using Spring Boot framework for building microservices in Java stack in a positive sense, some of them believe that microservices are just about using Spring Boot framework. I have seen lot of development teams who misinterpret that microservices are just about building services with Spring Boot and using Netflix OSS, and don’t perceive microservices as a distributed systems pattern.

So keep in mind that using a toolkit like Go kit or a framework just bootstrap your development towards microservices. Although microservices solve many of the challenges for scaling teams and systems, it make new set of problems because of the data in microservices based systems are scattered amongst various databases, which sometimes make lot of challenges for making business transactions and querying data. It’s all depend on the problem domain and the context of your systems. The good thing is that although Go kit was designed as a toolkit for building microservices, you can also use this toolkit for building elegant monoliths that enforces good architecture design for your systems.

And some of the functionalities of Go kit such as circuit breakers and rate limiting are available through service mesh platforms like Istio. So if you use a service mesh like Istio for running your microservices, you may not need those things from Go kit, but everyone will not have the bandwidth for using a service mesh for making service-to-service communications as it add another layer of complexity.

You can follow me on twitter at @shijucv. I do provide training and consulting on Go programming language (Golang) and distributed systems architectures, in India.