Introducing NATS to Go Developers

In this post, I will introduce NATS as the messaging system to Go developers for building distributed systems and Microservices. When you build distributed applications, a messaging system plays a key role to make communications between applications, most probably in asynchronous manner with an event-driven architecture. There are lot of distributed queueing and messaging systems are available for building modern distributed applications. Open source technologies like Kafka, NATS, NSQ, RabbitMQ, ActiveMQ, and Cloud PaaS offerings like Google Cloud Pub/Sub, Amazon SQS, Amazon SNS Topic, Azure Service Bus, are providing different capabilities and patterns for building messaging middlewares and distributed systems. Among the technologies mentioned above, both NATS and NSQ are written in Go. It has never been an easy job to build distributed applications, even with modern approaches like Microservices. Because building distributed systems are always complex, using a complex messaging system will make more complexities to your applications. And a modern messaging system should be able to easily run and scale on variety of environments such as on-premises server, Cloud platforms and containers.

Introducing NATS

NATS is an open source, lightweight, high-performance cloud native messaging system. It implements a publish-subscribe messaging system with greater level of scalability power. Although NATS is based on publish-subscribe distribution model, you can also implement distributed queuing via subscriber queue groups. NATS was created in 2010, originally built to serve as the messaging system for Cloud Foundry platform. It was initially written in Ruby, but later the NATS team rewrote it with Go.

NATS is available in two interoperable modules: the core NATS platform -the NATS server (executable name is gnatsd) referred to simply as NATS, and NATS Streaming (executable name is nats-streaming-server), an event streaming service that can be employed to add event streaming, delivery guarantees, and historical data replay to NATS. NATS server is designed for high performance, which is targets to be used for modern distributed systems architectures, and it doesn’t do persistent messaging. Thus, if your systems are offline, you don’t get the messages. If you need persistent messaging and delivery guarantees, you can use NATS Streaming instead of the core NATS platform. NATS Streaming is built on the top of core NATS platform. In this post, I will be focusing on the basic NATS server and will introduce NATS Streaming Server in a future post.

The NATS server (gnatsd) is the most performant distributed messaging system, which is capable of sending 15-18 million messages per second. The NATS platform is simple to use and scale. The simplicity and performant nature of NATS make it a great choice of messaging system for building modern cloud native distributed systems and Microservices. Having worked on many messaging systems in the past, I highly recommend NATS mainly for its performance and simplicity.

(Chart source: bravenewgeek.com/dissecting-message-queues)

Messaging Patterns

Here’re the three messaging patterns provided by NATS, while it works as a publish-subscribe engine:

• Publish-Subscribe
• Queueing
• Request-Reply

Components of Messaging Architecture

Here’re the major components in the NATS messaging infrastructure:

• Message: Messages are the unit of data exchange. A payload, which is used for exchanging the data between applications.
• Subject: Subject specifies the destination of messages.
• Producer: Producers send messages to the NATS server.
• Consumer: Consumers receive messages from the NATS server.
• Messaging Server: NATS Server distributes the messages from producers to consumers.

Install Server and Clients

Here’s the various distributions for download NATS server: http://nats.io/download/nats-io/gnatsd/

You can also install the NATS server using the go tool:

go get github.com/nats-io/gnatsd

You can start the NATS server by running the executable gnatsd:

gnatsd

Here’s the link for download various NATS clients:

http://nats.io/download/

You can install the Go NATS client using the go tool:

go get github.com/nats-io/go-nats

Using NATS with Go

Let’s explore NATS by using with an example distributed application in Go. In this example, we will use Request-Reply and Publish-Subscribe messaging patterns. With Publish-Subscribe, we will also use queueing using subscriber queue group. The example will use Protocol Buffers for sending and receiving messages with NATS. Here’s the Protocol Buffers definitions used for various messaging.

Listing 1. Message types in Protocol Buffers

message ServiceDiscovery {
    string order_service_uri = 1;
}

message EventStore {
  string aggregate_id = 1;
  string aggregate_type = 2;
  string event_id = 3;
  string event_type = 4;
  string event_data = 5;
}

The message type ServiceDiscovery is used for Request-Reply communication and type EventStore is used for Publish-Subscribe.

Example on NATS Request-Reply

The Request-Reply message pattern works like a normal Request-Response communication, in which publish request operation publishes a message with a reply subject expecting a response on that reply subject. Here we use this pattern for a simple service discovery that finds out an endpoint of a service. Note that I use NATS Request-Reply messaging here just for the sake of example. NATS Request-Reply is similar like gRPC’s simple RPC (Request-Reply), and hence I typically use gRPC APIs instead of NATS Request-Reply messaging. But I always use NATS for pub/sub communications for an event-driven architecture with Go microservices.

Here’s the code block that sends a request on subject “Discovery.OrderService” to get the service endpoint:

Listing 2. Request on NATS Request-Reply messaging

The Go NATS client library is imported into the program:

import “github.com/nats-io/go-nats”

Function nats.Connect will attempt to connect to the NATS system. By default, NATS server running at “nats://localhost:4222”. Here we use the DefaultURL to connect to the server.

natsConnection, _ := nats.Connect(nats.DefaultURL)

The NATS client is then used to send a request on a subject named “Discovery.OrderService” to get a response.

msg, err := natsConnection.Request("Discovery.OrderService", nil, 1000*time.Millisecond)

When you send a request on a subject, you can pass a request data and timeout. Here we don’t provide any data as we send the request just for getting a response on the subject. Here we use Protocol Buffers for sending and receiving messages so that the response data is decoded into Go struct value:

orderServiceDiscovery := pb.ServiceDiscovery{}
err := proto.Unmarshal(msg.Data, &orderServiceDiscovery)
if err != nil {
       log.Fatalf("Error on unmarshal: %v", err)
}
address := orderServiceDiscovery.OrderServiceUri
log.Println("OrderService endpoint found at:", address)

Here’s the code block from another application that subscribes the subject “Discovery.OrderService” to provide reply to the request:

Listing 3. Response on NATS Request-Reply messaging

The subject “Discovery.OrderService” is subscribed for sending the response. Here the response data is sent by encodng into Protocol Buffers.

Example on NATS Publish-Subscribe

I highly recommend NATS as the Publish-Subscribe engine for pub/sub communication model and for enterprise queueing, for building your distributed applications in Go. NATS Publish-Subscribe is a one-to-many communication pattern, in which a publisher sends a message on a subject, and all active subscribers listening on that subject receive the message. This model of communication is typically performed in asynchronous manner where published messages are delivered to the subscriber’s message handler. If there is no handler, the subscription works in synchronous model and the client may be blocked until it process the message. In most of the real-world scenario, you may not need synchronous way of communication for pub/sub communication.

Image Source: NATS Web SIte (http://nats.io)

When you create subscribers, you can also register with a queue name. All subscribers with the same queue name form the queue group. As messages on the registered subject are published, one member of the group is chosen randomly to receive the message. Although queue groups have multiple subscribers, each message is only consumed by only one. When you create subscribers, you can register with or without a queue name. Among the subscribers with a queue group, one subscriber receives the message, but for subscribers without queue group, all subscribers receive the message. The interesting thing about NATS is that it provides queueing even though NATS messaging is based on Publish-Subscribe pattern.

In our example on NATS Publish-Subscribe, we create a subscriber without a queue, and multiple subscribers with a queue group named “Order.OrdersCreatedQueue”. So one subscriber from queue group and the another subscriber (without queue group) can receive the message. The publisher client is a gRPC server, which publishes messages on subject “Order.OrderCreated” when new orders are created. Here’s the code block from publisher client that publishes message on subject “Order.OrderCreated”:

Listing 4. Publisher Client of NATS Publish-Subscribe messaging

The function Publish of NATS client, publishes the message to the given subject.Here the messages are marshalled into Protocol Buffers. Let’s create the subscribers to receive messages when messages are published on subject from publisher client.

Here’s the code block from subscriber client that subscribes messages on subject:

Listing 5. Subscriber Client of NATS Publish-Subscribe messaging

Here messages are subscribed using wildcard subject “Order.>”. NATS supports the use of wildcards in subject subscriptions. It supports asterisk character (*) and greater than symbol (>), also known as the full wildcard to be used for wildcards in subject subscriptions. The wildcarded subject Order.> will match Order.Created, Order.Shipped, Order.Delivered, Order.Delivered.Returned, etc. The wildcarded subject Order.* will match Order.Created, Order.Shipped, Order.Delivered, etc, but not Order.Delivered.Returned.

The function Subscribe of NATS client, subscribes messages using a handler to asynchronously receive messages when messages are published on the given subject.

natsConnection.Subscribe(subject, func(msg *nats.Msg) {
       eventStore := pb.EventStore{}
       err := proto.Unmarshal(msg.Data, &eventStore)
})

Because messages are published in Protocol Buffers encoding, received messages are decoded into Go struct values using proto.Unmarshal.

Let’s create subscribers with a queue group. Here’s the code block from subscriber client that subscribes messages on subject:

Listing 6. Subscriber Client with queue group of NATS Publish-Subscribe messaging

Here messages on subject “Order.OrderCreated”, is subscribed using function QueueSubscribe with a queue named “Order.OrdersCreatedQueue". When we create multiple subscribers on a subject with same queue name, it forms a queue group and one member of the group is chosen randomly to receive the message. If you want to use NATS just for queueing, you can create subscribers only with queue group.

In this post, I have just explored the basic capabilities of the core NATS platform. I will write another post on NATS Streaming Server later on.

The source of the example is available from here.

You can follow me on twitter at @shijucv. I do provide training and consulting on Go programming language (Golang) and distributed systems architectures, in India.