My Six Years of Experience as a Go Programming Language Mentor in India

In this post, I will share my experience of working as a consulting solutions architect and a mentor on Go programming language and distributed systems architectures, in India, and I will also share some of my recommendations based on my experience, when you adopt Go in your organisations.

I am an early adopter of Go, and authored two books on it, in 2015 and in 2016. Before Go, I worked extensively on Microsoft .net stack for more than a decade, and since 2008, I have been working as an Architect. When I left from .net technologies and moved to Go, I was a Microsoft MVP and an Azure insider. Since 2015, I have been working as a consulting solutions architect and a trainer to mentor companies to adopt Go in their organisations and to projects and products. During this period, I have mentored more than 50 companies, which includes start-up companies on product engineering, mid-sized companies and larger organisations. While I work with these companies, I went through different emotions — pleasant, unpleasant, and neither unpleasant nor pleasant. What was my experience working with these companies?

The experience of working with startups and mid-sized companies

My experience of working with start-up companies on product engineering and mid-sized companies on both product engineering and project development, has been very pleasant. While working with these companies, I have clearly observed that fresher engineers and junior engineers (below 5 years experience) have been quickly learning Go and writing brilliant piece of Go code when they get right mentoring and guidance, not just on Go, but also on SOLID principles and Clean architecture.

Whether Go is good for junior engineers or not

There are lot of people say that Go is not good for less-experienced people, and it’s mostly appealing for highly experienced engineers. Of course, Go might be really exciting for senior engineers and architects, who went through lot of experience with different technologies and struggled for solving some kind of real-world problems. But I don’t agree with the statement that Go is not good for less experienced people, based on my experience working with different teams. Regardless of whether you’re lesser experienced or highly experienced, it’s all about your understanding about programming, the baggage you have with your past experience and your attitude of unlearning and learning new things by breaking the baggage of the past, will decide everything, not just about learning and adopting Go, but everything about programming and architeture. The great thing about fresher and junior-level engineers are that they don’t have too much baggage from their past experiences with previous programming languages and with different architectures. Thus they are ready to accept to anything new and to the evolutionary way of programming because of the conditioning is very less. But the conditioning is very high among very senior people if their foundation of knowledge is not good. I have seen that a lot of people have been learning languages and technologies by emphasising the syntax of the language and technologies, rather than understanding the soul and concept of the programming languages, technologies and architecture. For those people, they will always try to look for equivalents for what they have learned in other programming languages.

Because Go is very simple, minimalistic and pragmatic language, any engineer regardless of fresher, junior-level, or highly experienced, can quickly learn this language and can build production-ready apps very quickly. If you are a startup company who want to adopt Go for your next product, I highly recommend to hire few fresher and junior-level engineers, and if you could give right mentoring they may perform better than even very senior people. But at the same time, if you don’t give proper mentoring to them, they may adopt Go in wrong way based on their Google search. These younger engineers blindly believe in everything what they are getting from Google search results.

In generally speaking (although generalising things are not right), software engineers working in product engineering startups are more enthusiastic and passionate on learning, embracing and building brilliant applications with Go.

The experience of working with larger organisations

The culture of a larger organisation in India — both India headquartered companies and India development centre of USA headquartered organisations, are completely different than other organisations. My experience of working with larger organisations has been a mixed feeling of emotions. Sometimes it has been very pleasant, but in fewer times, it has been very painful.

Java Baggage: The biggest bottleneck to adopt Go in larger organisations

The Java baggage is the biggest challenge to embrace Go programming language in many of the larger organisations. Although most of the people are really enthusiastic to learn Go, there has been a huge resistance coming from two kind of software engineers.

The first set of people are those who believe that all the programming language innovations had ended with Java programming language design. Thus learning a new language must be the equivalents of what they have learned in Java, and an application architecture of Go must be the equivalents of what they have built with Java. I have had experiences where teams were first looking for an inversion of control (IoC) container in Go or a Spring Boot framework equivalent, or looking for how to write Gang of Four design patterns and enterprise patterns in Go, even before learning the basics and design philosophy of Go programming language. These people are very disappointed with Go’s user-defined type system especially on interface, concurrency programming model, and even the way we build HTTP servers in Go. The second of set of people are those who don’t have much interest moving to Go. They were asked moving to Go by their organisations. These people will always oppose against each of the Go programming language design philosophy and make arguments during the period of training and mentoring. Although these kind of people are a smaller audience, it will make problems to everyone in the same team. When you give training to a 15-20 member team, where problematic attitude from just two persons will make problem to everyone. In very fewer times, I was hurt and humiliated by these smaller audience because of their resistance to Go due to the Java baggage, during a mentoring program.

IMHO, larger organisations must identify the Java baggage as one of the challenges of adopting Go programming language within their organisations.

Although there are few people are having the problem of Java baggage, there are lot of good people, who are really enthusiastic to learn and build applications with Go. So it’s more of a problem of an individual rather than a organisational problem although organisational culture may have contributed shaping the attitude of people.

Keep in mind that everything in the world is evolving, including in software engineering and the way we architect software systems. Unlearning and breaking baggage of the past, and learning new things with a fresher mindset, is the great quality you can keep if you would like to become successful in software engineering. Everything in this world is evolving. Things are arising, staying for sometime and passing away when causes and conditions are right: this is the universal truth.

Understanding the Go philosophy is most important to embrace the Go programming language

Regardless of in which organisation you’re working, the most important thing is understand the Go philosophy. Evaluate Go to understand whether Go is suitable for your team, projects and organisation, or not. Programming languages, tools, frameworks or even architecture approaches, are designed for solving some kind of problems. Many of these languages and tools are evolutionary things, which were evolved from our past experiences and from the challenges we had faced to solve some kind of problems.

IMHO, the most exciting things of Go programming language are its user-defined type system and concurrency primitives, along with its simplicity and pragmatism. In Go’s user-defined types, states and behaviours are isolated, where you simply define the states within the body of user-defined data structure, and then behaviours are added later on by writing functions specifying with receiver types. User-defined types can be either value type or reference type, unlike other programming languages where it is always reference type. When you add behaviours to an existing type, you specify the receiver type either with value or pointer to make it as pass by value or pass by reference. Go breaks many of the dogmas, which we have seen in many of traditional object-oriented programming languages. There is no type hierarchy in Go so that inheritance of types is not provided in its type system, but you can make composition of types by using type embedding. This is a brilliant pragmatic feature in Go language. Go interfaces are implemented implicitly, which is really exciting. When you look these pragmatic features with a fresher mindset, you may feel lot of opportunities. But if you look into these things by comparing with enterprise Java background, then you may disappoint with this language. If you’re looking for another Java, you’re adopting Go for all the wrong reasons. Because you can still stick with Java, if your objective is to embrace another Java.

Don’t adopt any technology just for the sake of it

I have been seeing that a lot of organisations are adopting Go just for the sake of adopting a new mainstream language. This is not the right way. Everything in this world is a perceptual reality and thus we can’t say that Go is a silver bullet, or Go is a great language. If Go is great for someone else, it doesn’t mean that it’s good for everyone else. So understand the language and it’s design philosophy. Then if you feel that it’s okay for your team and to projects, just adopt it in Go way, rather than writing Go applications in Java way or the way of philosophy which we had used with another language.

Recommendations for adopting Go

I don’t believe in providing general advice on anything because everything are contextual things and thus we can’t generalise things, and blindly following someone else’s advice doesn’t make any sense. Having said that I would like to provide some recommendations from my vast consulting experience and also based on my philosophy of writing software systems.

Don’t use frameworks unnecessarily, use standard library packages, and then use third-party libraries over frameworks for extending the functionalities

I have seen that some companies are adopting Go by starting with a Google search with a search text of “Golang+web frameworks” before learning anything on Go as a language or without understanding the Go philosophy. This behaviour, which I have mostly seen with startup companies. Based on the search results and analysing the number of stars in Github for each framework, these companies finally choose a web framework, mostly with gin or something similar. Once they choose a web framework, they start to learn how to use that web framework even before learning Go. Then the development of projects in Go using that web framework is going on. So you can imagine the quality of Go code, which they write with these projects. They adopt Go for the sake of adopting of Go. That’s it.

My advice is it’s better to stick with the Go philosophy of simplicity where you can start with standard library packages, and then when you need to extend some functionalities or want to provide more capabilities, use the essential third-party libraries instead of frameworks. The problems with frameworks are that they are highly opinionated implementations where your freedom for writing things in your own way would be bit difficult. For example, for building a RESTful API, you can stick on net/http package with a third-party http request router package like Gorilla Mux, along with other libraries like jwt-go for JWT, viper for configuration solution, etc. Along with the essential third-party libraries, build some reusable piece of code, which can be used across the Go projects within your organisation. If you could create reusable components like generic http handler, an HTTP middleware stack to easily add middlewares along with a bunch of reusable middlewares, then it would be useful across the projects. When you don’t use frameworks, you will get full freedom to write idiomatic Go code with greater testability. But if you believe that using a framework gives you lot of benefits without affecting the quality and testability of Go code, then it’s okay to use a framework, but that should be the last resort.

Isolate dependencies, communicate over contracts — interfaces, and inject the dependencies

Regardless of which programming language you’re using, it’s very important to write software systems based on loosely-coupled design. That’s more important than anything else. As the Clean architecture emphasises, the source code dependencies can only point inwards. Nothing in an inner circle can know anything at all about something in an outer circle.

When it comes to Go, leverage the user-defined type system to write loosely coupled, highly testable, clean Go code. When you write anything in Go, ask these questions to yourself:

• What’s the single responsibility of writing this component?
• What’re the dependencies of this component?

Then write code based on single responsibility principle (SRP), and then identify all the dependencies. For example, when you write an HTTP handler, the single responsibility is just serves the HTTP requests — just extracts the information from the request object and then write headers and bodies into HTTP response. All other things like business services and logging are dependencies. When you write business services, then things like persistence and logging are dependencies. You can isolate the dependencies by creating struct types where dependencies can be created as properties with interface types, as shown below

// service implements the Order Service
type service struct {
   repository ordersvc.Repository
   logger     log.Logger
}

// NewService creates and returns a new Order service instance
func NewService(repository ordersvc.Repository, logger log.Logger) ordersvc.Service {
   return &service{
      repository: repository,
      logger:     logger,
   }
}

In the above code block, ordersvc.Repository is an interface so that we can easily inject the concrete implementation of that contract type when we create instances by calling the factory method NewService.

Struct Methods Vs Free Functions

In Go, you can write free functions and struct methods. Then when to use struct methods over free functions, and free functions over struct methods. Here’re simple recommendations (please understand the context as well):

• If package level variables are used by free functions, then refactor with:

— Create structs by making package level variables as properties

— Add methods to the struct type by taking the code wrote in free functions

• Use structs for isolating dependencies by making dependencies as properties with interface types. Inject the dependencies from the place where you create instances of those struct types.
• Write free functions where you don’t references any state and don’t create dependent objects.
• In short, create structs for defining states and dependencies.

People say that don’t use global variables. I would say even don’t use too much package level variables. If there are package variables used by lot of free functions, then this might be a candidate for refactoring code with structs.

Recommending Go kit for Clean architecture

Go kit (http://gokit.io) is a collection of Go packages (libraries) that help you build robust, reliable, maintainable microservices. It was originally conceived as a toolkit to help larger organizations adopt Go as an implementation language. The good thing about Go kit is that it’s lightly opinionated unlike a full-fledged framework (again, I don’t recommend frameworks), and was designed for interoperability that works with different infrastructures, message encoding formats and transport layers.

Although Go kit is mainly labelled as a toolkit for building Microservices in Go, I recommend this for a different thing: improving the architecture quality of your applications. Besides a toolkit for building microservices, it encourages good design principles for your application architecture within a service. Go kit helps you embrace SOLID design principles, domain-driven design (DDD), and clean architecture. Go kit is well-suited for building elegant monoliths.

In Go kit, you write business services with pure business logic without coupling to anything (Service layer in Go kit). These service methods can be converted into RPC methods (Endpoint layer in Go kit), again, without tightly coupled with message encoding formats and transport protocols. And when you expose your services to outside world, you can use any of the transportation protocols (using Transport layer of Go kit) and with a message encoding format. In Go kit, a single service can be exposed with multiple transport protocols, including HTTP, gRPC, NATS, AMQP and Thrift. The middleware concept of Go kit for implementing decorators into services and endpoint domains are exceptionally brilliant.

Although Go kit is encouraging good design philosophy for your Go applications, Go kit based application architecture may not be good for everyone. It’s not good for developer productivity so that it makes problem for smaller teams who want to build their applications very quickly. And, IMHO, Go kit based architecture is not well suited for event-driven, streaming architectures when you build large-scalable distributed systems, although Go kit supports transport protocols like NATS. As a distributed systems architect, I’m an advocate for leveraging event-driven and streaming architectures for building massively scalable distributed systems.

Microservices architecture is all about functional decomposition, but not about containerisation and some tooling

In my mentoring experience, the most strange thing, which I have observed that majority of teams believe that Microservices architecture is all about containerisation and thus companies are chasing containerisation to be called their systems as Microservices. Few people now say that containerisation is not just adequate to be called a system as Microservices, you should also use a service mesh platform like Istio, in order to call a system as Microservices. These observations are completely wrong. I don’t know what kind of benefits they may get if some systems are being labelled as Microservices. I don’t believe in some labelling. Labelling doesn’t make any sense. What make sense is don’t become a poster child of someone else’s intellectual thoughts and solve your problems in your own way. Once a startup founder told me that he needs to be called the product as Microservices based system to get VC funds.

Among the Java community, some people believe that just using Spring Boot framework is Microservices. In the similar fashion, some Go developers believe that just using Go kit or Go Micro is Microservices. Although Go kit and Go Micro are helpful for building distributed systems, we can’t say that just using any of the one is Microservices.

In reality, Microservices is an architectural style for building distributed systems based on the idea of functional decomposition so that you can easily scale your systems by using independently deployable, small, modular services. Microservices are small, autonomous services that work together. You may probably built these services around the concept of Bounded Context in Domain-Driven Design. These autonomous services may communicate to each other by using an RPC mechanism or by using a messaging system with an event-driven architecture.The problem with real-world Microservices based architecture, is that it may ends with lot of complexity because most of the transactions may span into several services with persistence, where handling distributed transactions with a distributed saga pattern, and for querying data, it may need to aggregate data from multiple data stores (because data is scattered into several data stores), will make extreme complexity to the systems.

Embrace a middle-ground solution architecture for your systems

What I recommend for building distributed systems is to go for a middle-ground solution based on the problem domain instead of blindly adopting a Microservices based architecture. The most important thing is to solve your problem domain by providing a context-driven architecture, for which you can give any name. The context-driven architecture around the problem domain within the limitation of your organisation is the best architecture. Your company may not be Uber, then following the architectures being used at Uber (their scalability challenges are different than yourself) for your projects don’t make any sense.

Observability must be the number one priority for running distributed systems

Regardless of which architecture label you are using for building your distributed systems, you must properly add Observability components into your systems. Logging, and collecting distributed traces and metrics, must be the basic prerequisite for running any distributed system. OpenTelemetry, a CNCF project, is an observability framework (OpenCensus and OpenTracing have merged to form OpenTelemetry).

Recommending NATS, NATS JetStream (next-generation of NATS Streaming) and NGS for building distributed systems with event-driven, streaming architectures, and for building systems on IoT and Edge

In my vast consulting experience, I’ve realised that Kafka is more popular among Go teams when there is a need for messaging and event streaming platform. Although a lot of teams are using Kafka by understanding what it does, most of the teams are using Kafka without understanding its capabilities and limitations, and without evaluating whether using Kafka make sense for their systems. Some organisations are using Kafka for the sake of using it, similar like a lot of organisations are using microservices architecture for the sake of it based on hype and buzzword.

Although Kafka is a great distributed event streaming platform, I would like to recommend NATS ecosystem for building distributed systems in Go because of the simplicity, versatility of two capabilities — simple messaging for buffer at server and forward to consumers with industry leading performance (basic NATS) and event streaming platform with persistence (NATS Streaming/NATS JetStream), and the evolution of NATS ecosystem (NATS 2.0) for handling security concerns for building next-generation of systems on IoT and Edge. When Edge is evolving into mainstream, NATS and NATS JetStream along with NGS service, can play a major role for building next-generation of connected systems.

NATS, is a Cloud Native Computing Foundation (CNCF) project that comes with two messaging options. The first messaging option is a simple NATS server that forwards the published messages to consumers without any persistence for published messages, which is coming with an industry leading performance. For some kind of applications, the basic NATS platform is adequate where you don’t need an event streaming platform and can gain the performance benefits as well. The second messaging option is available with NATS Streaming, which is an extremely performant, lightweight reliable streaming platform built on the basic NATS. When you publish messages with NATS Streaming, the published messages persist into a customisable storage so that we can replay the messages into consumers that provides “At-least-once-delivery” model with the capability of ACK messages for publishers and subscribers, publisher rate limiting and rate matching/limiting per subscriber. Both basic NATS and NATS Streaming are written in Go.

Although NATS Streaming is extremely performant and lightweight, it is not much powerful as Kafka in-terms of some of the capabilities and maturity. But if you prefer simplicity over complexity, NATS Streaming is still a good choice for some kind of applications. But the next-generation of NATS Streaming for the era of NATS 2.0, known as NATS JetStream, is a great alternative to Kafka. NATS JetStream is coming with a lot of capabilities, which includes the support for creating both push based and pull based message consumers. I predict that NATS JetStream will be very popular within couple of years.

Unit testing is all about specifying the behaviour before writing production code

A lot of software engineers believe that unit testing is about testing something else which you do after writing production code. This is not right. The soul of the unit tests are specifying the behaviour rather than testing anything else, about what you’re going to develop. Then you can write your production code to verify the behaviour, which you have specified in the unit tests. If you believe that unit testing is about testing something else, it’s better to adopt BDD style tests (specs), which give emphasise into behaviour rather than testing. In Go, Ginkgo is a decent library for writing BDD style tests (specs), along with its preferred matcher library Gomega.

A lot of people think that writing unit testing methodology is just about using some testing frameworks and mocking frameworks. But when you follow testing methodology along with either TDD or BDD, the most important thing is all about how to write clean and testable code. If your production code does not contain testable and clean code, it would be difficult to embrace the unit testing methodology. Thus, when you write code, do follow good software engineering principles like SOLID and Clean architecture, so that you can easily embrace the unit testing methodology with right spirit.

Happy Go coding to everyone!.

By the way, here’s my basic technology stack: