Service autodiscovery in Go with sleuth
Service discovery and
remote procedure calls (RPC) are big subjects. There are many
existing solutions that require varying degrees of infrastructure (e.g. message queues,
Consul) or architectural decisions (e.g. Erlang, Akka, etc.). In this post, I’ll
introduce a minimalistic Go library called sleuth
that creates an ad hoc
peer-to-peer network requiring almost no configuration and only a couple lines of code to existing services and clients.
sleuth
is a library that lets web services announce
themselves to each other and to other clients on a LAN. It allows peers to send requests to each other without
having to coordinate where each one lives and what port it uses. It works without an external process because
under the hood, it is powered by magic, *a.k.a.* ØMQ
using the Gyre port of
Zyre.
TL;DR If you already feel comfortable with these topics and just want to jump straight to usage, skip directly to the
sleuth
example. Or, if you just want to see how to convert a server and a client that use HTTP requests to communicate with each other into peers on asleuth
network, check out this pull request.
If you’re not familiar with what ØMQ is or why I’m calling it magic, here’s the top-level Wikipedia definition:
ZeroMQ (also spelled ØMQ, 0MQ or ZMQ) is a high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker. The library’s API is designed to resemble that of Berkeley sockets.
Introducing the problem
Microservices are self-contained services that expose only a small, focused API. In REST services, this usually means that a particular microservice will answer some of the endpoints of an API, delegated to it by a reverse proxy. The idea is that the full API is composed of multiple microservices residing behind the proxy and each one only deals with a well-defined set of concerns and its own data source.
I wrote sleuth
because I needed a group of services on the same LAN to be able to find and to make requests to each other within a larger application. Since the problem is a general one, sleuth
is completely agnostic about what web framework, if any, that peers are using. As long as a service can expose an http.Handler
, then it can be a sleuth
service. Client-only peers that only consume services are similarly trivial; sleuth
’s API accepts native http.Request
objects to make requests to peers on the network.
To understand the use case for sleuth
, I’ll propose two simple services that need to communicate.
Sample service 1: article-service
The first service is an article-service
and it answers requests to paths that
begin with /articles
. Its purpose is to serve the meta information for
articles.
Here is a representative request:
GET /articles/049cd8fc-a66b-4a3d-956b-7c2ab5fb9c5d HTTP/1.1
It produces this response:
HTTP/1.1 200 OK
Content-Length: 304
Content-Type: application/json
{
"success": true,
"data": {
"guid": "06500da3-f9b0-4731-b0fa-fbc6cbe8c155",
"byline": "Brian Gallagher",
"headline": "Yoda Is Dead but Star Wars' Dubious Lessons Live On",
"url": "http://nautil.us/blog/yoda-is-dead-but-star-wars-dubious-lessons-live-on",
"time": 1452734432
}
}
Sample service 2: comment-service
The second service is a comment-service
and its purpose is to return the
comments for an article. It answers requests to paths that begin with
/comments
.
For example, here is a request for the comments related to the article above:
GET /comments/06500da3-f9b0-4731-b0fa-fbc6cbe8c155 HTTP/1.1
And the comment-service
responds:
HTTP/1.1 200 OK
Content-Length: 232
Content-Type: application/json
{
"success": true,
"data": [
{
"guid": "d7041752-6854-4b2c-ad6d-1b48d898668d",
"article": "06500da3-f9b0-4731-b0fa-fbc6cbe8c155",
"text": "(omitted for readability)",
"time": 1452738329
}
]
}
The article service needs comments
So what if the article-service
accepts a query string parameter includecomments
which denotes whether an article’s metadata should include its comments?
GET /articles/049cd8fc-a66b-4a3d-956b-7c2ab5fb9c5d?includecomments=true HTTP/1.1
Here is the response that would result if the article-service
had a way to communicate with the comment-service
:
HTTP/1.1 200 OK
Content-Length: 533
Content-Type: application/json
{
"success": true,
"data": {
"guid": "06500da3-f9b0-4731-b0fa-fbc6cbe8c155",
"byline": "Brian Gallagher",
"headline": "Yoda Is Dead but Star Wars' Dubious Lessons Live On",
"url": "http://nautil.us/blog/yoda-is-dead-but-star-wars-dubious-lessons-live-on",
"time": 1452734432,
"comments": [
{
"guid": "d7041752-6854-4b2c-ad6d-1b48d898668d",
"article": "06500da3-f9b0-4731-b0fa-fbc6cbe8c155",
"text": "(omitted for readability)",
"time": 1452738329
}
]
}
}
Naive implementation using HTTP requests
Since sleuth
can work with pretty much every Go web framework because all it needs is an http.Handler
to share services, I’ll use one of the more popular web packages, Gorilla, for URL routing.
The list of articles comes from my saved articles on Hacker News. The comments are the top-level comments for each of those articles. I used the Hacker News API on Firebase, now defunct, to save them.
Note that these examples are simplified and I’ll hold all of the data in memory in a map
. Obviously, in the real world, the data would reside a separate database of some sort for each service. I’ll use the init
function to populate the data from a local file for each service.
Both because API responses should be uniform and because services need to know how to read the data output by one another, I’ll create a shared types
package to define API responses.
package types
// Article holds the metadata for an article.
type Article struct {
GUID string `json:"guid"`
Byline string `json:"byline"`
Comments []*Comment `json:"comments,omitempty"`
Headline string `json:"headline"`
URL string `json:"url"`
Timestamp int64 `json:"time"`
}
// ArticleResponse is the format of article-service responses.
type ArticleResponse struct {
Success bool `json:"success"`
Message string `json:"message,omitempty"`
Data *Article `json:"data,omitempty"`
}
// Comment holds the text and metadata for an article comment.
type Comment struct {
GUID string `json:"guid"`
Article string `json:"article"`
Text string `json:"text"`
Timestamp int64 `json:"time"`
}
// CommentResponse is the format of comment-service responses.
type CommentResponse struct {
Success bool `json:"success"`
Message string `json:"message,omitempty"`
Data []*Comment `json:"data,omitempty"`
}
Implementing comment-service
Since it does not depend on being able to access any other service, I’ll start with the comment-service
. To simplify things, it will only support GET
requests like the one shown above and all the data will be stored in a flat file.
Here is a first pass at its implementation.
package main
import (
// Standard library imports are omitted for readability.
// See GitHub for complete source code.
"github.com/afshin/sleuth-example/types"
"github.com/gorilla/mux"
)
var data = make(map[string][]*types.Comment) // Key is article GUID.
func init() {
// Data loading code is omitted for readability.
// See GitHub for complete source code.
}
func handler(res http.ResponseWriter, req *http.Request) {
log.Println("GET " + req.URL.String())
response := new(types.CommentResponse)
guid := mux.Vars(req)["guid"]
if comments, ok := data[guid]; ok {
response.Data = comments
response.Success = true
res.WriteHeader(http.StatusOK)
} else {
response.Success = false
response.Message = guid + " not found"
res.WriteHeader(http.StatusNotFound)
}
output, _ := json.Marshal(response)
res.Header().Set("Content-Type", "application/json")
res.Write(output)
}
func main() {
router := mux.NewRouter()
router.HandleFunc("/comments/{guid}", handler).Methods("GET")
fmt.Println("ready...")
http.ListenAndServe(":9871", router)
}
The comment-service
implementation is straightforward: it reads the contents of data.json
, unmarshals the data and stores it in a local map
in memory which is grouped by article GUID. When GET
requests come in, it tries to locate the comments for a given GUID. If they are found, it returns a successful response, if they are not found it returns a 404.
Implementing article-service
Like the comment-service
, the article-service
also has its data stored in a flat file. If the includecomments
query string parameter is set to true
, its handler makes an HTTP request to find the comments for a given article.
package main
import (
// Standard library imports are omitted for readability.
// See GitHub for complete source code.
"github.com/afshin/sleuth-example/types"
"github.com/gorilla/mux"
)
const commentsURL = "http://localhost:9871/comments/%s"
var (
client = new(http.Client)
data = make(map[string]*types.Article) // Key is article GUID.
)
func init() {
// Data loading code is omitted for readability.
// See GitHub for complete source code.
}
func getData(guid string, includeComments bool) (article *types.Article) {
datum, ok := data[guid]
if !ok {
return
}
// Data source is immutable, so copy the data.
article = &types.Article{
GUID: datum.GUID,
Byline: datum.Byline,
Headline: datum.Headline,
URL: datum.URL,
Timestamp: datum.Timestamp}
if !includeComments {
return
}
url := fmt.Sprintf(commentsURL, guid)
req, _ := http.NewRequest("GET", url, nil)
if res, err := client.Do(req); err == nil {
response := new(types.CommentResponse)
if err := json.NewDecoder(res.Body).Decode(response); err == nil {
article.Comments = response.Data
}
}
return
}
func handler(res http.ResponseWriter, req *http.Request) {
log.Println("GET " + req.URL.String())
response := new(types.ArticleResponse)
guid := mux.Vars(req)["guid"]
include := strings.ToLower(req.URL.Query().Get("includecomments"))
if article := getData(guid, include == "true"); article != nil {
response.Data = article
response.Success = true
res.WriteHeader(http.StatusOK)
} else {
response.Success = false
response.Message = guid + " not found"
res.WriteHeader(http.StatusNotFound)
}
output, _ := json.Marshal(response)
res.Header().Set("Content-Type", "application/json")
res.Write(output)
}
func main() {
router := mux.NewRouter()
router.HandleFunc("/articles/{guid}", handler).Methods("GET")
fmt.Println("ready...")
http.ListenAndServe(":9872", router)
}
Solving the problem with sleuth
The solution above presents a few big problems that sleuth
can solve:
-
What happens if the
comment-service
port or location changes? There either needs to be additional complexity in deployment configuration or an external service that manages communication. -
Relatedly, what if there are multiple instances of
comment-service
running? Either an external service (like a proxy) needs to reside between the two services or I need to write some custom logic. -
How can I guarantee that
article-service
won’t even start untilcomment-service
is available?
Before continuing, I assume that you’ve already installed libzmq
on your system or are using a Docker container that comes with Go and ØMQ. See sleuth
installation for more information.
The pull request that switches from HTTP to sleuth
is straightforward and only a few lines of code for each service. I’ll discuss these changes below.
Again, I’ll start with the comment-service
because it has no dependencies. First, I import sleuth
and create a global client
variable:
package main
import (
// Imports are omitted for readability.
"github.com/ursiform/sleuth"
)
var (
client *sleuth.Client
data = make(map[string][]*types.Comment) // Key is article GUID.
)
Next, I update the main
function to instantiate client
and join the sleuth
network as a service called comment-service
, which uses the Gorilla router
as its handler, because it conforms to the http.Handler
interface:
func main() {
router := mux.NewRouter()
router.HandleFunc("/comments/{guid}", handler).Methods("GET")
// In the real world, the Interface field of the sleuth.Config object
// should be set so that all services are on the same subnet.
config := &sleuth.Config{Service: "comment-service", Handler: router}
client, _ = sleuth.New(config)
fmt.Println("ready...")
http.ListenAndServe(":9871", router)
}
And that’s it, the comment-service
can now be accessed by the article-service
.
To update the article-service
, I make some similar changes. First, I import the library, change the commentsURL
to be a sleuth://
URL, and change the client
variable to be a *sleuth.Client
instead of an *http.Client
.
package main
import (
// Imports are omitted for readability.
"github.com/ursiform/sleuth"
)
const commentsURL = "sleuth://comment-service/comments/%s"
var (
client *sleuth.Client
data = make(map[string]*types.Article) // Key is article GUID.
)
And finally, I instantiate the client
variable, and as a nice-to-have, I’ll tell the article-service
to wait until it has found at least one instance of a comment-service
before it starts up:
func main() {
router := mux.NewRouter()
router.HandleFunc("/articles/{guid}", handler).Methods("GET")
// In the real world, the Interface field of the sleuth.Config object
// should be set so that all services are on the same subnet.
client, _ = sleuth.New(&sleuth.Config{})
client.WaitFor("comment-service")
fmt.Println("ready...")
http.ListenAndServe(":9872", router)
}
Crucially, the HTTP request logic does not need to change at all because sleuth
uses the same semantics for making HTTP requests. It accepts a native http.Request
object in a function called Do
just like the native http.Client
so it functions as a drop-in replacement for http.Client
requests.
Conclusion
If your system is already large enough that you have a custom message-queue based solution or are using an external service to manage microservices, then sleuth
may not be right for you. However, if you’re building up a set of services and want the flexibility to add them organically and have them find each other in an ad hoc network, then sleuth
will simplify building services that operate in a distributed environment.
sleuth
is a library, not a framework, and it’s compatible with any Go web framework that uses the native http.Handler
interface; so it is unobtrusive when it comes to your architecture decisions.
The example code for this tutorial is separated into two branches:
-
The
http
branch is the original implementation that makes HTTP requests between the two services and assumes that the comment service is on the same machine and its port never changes. -
The
master
branch branch solves this dependency by creating asleuth
network between the two services. -
Again, here is the pull request that switches from HTTP to
sleuth
.
The sleuth
API documentation is available on GoDoc.
And finally, the sleuth
GitHub repository contains the most recent version of the library.