Tit Petric's Blog

Don’t optimize prematurely - but have a plan how you’re going to do it.With the services I write, a lot of the code may be clear, simple andreadable over actually optimizing for speed. Sometimes, the differencebetween that code, and the optimized code, is a short refactor.

One of the services I write, is a comments service for the Sloveniannational TV & Radio station, RTV Slovenia. On a typical day, the ElasticAPM transactions end up being something like this:

What this tells us, that w...

When it comes to logging in your Go services, the immediate package toreach for is the standard library log package.As it happens, you’ll outgrow the log package needs almost immediately,because the log package is able to send the relevant logs only to oneio.Writer stream, by default sent to os.Stderr.

There may be other needs that you may feel you want to address:

Structured logging that can be consumed by external services like Logstash, ELKSplit the log destination based on message se...

After a few years of writing JSON APIs for various purposes, with more
success than this article may lead you to believe, I have come to the
conclusion that the JSON support in the wider Go ecosystem is broken. And
it isn’t really Go’s fault, but fixing it so it plays nice with other
very common programming languages is something that hasn’t been given
much consideration to.

All the problems of JSON encoding and decoding stem from being strict about
the encoding/json package behavior, imagining...

Go is a program language which has basic concurrency syntax built in.
Maybe calling it basic isn’t exactly right - simplicity is the more
correct description. After all to run a function independent of the main
execution, all you have to do is prefix the invocation with the go
keyword - that function call will now live on it’s own goroutine.

For people not familiar with threading or concurrency in general, it may
take a while to figure out what the behaviour here is. After all, when
you run a f...

Chances are that if you’re writing Go code for a while now, your needs for command line arguments have grown beyond just what the standard library flag package is able to provide.

A quick list

To quickly list a few packages:

namsral/flag - add environment variable parsing, spf13/pflag - use posix/gnu-style --flags, urfave/cli - adds a default action and command actions, env aliases,… spf13/cobra - an interface for cli commands like git and go tools, spf13/viper - a larger...

With our background jobs interface in place, we can create an API that will queue our incoming data and flush it to the database at a defined frequency. Flushing our data in bulk will give us performance advantages.

The Queue

A basic Queue in Go is a slice. To extend the queue, you can call append, which allocates a larger slice if required. Since this is not thread safe, we need to use locking to ensure that the queue is modified safely.

// Queue provides a queuing structure for Incoming{}...

API services, especially low latency ones, will resort to background jobs to process data faster on ingesting it, and aggregating and flushing it to backend services in an optimal way.

Adding background jobs to our service

In our stats microservice, we issue insert queries for each request directly into the database. The database supports extended inserts, meaning we could flush thousands of rows at the same time, and actually increase throughput, since adding 10K rows to the database with a...

As we implemented our service that adds pageviews to the database, we realize an unfortunate circumstance. Each insert query takes about 12ms, which means our request rate, at least per single CPU core, is poor. We’re talking about 83 req/s poor.

But as we know that scaling isn’t linear and you shouldn’t judge a single request, we need to perform some stress testing to give us a better idea of where we are.

Now, the best practices of stress testing suggest that you need...

After we set up Elastic APM to log our request transaction, the following thing we wish to instrument are the SQL queries going to our database endpoint. Elastic APM provides instrumentation that wraps the database/sql driver, which produces an *sql.DB.

Extending DB connection

We already planned to produce a *sqlx.DB for this eventuality with the Connector field function in db.ConnectionOptions:

// Connector is an optional parameter to produce our // own *sql.DB, which is then wrapped in...