It’s also remarkably extensible. You can write plugins in TypeScript, apply custom themes, build tools and web apps around it, and even extend it with Skills. The community around it is growing fast — check out awesome-opencode or opencode.cafe if you want to explore what people are building.

What I recently discovered, though, is that Opencode has a Go SDK (not to be confused with Opencode Go), and that’s what inspired this whole project. Let’s see what we can build with it.

Subscribe now

The Problem: Catching dumb mistakes before commit

I’m the type of developer who absolutely hates typos in code and stray debug print statements — and also the type who produces them constantly. Regular formatters and linters don’t always catch these things, and let’s be honest: other code reviewers aren’t exactly trustworthy either. Nobody really reads the code anymore.

There are tools like CodeRabbit, GitHub Copilot, and Graphite that can review your pull requests after the fact. But what if I want to run similar checks before committing my code? And what if I want those checks to be configurable?

The idea: run an AI-powered code review as a pre-commit hook, using a coding agent that’s already aware of the codebase.

Since I use Opencode daily, let’s build it with that.

Opencode Server

Here’s something a lot of people don’t realize: Opencode isn’t simply a TUI wrapper. It ships with a full server, and the TUI is just one client that talks to it. That means you can connect to the server from anywhere — your phone, a web browser, or a custom Go program.

You can start the server standalone on any port, optionally password-protected:

opencode serve --port 4096

This also means you can run multiple Opencode instances and agents simultaneously.

The Architecture

Here’s the high-level flow of what we’re building:

┌──────────────┐     ┌──────────────────┐     ┌────────────────┐
│  git commit  │────▶│  pre-commit hook │────▶│ Opencode Server│
│  (triggers)  │     │  (Go binary)     │     │                │
└──────────────┘     └──────────────────┘     └────────┬───────┘
                              │                         │
                              │  1. Get staged diff      │
                              │  2. Create session       │
                              │  3. Send diff + prompt   │
                              │  4. Parse JSON response  │
                              │  5. Pass / Fail commit   │
                              ▼                         ▼
                     ┌──────────────────┐     ┌────────────────┐
                     │  Terminal Output │     │   LLM (Opus)   │
                     │  issues / pass   │     │   via Opencode │
                     └──────────────────┘     └────────────────┘

The Go SDK

The Opencode Go SDK wraps the standard REST API that the server exposes. For our pre-commit hook, we only need three operations:

SDK Method Purpose Session.New Create a fresh review session Session.Prompt Send the diff and get the review back Session.Delete Clean up the session when we’re done

That’s it. Minimal surface area for a focused tool.

Crafting the Prompt

The pre-commit hook ships with a default prompt that asks the LLM to review for typos, unnecessary debug statements, security issues, bugs, and code style violations — essentially a lightweight CodeRabbit that runs locally.

The tricky part is that LLMs answer in an unstructured way by default, which is hard to parse programmatically. So we need to instruct the model to return its answer as strict JSON and hope it actually follows those instructions.

Here’s the full default prompt, which includes the staged git diff:

out, err := exec.Command("git", "diff", "--cached", "--diff-algorithm=minimal").Output()
if err != nil {
    fatal("unable to get git diff: %v", err)
    return
}
diff := strings.TrimSpace(string(out))
if diff == "" {
    fmt.Println("no staged changes to review")
    return
}

prompt := `You are a code reviewer. Review the staged git diff below.

Look for typos, unnecessary debug statements, bugs, security issues, and code style problems.

Respond ONLY with a JSON object (no markdown fences, no extra text):

{
  "status":"pass|fail|warn",
  "issues": [
    {"file":"...","line":0,"severity":"error|warning|info","message":"..."}
  ]
}

If everything looks good, return {"status":"pass","issues":[]}

` + "```git diff:\n" + diff + "\n```"

The idea is that users can later customize this prompt per-repo to match their own conventions and priorities.

Talking to Opencode

With the prompt ready, we make a sequence of SDK calls. The flow is straightforward — create a session, send the prompt, read the response, then clean up:

const baseURL = "http://127.0.0.1:4096"

client := opencode.NewClient(
    option.WithBaseURL(baseURL),
    option.WithMaxRetries(1),
)

ctx, cancel := context.WithTimeout(context.Background(), 1*time.Minute)
defer cancel()

// Create a session
session, err := client.Session.New(ctx, opencode.SessionNewParams{
    Title: opencode.F("pre-commit review"),
})
if err != nil {
    fatal("unable to create session: %v", err)
    return
}

// Always clean up
defer client.Session.Delete(context.Background(), session.ID, opencode.SessionDeleteParams{})

fmt.Fprintln(os.Stderr, "Reviewing staged changes...")

// Send the prompt
resp, err := client.Session.Prompt(ctx, session.ID, opencode.SessionPromptParams{
    Parts: opencode.F([]opencode.SessionPromptParamsPartUnion{
        opencode.TextPartInputParam{
            Type: opencode.F(opencode.TextPartInputTypeText),
            Text: opencode.F(prompt),
        },
    }),
})
if err != nil {
    fatal("unable to prompt: %v", err)
    return
}

// Extract text from response parts
var text string
for _, part := range resp.Parts {
    if tp, ok := part.AsUnion().(opencode.TextPart); ok {
        text += tp.Text
    }
}

A one-minute timeout is more than enough for a typical diff review.

Parsing the Result

The response comes back as a JSON string (if the model cooperated), which we deserialize into simple Go structs:

type Review struct {
    Status string  `json:"status"`
    Issues []Issue `json:"issues"`
}

type Issue struct {
    File     string `json:"file"`
    Line     int    `json:"line"`
    Severity string `json:"severity"`
    Message  string `json:"message"`
}

Then we parse and display the results:

var review Review
if err := json.Unmarshal([]byte(text), &review); err != nil {
    fatal("unable to parse json: %v\nraw response:\n%s", err, text)
    return
}

fmt.Printf("Review status: %s\n", review.Status)
for _, issue := range review.Issues {
    fmt.Printf("  [%s] %s:%d — %s\n", issue.Severity, issue.File, issue.Line, issue.Message)
}
if len(review.Issues) == 0 {
    fmt.Println("No issues found!")
}

if review.Status == "fail" {
    os.Exit(1)
}

If the status is "fail", we exit with code 1, which tells Git to abort the commit. Simple and effective.

Installing the Hook

There are fancier ways to manage pre-commit hooks — frameworks like pre-commit or tools like Husky. But a pre-commit hook is really just an executable file, so manual installation is trivial.

Drop the following into .git/hooks/pre-commit:

#!/usr/bin/env bash
exec opencode-pre-commit

Make sure the Go binary (opencode-pre-commit) is somewhere on your $PATH, and you’re set.

Testing It Out

To see everything in action:

1. Start the Opencode server:

opencode serve --port 4096

2. Install the hook in any repo you want to test:

vim .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

3. Make a deliberately bad change (introduce a typo, leave a hardcoded secret, add a debug statement) and try to commit:

git add .
git commit -m "hello people"

If the AI reviewer catches something, you’ll see output like this:

Reviewing staged changes...
Review status: fail
  [error] oc/main.go:15 — Hardcoded secret/API key in source code — remove and rotate
  [warning] oc/go.mod:3 — Go version 1.25.5 may cause build failures for older toolchains
  [info] oc/main.go:16 — baseURL is hardcoded — consider an ENV override for flexibility
exit status 1

The commit is blocked. Fix the issues, re-stage, and try again.

Things to Keep in Mind

I used the Opus 4.6 model for testing and it consistently respected the JSON format and produced useful reviews. Other coding-focused models should work well too.

That said, a few caveats worth noting. Like any LLM output, the results aren’t deterministic — you may get slightly different feedback each time. You might also hit API rate limits depending on your usage. And while the model is surprisingly good at catching real issues, it can occasionally flag things that don’t matter. Treat it as a helpful second pair of eyes, not an infallible gatekeeper.

Subscribe now

Links

• Opencode: opencode.ai

• Opencode Go SDK: github.com/anomalyco/opencode-sdk-go

• Source Code: github.com/plutov/opencode-pre-commit

• Watch on YouTube

I was hoping that I could try the latest version 3, but it’s not yet released. But that shouldn’t be a problem as v2 is more than enough to create a lightweight desktop app.

Short recap of how Wails works, I won’t Go into the details, you can check it yourself:

A Wails application is a standard Go application, with a webkit frontend. It is possible to bind Go methods to the frontend, and these will appear as JavaScript methods that can be called, just as if they were local JavaScript methods.

Quite handy right? You can write desktop applications in Go. In terms of the frontend the following options are available:

• Svelte

• React

• Vue

• Preact

• Lit

• Vanilla

Also, it’s not Electron, it’s more lightweight and fast!

Could we vibe-code our 2FA App in under an hour with AI assistance? - Probably.

But would we actually learn how 2FA works? - No.

So let’s build this properly, understanding every piece, that’s the way of this newsletter.

Subscribe now

How 2FA actually works

We all used 2FA apps.

Before we write a single line of code, we need to understand what we’re actually building. Most tutorials skip this part and just import a library. But that’s not how we learn.

When you enable 2FA, the server generates a random secret - just a long random string. This secret is shared ONCE between the server and your authenticator app, usually via QR code, in base32 format.

Example Secret JBSWY3DPEHPK3PXP (without padding)

Both the server and your app now have this secret. The server stores it in their database, your app stores it locally. This is the ONLY time they communicate about the secret.

The first “T” in TOTP stands for “Time-based”. TOTP usually refreshes every 30s.

Current Time: 1707912345 (Unix timestamp in seconds)
Divide by 30: 56930411 (This is our counter)

Every 30 seconds, this counter increments by 1.

Why 30 seconds? It’s a balance to get a counter.

Now we need to combine the secret and the counter in a way that:

• Can’t be reversed (you can’t figure out the secret from the code)

• Is deterministic (same inputs = same output)

• Is practically impossible to predict future codes

This is where HMAC-SHA1 comes in. It’s a cryptographic hash function.

Secret Key + Counter → HMAC-SHA1 → 20-byte hash
Example:
Secret: JBSWY3DPEHPK3PXP
Counter: 56930411
Hash: 1f8698690e02ca16618550ef7f19da8e945b555a

We have a 20-byte hash, but we need a 6-digit number. Can we just take the first 6 digits? No! That would be predictable.

Instead, we use the “dynamic truncation” - a fancy way of saying we pick a spot in the hash based on the hash itself:

Hash: 1f 86 98 69 0e 02 ca 16 61 85 50 ef 7f 19 da 8e 94 5b 55 5a

Last byte: 5a → Last 4 bits: 0xa (10 in decimal)
So we start at position 10 (0-indexed)

Extract 4 bytes starting at position 10:
Bytes: 50 ef 7f 19
Binary: 01010000 11101111 01111111 00011001
Integer: 1358806809

Take last 6 digits: 806809

The complete flow

┌─────────────────────────────────────────────────────┐
│                    USER'S APP                       │
│  Secret: JBSWY3DPEHPK3PXP (stored locally)          │
└─────────────────────────────────────────────────────┘
                        ↓
        ┌───────────────────────────────┐
        │  Current Time: 1707912345     │
        │  Counter: 1707912345 / 30     │
        │         = 56930411            │
        └───────────────────────────────┘
                        ↓
        ┌───────────────────────────────┐
        │  HMAC-SHA1(secret, counter)   │
        │  = 20-byte hash               │
        └───────────────────────────────┘
                        ↓
        ┌───────────────────────────────┐
        │  Dynamic Truncation           │
        │  → 6-digit code: 806809       │
        └───────────────────────────────┘
                        ↓
┌─────────────────────────────────────────────────────┐
│                   SERVER                            │
│  (Does EXACT same calculation)                      │
│  Secret: JBSWY3DPEHPK3PXP (in database)             │
│  Code: 806809 ✅ Match!                             │
└─────────────────────────────────────────────────────┘

What about the QR code?

So how does the secret get from the server to your app? QR codes!

The decode to the URL of the following format:

otpauth://totp/GitHub:john@example.com?secret=JBSWY3DPEHPK3PXP&issuer=GitHub

Let’s break this down:

• otpauth:// - URI scheme (tells apps this is for authentication)

• totp - Type (TOTP vs HOTP which is counter-based)

• GitHub:john@example.com - Label (shown in your app)

• secret=JBSWY3DPEHPK3PXP - The shared secret

• issuer=GitHub - Who’s providing the service

Optional parameters you might see:

• algorithm=SHA1 (or SHA256, SHA512)

• digits=6 (or 8)

• period=30 (time window in seconds)

Some applications let you set up the 2FA manually and show you the secret instead.

Now knowing the theory, let’s start building our app. it could be a CLI app, but why not to have a nice looking desktop app using Wails?

Project setup

We are only implementing the client side in this video, and will use external service as a server, for example totp.app

I chose Vue.js and Typescript.

go install github.com/wailsapp/wails/v2/cmd/wails@latest

wails init -n mfaclient -t vue-ts
cd mfaclient
wails dev

Wails will generate some code for us, not much luckily, it’s still easy to navigate:

mfaclient/
├── app.go              # Main application logic
├── main.go             # Entry point
├── totp.go             # TOTP generation logic
├── storage.go          # Encrypted storage for accounts
├── frontend/App.vue.   # Our Vue frontend
└── wails.json

Go implementation of TOTP

The implementation will follow the algorithm described above and is quite straightforward. As you can see we only use the Go’s standard library.

totp.go

package main

import (
	"crypto/hmac"
	"crypto/sha1"
	"encoding/base32"
	"encoding/binary"
	"fmt"
	"strings"
	"time"
)

func GenerateTotp(secret string) (string, int, error) {
	now := time.Now()

	counter := getCounter(now)

	hash, err := calculateHash(secret, counter)
	if err != nil {
		return "", 0, err
	}

	code := truncate(hash)

	remaining := 30 - (int(now.Unix()) % 30)

	return fmt.Sprintf("%d", code), remaining, nil
}

func getCounter(t time.Time) uint64 {
	return uint64(t.Unix()) / 30
}

func calculateHash(secret string, counter uint64) ([]byte, error) {
	secret = strings.ReplaceAll(secret, " ", "")
	secret = strings.TrimRight(secret, "=")
	secret = strings.ToUpper(secret)

	key, err := base32.StdEncoding.WithPadding(base32.NoPadding).DecodeString(secret)
	if err != nil {
		return nil, err
	}

	buf := make([]byte, 8)
	binary.BigEndian.PutUint64(buf, counter)

	h := hmac.New(sha1.New, key)
	h.Write(buf)

	return h.Sum(nil), nil
}

func truncate(hash []byte) int {
	offset := hash[len(hash)-1] & 0x0f

	code := int(hash[offset]&0x7f)<<24 |
		int(hash[offset+1])<<16 |
		int(hash[offset+2])<<8 |
		int(hash[offset+3])

	return code % 1000000
}

The only hacky part that is there is the truncate function, which does some bit manipulation to get an integer.

Having this part coded, we can start thinking about how to store the 2FA accounts locally. And there are endless options here, we could store them in the database, could encrypt them and so forth. To make it simple for now, let’s store them into a plain JSON file locally.

storage.go

package main

import (
	"crypto/rand"
	"encoding/hex"
	"encoding/json"
	"os"
)

type Account struct {
	Id     string `json:"id"`
	Issuer string `json:"issuer"`
	Secret string `json:"secret"`
}

type Storage struct {
	Filepath string
}

func generateID() string {
	bytes := make([]byte, 8)
	rand.Read(bytes)
	return hex.EncodeToString(bytes)
}

func (s *Storage) save(accounts []Account) error {
	data, err := json.Marshal(accounts)
	if err != nil {
		return err
	}

	return os.WriteFile(s.Filepath, data, 0o600)
}

func (s *Storage) LoadAccounts() ([]Account, error) {
	data, err := os.ReadFile(s.Filepath)
	if err != nil {
		if os.IsNotExist(err) {
			return []Account{}, nil
		}

		return nil, err
	}

	var accounts []Account
	json.Unmarshal(data, &accounts)
	return accounts, nil
}

func (s *Storage) AddAccount(a Account) error {
	accounts, err := s.LoadAccounts()
	if err != nil {
		return err
	}

	a.Id = generateID()
	accounts = append(accounts, a)
	return s.save(accounts)
}

func (s *Storage) DeleteAccount(id string) error {
	accounts, err := s.LoadAccounts()
	if err != nil {
		return err
	}

	filtered := []Account{}
	for _, acc := range accounts {
		if acc.Id != id {
			filtered = append(filtered, acc)
		}
	}

	return s.save(filtered)
}

Hooking up with frontend

Once backend Go part is ready we can start exposing these methods as bindings from app.go and use them in frontend/App.vue

I will not share the whole code, you can see the link at the end of the article.

But developing the Wails frontend feels very similar to developing in Vue on the web, here is the snippet from our App.vue:

<script lang="ts" setup>
import { ref, onMounted, onUnmounted } from "vue";
import { GetAccounts, AddAccount, DeleteAccount } from "../wailsjs/go/main/App";
import { main } from "../wailsjs/go/models";

const accounts = ref<main.AccountWithCode[]>([]);
const error = ref("");
const showForm = ref(false);
const issuer = ref("");
const secret = ref("");
const addError = ref("");
const copiedId = ref("");

// ...

Testing

Secrets are always distributed in base32 format, so let’s generate one random first:

openssl rand -hex 10 | base32

Wails has a nice dev mode where it compiles all the parts of the app and has a hot reload for all components, be it backend or frontend.

wails dev

Now we can see our shiny app is working and generating the valid TOTP codes. We confirm it by using the same secrets on totp.app

We could also test it on a real service, but I suggest not trying it yet with your real accounts. Instead, you can run something like Keycloak locally.

docker run -p 8080:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:latest start-dev

Conclusion

Obviously this can be compiled so we have everything in a single binary.

wails build

This app needs many improvements such as: better UI, more secure account storage, support for different algorithms.

Otherwise Wails has proven once again to be a simple library to build lightweight desktop apps.

Links:

• Apps built with Wails

• Source code

Subscribe now

Intro

Building services that sit on top of a Postgres replication slot and continuously stream data into Elasticsearch is a great way to get low‑latency search without hammering your primary database with ad‑hoc queries. But as soon as traffic ramps up, these services become a stress test for Go’s memory allocator, garbage collector, and JSON stack.

This post walks through optimizations applied to a real-world service that:

• Connects to a Postgres replication slot

• Transforms and enriches the change events

• Uses Elasticsearch’s bulk indexer to index and delete documents

The constraints: the service cannot stop reading from the replication slot for long (or Postgres disk will grow), and it cannot buffer unbounded data in memory (or Go’s heap will). The goal is to keep latency and memory stable under sustained high volume.

The core problem: unbounded streams under tight constraints

Replication slots are relentless: as long as your primary receives writes, the slot will keep producing changes. If your consumer slows down, Postgres has to retain more WAL segments, increasing disk usage on the database server. If your consumer tries to “just buffer more” in memory, the heap will balloon and garbage collection will kick in more frequently, stealing CPU from useful work.

In this setup, you typically have three competing forces:

• Backpressure from Elasticsearch bulk indexing

• The continuous stream of changes from the replication slot

• The Go runtime’s allocator and garbage collector trying to keep up with allocations in your hot path

The design work is about turning this into a stable flow: limit in-flight work, keep memory usage predictable, and reduce per-message overhead.

JSON performance: why switch from encoding/json to jsoniter

One of the earliest hot spots in services like this is JSON encoding/decoding for documents going into Elasticsearch. The standard library’s encoding/json is correct and convenient, but it trades some performance for safety and reflection-based flexibility.

High-volume services often switch to jsoniter (github.com/json-iterator/go) for a few reasons:

• Faster encoding/decoding for common patterns

• Less reflection overhead when configured with codegen or field caching

• Better throughput when serializing large batches of similar structs

The wins are most visible when:

• You are serializing many small documents at high frequency (typical for bulk indexing)

• You can keep your types stable and avoid excessive use of interface{} and maps

• You care about reducing allocations in the JSON path and shaving microseconds off each object

However, replacing encoding/json is never just a drop-in optimization; it changes behavior in subtle ways, especially around nulls and omitted fields.

Jsoniter compatibility

Jsoniter provides different configurations, one of them is the ConfigCompatibleWithStandardLibrary which attempt to be generating a really similar payload than the standard library.

There is, however, some edge cases.

Here one I went through:

• Jsoniter doesn’t seems to work well with the json tag “omitzero” and libraries like guregu/null.v4. Prefer using “omitempty” to have the same behavior because jsoniter will check the .Valid() method.

All in all, this is a drop in replacement, but adding some tests to make sure you don’t have side effects in complex systems will make a big difference.

Controlling allocations with sync.Pool

Once the JSON serialization hot path is reasonably efficient, memory allocations often become the next bottleneck. Every replication event that flows through your service may involve:

• Allocating a struct to represent the change

• Allocating buffers for JSON encoding

• Allocating intermediate slices and maps during transformations

Under sustained load, this can produce a flood of short-lived objects. The garbage collector has to scan and reclaim them, and that work shows up as CPU usage and latency spikes.

sync.Pool is a practical tool for these patterns:

• It lets you reuse objects (structs, buffers, small slices) across requests without manual “object lifecycle” tracking

• Objects in the pool are eligible for garbage collection if they are not in use, so the pool does not create a permanent memory leak

• For hot types (e.g., your “replication event” struct or reusable []byte buffers for JSON encoding), pooling can significantly reduce the number of allocations per event

Good use cases in this pipeline include:

• Reusing buffers for building bulk requests (e.g., bytes.Buffer or []byte)

• Reusing small structs that hold metadata for each change event

• Reusing temporary scratch space used during transformations

Some practical guidelines:

• Only pool objects that are frequently allocated and easy to reset to a zero state

• Add helper methods like Reset() so the code path that returns an object to the pool always leaves it in a clean state

• Avoid pooling objects that embed context, locks, or anything with complex lifecycle or ownership semantics

Used carefully, sync.Pool can cut heap allocations dramatically in a high-throughput Go service, which brings GC frequency and pause times down.

Garbage collection tuning and experimental GCs

Even after optimizing allocations, GC behavior remains critical in long-lived services under high load.

Starting with go1.25, you can enable an experimental GC at build time that promise better performances. https://go.dev/blog/greenteagc

• Reduce GC-induced latency spikes in services that care more about throughput and tail latency than about absolute minimal memory usage

• Provide more even performance under bursts by scheduling GC work more smoothly over time

In a pipeline that must keep up with a replication slot and a bulk indexer, that trade-off is often desirable:

• Slightly higher steady-state memory usage is acceptable if it avoids GC pauses that temporarily slow down ingestion

• Less erratic latency helps keep Elasticsearch batches flowing and prevents backpressure from building up

However, tuning or switching GC behavior should be the last step, not the first. It works best when:

• Allocations in hot paths have already been reduced with pooling, pre-allocation, and careful data structures

• JSON and other serialization work has been profiled and streamlined

• The service has clear SLOs around memory and latency, and you are comfortable trading one for the other within defined bounds

GC tweaks can then be used to shift the balance slightly rather than to compensate for fundamentally inefficient code.

Putting it together: a stable, high‑volume change pipeline

When all of these optimizations come together, the architecture looks like this:

• A controlled number of goroutines read from the replication slot and push events through a bounded internal queue

• Each event passes through transformation and enrichment code that avoids unnecessary allocations and uses sync.Pool for reusable buffers and structs

• JSON serialization uses a faster encoder like jsoniter

• A bulk indexer sends batched operations to Elasticsearch with size and concurrency tuned to avoid large in-memory batches but still keep the cluster saturated

• GC behavior is tuned only after profiling, to smooth out remaining latency spikes without blowing out memory usage

The result is a Go service that can keep up with a constant stream of database changes, avoid unbounded buffering, and make efficient use of CPU and memory—all while staying within the operational constraints of Postgres and Elasticsearch.

• It isn’t the most character efficient or human-readable.

• UUID v1/v2 is impractical in many environments, as it requires access to a unique, stable MAC address.

• UUID v3/v5 requires a unique seed.

• UUID v4 provides no other information than true randomness, which can lead to database fragmentation in data structures like B-trees, ultimately hurting write performance.

Introducing the ULID Identifier

Few projects I worked on used the ULID (Universally Unique Lexicographically Sortable Identifier), and I really enjoyed working with it, and would love to share this experience with you. Specifically for Go programs using Postgres database. But the same applies to other languages or databases too.

You can find the full spec here - github.com/ulid/spec

ulid() // 01ARZ3NDEKTSV4RRFFQ69G5FAV

What makes ULID great?

ULID addresses the drawbacks of traditional UUID versions by focusing on four key characteristics:

• Lexicographically sortable. Yes, you can sort the IDs. This is the single biggest advantage for database indexing.

• Case insensitive.

• No special characters (URL safe).

• It’s compatible with UUID, so you can still use native UUID columns in your database, for example.

ULID’s structure is key to its sortability. It is composed of 128 bits, just like a UUID, but those bits are structured for function: 48 bits of timestamp followed by 80 bits of cryptographically secure randomness.

 01AN4Z07BY      79KA1307SR9X4MV3

|----------|    |----------------|
 Timestamp          Randomness
   48bits             80bits

Go+Postgres example

The power of ULID is its seamless integration into existing systems, even those relying on the UUID data type. Here is a demonstration using Go with the popular pgx driver for PostgreSQL and the oklog/ulid package.

The code below first connects to a running PostgreSQL instance and creates a table where the primary key is of type UUID. We then insert records using both standard UUID v4 and ULID.

package main

import (
  "context"
  "fmt"
  "os"

  "github.com/google/uuid"
  "github.com/jackc/pgx/v5"
  "github.com/oklog/ulid/v2"
)

func main() {
  ctx := context.Background()

  conn, err := pgx.Connect(ctx, "postgres://...")
  if err != nil {
    panic(err)
  }
  defer conn.Close(ctx)

  _, err = conn.Exec(ctx, `
CREATE TABLE IF NOT EXISTS ulid_test (
  id UUID PRIMARY KEY,
  kind TEXT NOT NULL,
  value TEXT NOT NULL
);`)
  if err != nil {
    panic(err)
  }

  insertUUID(ctx, conn, “1”)
  insertUUID(ctx, conn, “2”)
  insertUUID(ctx, conn, “3”)
  insertUUID(ctx, conn, “4”)
  insertUUID(ctx, conn, “5”)

  insertULID(ctx, conn, “1”)
  insertULID(ctx, conn, “2”)
  insertULID(ctx, conn, “3”)
  insertULID(ctx, conn, “4”)
  insertULID(ctx, conn, “5”)
}

func insertUUID(ctx context.Context, conn *pgx.Conn, value string) {
  id := uuid.New()
  conn.Exec(ctx, "INSERT INTO ulid_test (id, value, kind) VALUES ($1, $2, 'uuid')", id, value)

  fmt.Printf("Inserted UUID: %s\n", id.String())
}

func insertULID(ctx context.Context, conn *pgx.Conn, value string) {
  id := ulid.Make()

  // as you can see, we don’t need to format the ULID as a string, it can be used directly
  conn.Exec(ctx, "INSERT INTO ulid_test (id, value, kind) VALUES ($1, $2, 'ulid')", id, value)

  fmt.Printf("Inserted ULID: %s\n", id.String())
}

The oklog/ulid package implements the necessary interfaces (specifically, database/sql/driver.Valuer and encoding.TextMarshaler) that allow it to be automatically converted into a compatible format (like a string or []byte representation of the UUID) that the pgx driver can successfully map to the PostgreSQL UUID column type. This allows developers to leverage the sortable advantages of ULID without having to change the underlying database schema type in many popular environments.

This allows developers to leverage the sortable advantages of ULID without having to change the underlying database schema type in many popular environments.

Sortability and Performance

The time-based prefix means that new ULIDs will always be greater than older ULIDs, ensuring that records inserted later will be physically placed at the end of the index. This contrasts sharply with UUID v4, where the sheer randomness means records are scattered throughout the index structure.

With traditional UUID v4, sorting records by their insert time is not possible without an extra column. When using ULID, the sort order is inherent in the ID itself, as demonstrated by the following database query output:

select * from ulid_test where kind = 'ulid' order by id;

019aaae4-be9c-d307-238f-be1692b3e8d7 | ulid | 1
019aaae4-be9d-011f-b82e-b870ca2abe9d | ulid | 2
019aaae4-be9f-e9d7-6efc-5b298ecc572b | ulid | 3
019aaae4-bea0-deae-6408-d89e7e3ce030 | ulid | 4
019aaae4-bea1-8ed2-c2f5-144bb1ffedde | ulid | 5

As we can see, the records are returned in the same order they were inserted. Furthermore, ULID is much shorter and cleaner when used in contexts like a URL:

/users/01KANDQMV608PBSMF7TM9T1WR4

ULID can generate 1.21e+24 unique ULIDs per millisecond, which should be more than enough for most applications.

Limitations and the future

There’s really no major drawback to using ULID, but you should understand its limitations. For very (and I mean very) high-volume write systems, ULIDs can become problematic. Since all writes are clustered around the current timestamp, you will have hot spots around the current index key, which can potentially lead to slower writes and increased latency on that specific index block.

While other alternative identifiers exist, such as CUID or NanoID, the benefits of ULID have become a major factor in the evolution of unique identifier standards.

It is worth noting that the newest proposed standard for unique identifiers, UUID v7, aims to address the sortability and database performance issues of older UUID versions by adopting a similar time-ordered structure to ULID.

Resources

• https://github.com/ulid/spec

• https://github.com/oklog/ulid

• https://github.com/jackc/pgx

• Watch on our Youtube channel

However, I wanted to do something in Go, so I started with Conway’s Game of Life. There are various options from C bindings to OpenGL and SDL2 to full game engines such as Ebitengine. I went with raylib-go as it was a good mixture of low-level and abstracted programming.

In this tutorial, we’ll use Go and raylib-go to create Conway’s Game of Life, a simulation where simple rules create endless patterns. By the end, you’ll have gliders drifting, pulsars pulsing, and even a glider gun firing across your screen.

The Game of Life

Conway’s Game of Life is a cellular automaton created by John Conway. It is a zero-player game which evolves. Each generation is built on the previous generation using a set of simple rules.

The game requires no players as you start it off, and it starts to evolve without intervention. It is Turing Complete and can simulate a wide variety of different patterns and Turing Machines.

It starts with an initial game start, then it uses the following rules to determine which cell lives or dies:

1. Any live cell with fewer than 2 live neighbours dies, as if by underpopulation.

2. Any live cell with 2 or 3 live neighbours lives on to the next generation.

3. Any live cell with more than 3 live neighbours dies, as if by overpopulation.

4. Any dead cell with exactly 3 live neighbours becomes a live cell, as if by reproduction.

Each generation or frame in the context of our game will use these rules to determine the next state.

Demo

In this tutorial, we will be creating a simple cellular automation in Go using raylib-go. It will start with a state we define, and allow us to modify the state to see different evolutions.

Setting up the project

First thing we need to do is create a go package for our project and pull down raylib-go so we can start to build with it.

mkdir go-gameoflife  
go mod init gameoflife  
go get -v -u github.com/gen2brain/raylib-go/raylib

When learning anything new in programming, we always start with the classic “Hello World”. To do that, we need to import raylib-go, create a window and display some text.

package main  
  
import rl "github.com/gen2brain/raylib-go/raylib"  
  
type Game struct {  
 Height int  
 Width  int  
}  
  
func main() {  
 // Create the game metadata and state holding object  
 game := Game{Width: 800, Height: 400}  
  
 // Create the Raylib window using the state  
 rl.InitWindow(int32(game.Width), int32(game.Height), "Game of life")  
  
 // Close the window at the end of the program  
 defer rl.CloseWindow()  
  
 // We dont need a high FPS for the game, so 10 should be enough  
 rl.SetTargetFPS(10)  
  
 // Loop until the window needs to close  
 for !rl.WindowShouldClose() {  
  // Starting drawing to the canvas  
  rl.BeginDrawing()  
  
  // Create a black background  
  rl.ClearBackground(rl.Black)  
  
  // Draw Hello world  
  rl.DrawText("Hello world!", 350, 200, 20, rl.RayWhite)  
  
  // End the drawing  
  rl.EndDrawing()  
 }  
}

We have also created a struct to store the game metadata, and we can later use it to store the state of the game. We use that to initialise the window.

To save a little bit of time, we can create a Makefile, which will allow us to simplify the building and running of our code. In this case, we only need one target run, however, this can be expanded to run tests, build more files, clean up build states and more.

.DEFAULT_GOAL := run  
  
run:  
 go run .  
  
help:  
 @echo "run - run the game"  
  
.PHONY: run

Now that we have the make file, we can run make to start the program. The same could be done by running. go run .

make

Win! We now have a window with our Hello World.

In the background, raylib is doing all the hard work for us, working with OpenGL to create the window in the system-specific libraries for your operating system. However, we do not need to worry about that.

For each step of this tutorial, I will provide the code so you can compare. The full code for this step is in the GitHub Repo.

Creating a 2D map

The next thing we need to do is to create an initial state for our game, which will consist of a 2D matrix which we use to draw the cells to the screen.

We will need to update our Game struct to store our state and to specify how big we want the cells.

 type Game struct {  
-       Height int  
-       Width  int  
+       Height   int  
+       Width    int  
+       tileSize int  
+       State    [][]int  
+}

We can then create a function to create the new start for us

func NewGame(width, height, tileSize int) *Game {  
 g := &Game{Width: width, Height: height, tileSize: tileSize}  
 g.State = [][]int{  
  {0, 1, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 1, 0, 0, 0, 0, 1, 0, 0},  
  {1, 1, 1, 0, 0, 0, 0, 1, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 1, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
 }  
 return g  
}  

Next, we need a way to display this on our window. We can simply loop through the array and check if the cell is alive. If it is, we can draw the cell to the screen. We do this with a Draw() method on the Game struct.

func (g *Game) Draw() {  
 // Loop through all of the rows  
 for y := range g.State {  
  
  // Loop through all of the columns  
  for x := 0; x < len(g.State[y]); x++ {  
  
   // If we have marked the column as a 1, draw it as white  
   if g.State[y][x] == 1 {  
  
    // We will need to scale our blocks to the size of the window  
    pixelX := x * g.tileSize  
    pixelY := y * g.tileSize  
  
    // Draw the block to the screen  
    rl.DrawRectangle(int32(pixelX), int32(pixelY), int32(g.tileSize), int32(g.tileSize), rl.RayWhite)  
   }  
  }  
 }  
}

Now we need to update our main()function to use the new game state, then draw it to the window.

-       game := Game{Width: 800, Height: 400}  
+       var game = NewGame(800, 400, 80)  
  
...  
  
-               // Draw Hello world  
-               rl.DrawText("Hello world!", 350, 200, 20, rl.RayWhite)  
+               // Draw the game state  
+               game.Draw()

Now we can run the game again.

make

We now have the starting state for our game. I have used a cell size of 80x80px here to allow us to render it in an 800x400px window, so we can play with the logic and see the results.

Full code for this step is in the GitHub Repo

Getting neighbours

Now we get to the part where we bring our game to life. We can start to create the rule to evolve the state.

To do this, we need a way of counting how many live neighbours a cell has. We can do this by looping through all of the neighbours and checking the surrounding cells.

We can start with the top left cell, the move to the next until we have covered all the cells. We will need to exclude the current cell and account for whether the cell we are calculating is at the edge of the window.

func CountNeighbours(x, y int, gameState [][]int) int {  
 // Counter for the neighbours  
 count := 0  
  
 // Loop through all the rows  
 for cellX := x - 1; cellX <= x+1; cellX++ {  
  
  // Loop through all the columns  
  for cellY := y - 1; cellY <= y+1; cellY++ {  
  
   // We want to make sure we do not count past the boundary of the board  
   if cellY < 0 || cellX < 0 || cellY >= len(gameState) || cellX >= len(gameState[0]) {  
    continue  
   }  
   // If current cell, we can skip it  
   if cellY == y && cellX == x {  
    continue  
   }  
   //  Check if cell is alive  
   if gameState[cellY][cellX] == 1 {  
    count++  
   }  
  
  }  
 }  
 return count  
}

Next, we need to codify the rules of the game using the count of the neighbours. We can do that with a switch statement for each of the rules.

func IsCellAlive(current, neighbours int) int {

 switch {

 // Any live cell with fewer than two live neighbours dies  
 // as if by underpopulation.  
 case neighbours < 2:  
  return 0

 // Any live cell with two or three live neighbours lives  
 // on to the next generation.  
 // Any dead cell with two neighbours, remains dead  
 case neighbours == 2:  
  return current

 // Any dead cell with exactly three live neighbours becomes a  
 // live cell, as if by reproduction.  
 case neighbours == 3:  
  return 1

 // Any live cell with more than three live neighbours dies  
 // as if by overpopulation.  
 case neighbours > 3:  
  return 0  
 }
 
 return 0  
}

Now that we have the calculations for our next state implemented, we need to take our existing state and update it based on the rules. We can create an Update() method on our Game struct to do this.

func (g *Game) Update() {  
  
 // We can stat with hardcoded state for now  
 // However, we would want to update thisbased on the current state  
 var newState = [][]int{  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
 }  
  
 // Loop through each row  
 for indexY, cellY := range g.State {  
  
  // Loop through each column  
  for indexX, cellX := range cellY {  
     
   // Count how many neighbours the current cell has  
   neighbours := CountNeighbours(indexX, indexY, g.State)  
  
   // Update the new state using the rule based on neighbours  
   newState[indexY][indexX] = IsCellAlive(cellX, neighbours)  
  }  
 }  
  
 // Set the new state to the updated state  
 g.State = newState  
}

We have used another slicee to build the new state, then we replace the old one. This implementation increases memory usage, however, we could offset and update it in-place with a truth table and two passes of the slice. However, let's keep it simple.

rl.ClearBackground(rl.Black)  
  
+               // Update the game state before drawing  
+               game.Update()  
+  
                // Draw the game state

Now we can add the Update function to our main game loop and start the game again to see if it works.

make

Run it and you will see the cells start to evolve!

We can see the patterns start to evolve with each generation, and they will converge into a steady state eventually.

Full code for this step is in the GitHub Repo

Update the map

Now that the core functionality of our game is working, we can start to scale the map size to make more complex patterns.

Let's replace the hard-coded map with a function we can use to generate a game state as big as we want.

func CreateGameState(newWidth, newHeight int) [][]int {  
 // Create a new game state with the right height  
 newState := make([][]int, newHeight)  
  
 // Create the rows with the right length  
 for i := range newHeight {  
  newState[i] = make([]int, newWidth)  
 }  
  
 // Return the new state map  
 return newState  
}

We can use this blank game state in our code where we are manually defining the state. We are defining the state in both the Update() and NewGame() functions.

        // However, we would want to update this based on the current state  
-       var newState = [][]int{  
-               {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
-               {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
-               {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
-               {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
-               {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
-       }  
+       newState := CreateGameState(len(g.State[0]), len(g.State))  
  
        // Loop through each row

 func NewGame(width, height, tileSize int) *Game {  
        g := &Game{Width: width, Height: height, tileSize: tileSize}  
-       g.State = [][]int{  
-               {0, 1, 0, 0, 0, 0, 0, 0, 0, 0},  
-               {0, 0, 1, 0, 0, 0, 0, 1, 0, 0},  
-               {1, 1, 1, 0, 0, 0, 0, 1, 0, 0},  
-               {0, 0, 0, 0, 0, 0, 0, 1, 0, 0},  
-               {0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
-       }  
+       g.State = CreateGameState(g.Width/g.tileSize, g.Height/g.tileSize)  
        return g  
 }

That looks much cleaner. However, we have now lost our initial state, which triggers the game. We can fix this by creating a function that will add a new pattern. We will start with the Glider pattern.

func CreateGliders(x, y int, gameState *[][]int) {  
 // Draw the glider patter in the game state  
 (*gameState)[y][x+1] = 1  
 (*gameState)[y+1][x+2] = 1  
 (*gameState)[y+2][x] = 1  
 (*gameState)[y+2][x+1] = 1  
 (*gameState)[y+2][x+2] = 1  
}

Now that we have the function to create a Glider, we can add them to our game state in the main() function.

-       var game = NewGame(800, 400, 80)  
+       var game = NewGame(800, 400, 10)  
+       CreateGliders(0, 0, &game.State)  
+       CreateGliders(10, 0, &game.State)  
+       CreateGliders(20, 0, &game.State)  
+       CreateGliders(30, 0, &game.State)

Let's run the game again.

make

Great!

We can see the addition of 4 gliders to a much larger window, and we can see them move across the screen as the generations increase.

Full code for this step is in the GitHub Repo

Setting up different simulations

Now that we have the game working, we can play around with different initial states to see how they evolve.

The other patterns are more complex compared to the glider, so I will create a helper function for adding the pattern, then define the patterns separately. More patterns can be found here.

func addPattern(x, y int, pattern [][]int, gameState *[][]int) {  
 // Loop through the row  
 for row := range pattern {  
  
  // Loop through the  
  for col := 0; col < len(pattern[row]); col++ {  
  
   // Update the game state if cell alive  
   if pattern[row][col] == 1 {  
    (*gameState)[y+row][x+col] = 1  
   }  
  }  
 }  
}

func CreateGliderGun(x, y int, gameState *[][]int) {  
 // Create a slice of the pattern  
 pattern := [][]int{  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1},  
  {1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
 }  
 addPattern(x, y, pattern, gameState)
}

func CreatePulsar(x, y int, gameState *[][]int) {  
 // Create a slice of the pattern  
 pattern := [][]int{  
  {0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1},  
  {1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1},  
  {1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1},  
  {0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0},  
  {1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1},  
  {1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1},  
  {1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1},  
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},  
  {0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0},  
 }  
  
 addPattern(x, y, pattern, gameState)  
}

func CreatePentadecathlon(x, y int, gameState *[][]int) {  
 // Create a slice of the pattern  
 pattern := [][]int{  
  {0, 0, 1, 0, 0, 0, 0, 1, 0, 0},  
  {1, 1, 0, 1, 1, 1, 1, 0, 1, 1},  
  {0, 0, 1, 0, 0, 0, 0, 1, 0, 0},  
 }  
  
 addPattern(x, y, pattern, gameState)  
}

Now we have our patterns, let's add them to the state.

var game = NewGame(800, 400, 10)  
-       CreateGliders(0, 0, &game.State)  
-       CreateGliders(10, 0, &game.State)  
-       CreateGliders(20, 0, &game.State)  
-       CreateGliders(30, 0, &game.State)  
+       CreateGliderGun(0, 0, &game.State)  
+       CreatePentadecathlon(40, 10, &game.State)  
+       CreatePulsar(60, 20, &game.State)  
  
        // Create the Raylib window using the state

Run our game again.

make

Now our game is busy, with different interactions and patterns.

The full code can be found on the GitHub repo

Testing

Something I have omitted from this tutorial is the testing. When developing this post, I created tests alongside the code to validate that the evolution rules and update function worked as expected. You can find my testing in the main_test.go.

The example, the code to test the IsCellAlive function, looks like:

func TestIsCellAlive(t *testing.T) {  
 testCases := []struct {  
  name       string  
  want       int  
  current    int  
  neighbours int  
 }{  
  {name: "Live cell should not live with <2", neighbours: 1, current: 1, want: 0},  
  {name: "Live cell should live with 2", neighbours: 2, current: 1, want: 1},  
  {name: "Live cells should live with 3", neighbours: 3, current: 1, want: 1},  
  {name: "Dead cells should live with 3", neighbours: 3, current: 0, want: 1},  
  {name: "Live cell should not live with >3", neighbours: 4, current: 1, want: 0},  
  {name: "Dead cells should not live if already dead", neighbours: 0, want: 0},  
 }  
  
 for _, tt := range testCases {  
  t.Run(tt.name, func(t *testing.T) {  
   got := IsCellAlive(tt.current, tt.neighbours)  
  
   if got != tt.want {  
    t.Errorf("got %d, want %d", got, tt.want)  
   }  
  })  
 }  
}

In future posts, I can cover testing more and potentially walk through how we could have done this with Test Driven Development (TDD)

Conclusion

We have walked through:

• Creating a window with raylib-go

• Adding shapes to the window

• Creating Conway’s Game of Life state

• Updating the state with new generations

• Adding new patterns to the state

We just built Conway’s Game of Life in Go with raylib-go, starting from a blank window all the way to gliders, pulsars, and even a glider gun. Along the way, we covered rendering, updating state, and adding reusable patterns.

You can find the full source code in the GitHub repo. Try running it, tweak the patterns, or create your own.

If you build something cool with it, I’d love to see it! Share your experiments in the comments, or tag me on LinkedIn.

I was at a ContainerDays conference recently and attended a great talk from Nikolay Kuznetsov about the Outbox pattern and resilient system design. It seemed like a powerful solution to a common problem in distributed systems, so I decided to dive deeper, and I want to share what I've learned and summarize for our readers in this post.

The challenge in event-driven systems

In modern, event-driven architectures, services often communicate asynchronously using a message broker. A typical flow looks like this: a service receives a request, updates its own database, and then publishes an event to notify other services about the change. Or these two actions happen in parallel.

Here's the problem: what happens if the database commit succeeds, but the subsequent call to the message broker fails? Maybe the broker is temporarily down, or there's a network glitch. Or what if the database is not available at that time? Or what if the program somehow crashes?

You end up in an inconsistent state. Your local database has the new data, but the rest of the system never gets the notification. This is a serious issue because the database operation and the message publishing are not atomic - they don't succeed or fail as a single, indivisible unit.

Presenting the Outbox pattern

This is where the Outbox pattern comes to the rescue! The core idea is simple but powerful: instead of directly publishing a message to the broker, you save the message to a dedicated "outbox" table within your local database. Crucially, you do this as part of the same database transaction as your business data changes.

This approach leverages the atomicity of database transactions to guarantee that either both the business data is saved and the event is queued for sending, or neither is. The message can't get lost.

A possible Outbox table schema

To implement this, you need an outbox table in your database (or multiple ones). A typical schema might look something like this, storing the message content and its processing state.

-- Example schema for a PostgreSQL outbox table
CREATE TABLE outbox (
    id uuid PRIMARY KEY,
    topic varchar(255) NOT NULL,
    message jsonb NOT NULL,
    state varchar(50) NOT NULL DEFAULT 'pending', -- e.g., pending, processed
    created_at timestamptz NOT NULL DEFAULT now(),
    processed_at timestamptz
);

The message relay

Of course, just putting messages in a database table doesn't send them. You need a separate background process, often called a Message Dispatcher or Relay. This worker's job is to:

• Periodically query the outbox table for new, unprocessed messages.

• Publish these messages to the actual message broker.

• Once the broker confirms receipt, update the message's record in the outbox table to mark it as processed.

• Handle errors and retries.

This process guarantees at-least-once delivery. A message might be sent more than once if the relay publishes it but fails before it can mark the record as processed. Because of this, your message consumers should always be designed to be idempotent, meaning they can safely process the same message multiple times without causing issues.

Minimal Go & Postgres implementation

Let's look at a concrete example using Go, pgx for Postgres, and Google Cloud Pub/Sub. Imagine we have an orders service.

First, our business logic for creating an order will also create the outbox message within a single transaction.

// orders/main.go

package main

import (
	"context"
	"encoding/json"
	"log"
	"os"

	"github.com/google/uuid"
	"github.com/jackc/pgx/v5"
	"github.com/jackc/pgx/v5/pgxpool"
)

type Order struct {
	ID       uuid.UUID `json:"id"`
	Product  string    `json:"product"`
	Quantity int       `json:"quantity"`
}

type OrderCreatedEvent struct {
	OrderID uuid.UUID `json:"order_id"`
	Product string    `json:"product"`
}

// createOrderInTx creates an order and its corresponding outbox event atomically.
func createOrderInTx(ctx context.Context, tx pgx.Tx, order Order) error {
	// 1. Insert the order
	_, err := tx.Exec(ctx, "INSERT INTO orders (id, product, quantity) VALUES ($1, $2, $3)",
		order.ID, order.Product, order.Quantity)
	if err != nil {
		return err
	}
	log.Printf("Inserted order %s into database", order.ID)

	// 2. Prepare the event message for the outbox
	event := OrderCreatedEvent{
		OrderID: order.ID,
		Product: order.Product,
	}
	msg, err := json.Marshal(event)
	if err != nil {
		return err
	}

	// 3. Insert the event into the outbox table
	_, err = tx.Exec(ctx, "INSERT INTO outbox (topic, message) VALUES ($1, $2, $3)",
		"orders.created", msg)
	if err != nil {
		return err
	}
	log.Printf("Inserted outbox event for order %s", order.ID)

	return nil
}

func main() {
	ctx := context.Background()
	pool, err := pgxpool.New(ctx, os.Getenv("DATABASE_URL"))
	if err != nil {
		log.Fatalf("Unable to connect to database: %v", err)
	}
	defer pool.Close()

	tx, err := pool.Begin(ctx)
	if err != nil {
		log.Fatalf("Unable to begin transaction: %v", err)
	}
	defer tx.Rollback(ctx)

	newOrder := Order{
		ID:       uuid.New(),
		Product:  "Super Widget",
		Quantity: 10,
	}

	if err := createOrderInTx(ctx, tx, newOrder); err != nil {
		log.Fatalf("Failed to create order: %v", err)
	}

	if err := tx.Commit(ctx); err != nil {
		log.Fatalf("Failed to commit transaction: %v", err)
	}

	log.Println("Successfully created order and outbox event.")
}

Next, the Relay process polls the database, sends the message, and updates the state. Using FOR UPDATE SKIP LOCKED is a great way to allow multiple relay instances to run concurrently without processing the same message.

// relay/main.go
package main

import (
	"context"
	"log"
	"time"

	"cloud.google.com/go/pubsub"
	"github.com/google/uuid"
	"github.com/jackc/pgx/v5/pgxpool"
)

type OutboxMessage struct {
	ID      uuid.UUID
	Topic   string
	Message []byte
}

func processOutboxMessages(ctx context.Context, pool *pgxpool.Pool, pubsubClient *pubsub.Client) error {
	tx, err := pool.Begin(ctx)
	if err != nil {
		return err
	}
	defer tx.Rollback(ctx)

	// 1. Lock the next pending message so other relay instances don't grab it
	rows, err := tx.Query(ctx, `
        SELECT id, topic, message
        FROM outbox
        WHERE state = 'pending'
        ORDER BY created_at
        LIMIT 1
        FOR UPDATE SKIP LOCKED
    `)
	if err != nil {
		return err
	}
	defer rows.Close()

	// 2. If we found a message, publish it to Pub/Sub
	var msg OutboxMessage
	if rows.Next() {
		if err := rows.Scan(&msg.ID, &msg.Topic, &msg.Message); err != nil {
			return err
		}
	} else {
		// No new messages
		return nil
	}

	log.Printf("Publishing message %s to topic %s", msg.ID, msg.Topic)
	result := pubsubClient.Topic(msg.Topic).Publish(ctx, &pubsub.Message{
		Data: msg.Message,
	})
	_, err = result.Get(ctx)
	if err != nil {
		return err
	}

	// 3. Mark the message as processed
	_, err = tx.Exec(ctx, "UPDATE outbox SET state = 'processed', processed_at = now() WHERE id = $1", msg.ID)
	if err != nil {
		return err
	}
	log.Printf("Marked message %s as processed", msg.ID)

	return tx.Commit(ctx)
}

func main() {
	// TODO: initialize actual Postgres and Pubsub connections
	var (
		pool         *pgxpool.Pool
		pubsubClient *pubsub.Client
	)

	// feel free to use another interval
	ticker := time.NewTicker(1 * time.Second)
	defer ticker.Stop()
	for range ticker.C {
		if err := processOutboxMessages(context.Background(), pool, pubsubClient); err != nil {
			log.Printf("Error processing outbox: %v", err)
		}
	}
}

Why then we say that the message is sent at least once? That's because of the rare case when the message is sent, but something happened when updating the message status in the database, for example due to a service crash.

Another alternative: Postgres logical replication

We touched this topic briefly in the following post.

While polling is a simple and effective strategy, it can introduce some latency and be resource-intensive. For those using PostgreSQL, there's a more advanced, push-based alternative: logical replication.

Databases like Postgres maintain a Write-Ahead Log (WAL), which is an append-only log of every change made to the database. Logical replication allows you to tap into this log and stream changes for specific tables as they happen.

Instead of your relay constantly asking the database "Anything new?", the database effectively tells your relay, "Hey, a new row was just inserted into the outbox table!". This can be more efficient and provide lower latency, though it adds some implementation complexity.

In Go you can use pglogrepl which is a Go library designed for interacting with PostgreSQL’s logical replication protocol.

Conclusion

To summarize, the Outbox Pattern ensures that a message was sent (e.g. to a queue) successfully at least once. With this pattern, instead of directly publishing a message to the queue, we store it in temporary outbox table first. We're wrapping the entity save and message storing in a single transaction so this operation is atomic. It will be published later through a background process called Message Relay.

While the concept is simple, it can be complicated to implement the actual solution and there multiple ways of doing so.

Resources

• Source code

• Write-Ahead Logging (WAL)

• pglogrepl

I don’t know about you, but I love code reviews.

The power of code reviews

There are different ways to conduct a code review (mailing lists, pull/merge requests, even the code printouts), but they all serve the same main purposes.

• 🐛 Help catch bugs early, which can save time and resources later in the development cycle.

• ✅ Help maintain high code quality and adhere to the code standards, performance considerations, etc.

• 💬 Code reviews can improve team communication. They create a platform for discussing design decisions and coding standards. This dialogue can lead to better solutions and a more cohesive team.

• 📚 Promote knowledge sharing among team members. When developers review each other's code, they learn different approaches and techniques, which can enhance their skills.

Many would say that you need a code review process in teams only, but I would argue and say that it’s also helpful when you’re working on a project alone, mainly for the documentation purpose and self-review.

But code reviews are time consuming, reviewing code manually takes hours, slowing down development. As well as there are other challenges. For example different reviewers have different standards, leading to inconsistent feedback. Reviewers may overlook issues due to familiarity with the project. And as teams grow, manual review processes become bottlenecks.

I have a good quote for you about the inconsistencies, which is unfortunately very true very often:

AI-powered code reviews

And LLMs can help with these issues quite well, and can be a great companion to software developers. I say “companion“ because LLMs excel in some routine parts, while humans excel in software design, architecture decisions, they have taste and know everything about the software system. And that’s great, let’s outsource routine boring tasks to LLMs to have more time for creativity.

There is probably a world where these 2 circles intersect a bit, as there is no black and white.

Let’s move closer to the tools, it’s extremely easy to integrate such solutions into an existing system and workflows, as they don’t require teams to radically change the way they work. So far I used only GitHub Copilot and CodeRabbit, and both were really easy to setup in the GitHub. But I would say that I find CodeRabbit settings more advanced, as you can fine tune everything on per-repository level.

After the integration is ready, these tools start automatically reviewing your pull requests and enhancing them with additional info, such as summary, comments, suggestions.

Disclaimer: I am a big fan of CodeRabbit, and am grateful they partnered on this article. However, I will try to only write my honest opinion on this topic of code review practices using AI and tools like CodeRabbit.

Get Started Today

Team vs Solo

Likely, in a team, there are already processes in place that involve some code review step. It could be a slow and blocking phase. Developers need to switch context to review someone else’s code, then spend a lot of time writing the summary of their own PR. I honestly remember days when half of my day was about reviewing the code, I am sure it would be much faster nowadays was I using something like CodeRabbit.

But I also wanted to talk about the projects where there is no clear concept of a “team” (for example an OSS project), or it’s your own solo project.

For example, as a maintainer of an open source software, you may receive some pull request and spend a lot of your free time understanding what the change is about, then ditching obvious errors, etc.

And exactly here I see a great value of tools like CodeRabbit.

Real example

Let’s take an example from a real project I’m maintaining - plutov/paypal. Few days ago I received a new PR from a contributor and when I opened it, it had a nice summary:

That helped both me and the author of a PR. On another PR CodeRabbit was able to generate a sequence diagram to explain the database comms, isn’t that cool?

And you can also work with CodeRabbit directly in your VSCode (no Neovim yet unfortunately).

Conclusion

In conclusion, the integration of AI into the code review process isn't about replacing humans, but rather augmenting them by automating routine tasks and enabling engineers to concentrate on higher-order concerns like architecture, business logic, scalability.

And if you want to learn more about CodeRabbit, check the link below, it’s free.

Get Started Today

What are Redis Streams ?

A Redis stream acts as an append-only log where each new entry is assigned a unique, time-based ID. Unlike traditional logs, Redis Streams support random access in O(1) time and enable complex consumption patterns, such as consumer groups for distributed processing. This makes them suitable for scenarios where you need to record, syndicate, and process events in real time.

Key characteristics:

• Append-only: Once data is written, it cannot be modified.

• Immutable entries: Each entry is a set of key-value pairs with a unique ID.

• Real-time syndication: Multiple consumers can process events as they arrive.

• Trimming and retention: Streams can be trimmed to prevent unbounded growth.

Cut Code Review Time & Bugs in Half (Sponsor)

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo's!

Get Started Today

How to use

Single client

For basic usage, you interact with Redis Streams using a set of core commands:

• XADD: Add a new entry to a stream. Each entry consists of field-value pairs, and Redis generates a unique ID if you use the * wildcard.

• XRANGE: Retrieve a range of entries by specifying start and end IDs, making it easy to access historical data or iterate over the stream.

• XLEN: Get the number of entries in a stream—useful for monitoring or batch processing.

Example workflow:

XADD mystream * field1 value1 field2 value2
XLEN mystream
XRANGE mystream - +

The - and + special IDs mean respectively the minimum ID possible and the maximum ID possible inside a stream, so the XRANGE command will just return every entry in the stream.

Consumer groups

For scalable processing, Redis Streams support consumer groups. This allows multiple consumers to cooperatively process messages from the same stream, with Redis distributing entries among them.

How it works:

• Consumer group: A named group coordinating multiple consumers.

• Consumers: Each consumer reads a subset of entries, ensuring parallelism.

• Acknowledgement: Consumers must acknowledge (XACK) processed entries. Acknowledgements are tied to a consumer group and its consumers.

Example workflow:

XGROUP CREATE mystream mygroup $
XREADGROUP GROUP mygroup consumer1 STREAMS mystream >
XACK mystream mygroup <entry-id>

Handling failures and message replays

One of the strengths of Redis Streams is the ability to replay messages in case of incidents or failures. Since messages are not deleted automatically after acknowledgment, you can reprocess them if needed.

Auto-claiming and message management:

• XAUTOCLAIM: Automatically transfer ownership of pending messages that have been idle for too long to another consumer, ensuring that no message is left unprocessed if a consumer crashes.

• Manual deletion: Messages are not deleted upon acknowledgment; you must use XDEL to remove them. This gives you control but also requires you to manage retention, typically after all consumer groups have processed the message.

Trade-offs:

• Manual cleanup: You need to implement logic to delete messages once they're no longer needed, as Redis does not do this automatically even if all consumer groups have acknowledged a message.

• Replay capability: The upside is that you can replay messages without recreating payloads, simply by reading from the stream history as needed.

Practical considerations

• Memory management: Use XTRIM or the MAXLEN option with XADD to cap stream size and prevent unbounded memory growth.

• Visibility: Inspect pending messages with XPENDING to monitor which messages are yet to be processed or acknowledged by consumers.

Redis Streams provide a flexible, high-performance foundation for event-driven architectures. By leveraging consumer groups, auto-claiming, and careful retention management, you can build scalable, resilient systems that can recover from failures and replay events as needed, all with the speed and simplicity Redis is known for.

Also, it’s nice to see Redis to be open-source (AGPL) again.

Cut Code Review Time & Bugs in Half (Sponsor)

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo's!

Get Started Today

For example, there is a circumflex TUI that lets you read Hacker News from your Terminal.

clx

And there are more examples like that, check out this article that lists just few.

Developing CLIs and TUIs in Go is fun, and there are really good packages out there to make it so. For example, for developing the command-based imperative CLIs you can use Cobra library from Steve Francia, which was used to build popular kubectl, hugo and github CLIs.

And when it comes to terminal apps, there is an amazing library called Bubble Tea to build beautiful interactive TUIs.

For example the circumflex TUI that we've seen before was developed using Bubble Tea.

Cut Code Review Time & Bugs in Half (Sponsor)

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo's!

Get Started Today

Note-taking TUI

In this article we will build a terminal-based note taking app using Bubble Tea and some other libraries. It will be a very simple application with SQLite store for storing the notes.

Installation

If you don’t have Bubble Tea installed yet, run the following commands to install it as well as few other adjacent packages :

go get github.com/charmbracelet/bubbletea
go get github.com/charmbracelet/lipgloss
go get github.com/charmbracelet/bubbles

Bubble Tea is usually used with other libraries, as you can see above, we installed them too.

lipgloss is a great styling library from Charm.

And bubbles is a TUI components library for Bubble Tea (also from Charm). For example File picker:

Bubble Tea Architecture

Before jumping to our program, let’s review how Bubble Tea actually works.

Bubble Tea is based on the functional design paradigms of The Elm Architecture, which happens to work nicely with Go. It's a delightful way to build applications.

Bubble Tea programs are comprised of a model that describes the application state and three simple methods on that model:

• Init, a function that returns an initial command for the application to run.

• Update, a function that handles incoming events and updates the model accordingly.

• View, a function that renders the UI based on the data in the model.

Now, how does it translate to code?

Model

A good place to start is the Model. Our note-taking application can be in multiple states (list view, edit title view, edit body view), and as you can imagine we will need UI elements such as text input, textarea, list. What’s cool is that our main model can contain other models as well, so in a way we’re building a tree of models.

So here is how our model may look like:

model.go

package main

import (
	"log"

	"github.com/charmbracelet/bubbles/textarea"
	"github.com/charmbracelet/bubbles/textinput"
	tea "github.com/charmbracelet/bubbletea"
)

const (
	listView uint = iota
	titleView
	bodyView
)

type model struct {
	store     *Store
	state     uint
	textarea  textarea.Model
	textinput textinput.Model
	currNote  Note
	notes     []Note
	listIndex int
}

func NewModel(store *Store) model {
	notes, err := store.GetNotes()
	if err != nil {
		log.Fatalf("unable to get notes: %v", err)
	}

	return model{
		store:     store,
		state:     listView,
		textarea:  textarea.New(),
		textinput: textinput.New(),
		notes:     notes,
	}
}

func (m model) Init() tea.Cmd {
	return nil
}

As you can see our model is basically a state of our application. It also knows about the notes storage (sqlite in my case, but we will not focus on much on that). NewModel() function creates a new fresh state, and Init() is empty in our case, as initial command for the application is not required and not needed in our case.

Main Loop

With the model in place we can initiate a Bubble Tea program in main.go

package main

import (
	"log"

	tea "github.com/charmbracelet/bubbletea"
)

func main() {
	store := new(Store)
	if err := store.Init(); err != nil {
		log.Fatalf("unable to init store: %v", err)
	}

	m := NewModel(store)

	p := tea.NewProgram(m)
	if _, err := p.Run(); err != nil {
		log.Fatalf("unable to run tui: %v", err)
	}
}

As you can see, we can pass our model to tea.NewProgram() and Bubble Tea will do the rest for us, assuming that our Model implements the interface with Init(), Update(), View() methods.

type Model interface {
    Init() Cmd
    Update(msg Msg) (Model, Cmd)
    View() string
}

Update

The Update() method handles user input (or any other events such as Ticks for example).

Here we can react to various events and update our model accordingly. For example, when a user presses the hotkeys, we switch to another view.

What’s interesting is that our Model contains other models so we must propagate the Update() accordingly.

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	m.textarea, _ = m.textarea.Update(msg)
	m.textinput, _ = m.textinput.Update(msg)

	switch msg := msg.(type) {
	// handle key strokes
	case tea.KeyMsg:
		key := msg.String()

		switch m.state {
		// List View key bindings
		case listView:
			switch key {
			case "q":
				return m, tea.Quit
			case "n":
				// ...
			case "up", "k":
				// ...
			case "down", "j":
				// ...
			case "enter":
				// ...
			}

		// Title Input View key bindings
		case titleView:
			switch key {
			case "enter":
				// ...
			case "esc":
				m.state = listView
			}

		// Body Textarea key bindings
		case bodyView:
			switch key {
			case "ctrl+s":
				// ...
			case "esc":
				// ...
			}
		}
	}

	return m, nil
}

As you can see here in our Update() function we react to the following keystrokes:

• q - quit the app

• n - new note

• j,k - move up,down between notes

• enter - open note

• ctrl+s - save note

• esc - exit the step

The Msg type in Bubble Tea is flexible and can carry various data. In this scenario, it resembles browser events in JavaScript. For instance, a timer event might not carry data, while a click event specifies what was clicked.

But be aware, that messages are not necessarily received in the order they are sent, in Go, if you have more than one go routine sending to a channel, the order in which the sends and receives occur is unspecified.

View

The View method is where we take the state of our Model and render it. Here we will be using the libraries libgloss and bubbles that we installed previously.

view.go

package main

import (
	"strings"

	"github.com/charmbracelet/lipgloss"
)

var (
	appNameStyle = lipgloss.NewStyle().Background(lipgloss.Color("99")).Padding(0, 1)

	faint = lipgloss.NewStyle().Foreground(lipgloss.Color("255")).Faint(true)

	listEnumeratorStyle = lipgloss.NewStyle().Foreground(lipgloss.Color("99")).MarginRight(1)
)

func (m model) View() string {
	s := appNameStyle.Render("NOTES APP") + "\n\n"

	if m.state == titleView {
		s += "Note title:\n\n"
		s += m.textinput.View() + "\n\n"
		s += faint.Render("enter - save • esc - discard")
	}

	if m.state == bodyView {
		s += "Note:\n\n"
		s += m.textarea.View() + "\n\n"
		s += faint.Render("ctrl+s - save • esc - discard")
	}

	if m.state == listView {
		for i, n := range m.notes {
			// render each note
		}
		s += faint.Render("n - new note • q - quit")
	}

	return s
}

Again, as we have other models like textarea and textinput, we call View() on them too to embed them into our top-level view. Also, because the View() method returns a simple string, it becomes easy to test this deterministic function.

Conclusion

I didn’t include all the code, like SQLite storage and some utils, but you can find the full program here and even run it yourself, all you need is Go installed.

Building interactive TUIs in Go is genuinely enjoyable, thanks to powerful libraries like Bubble Tea.

To me the main strength of Bubble Tea is its simplicity with Elm Architecture, allowing developers to craft robust and responsive TUI applications, but also very modular.

The applications developed using Bubble Tea are usually very performant as well.

Let me know in the comments below if you built anything fun with Bubble Tea. I for example built this simple ultrafocus TUI to block distracting websites and boost productivity.

Cut Code Review Time & Bugs in Half (Sponsor)

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo's!

Get Started Today

In modern CI/CD pipelines, supporting multiple architectures for Docker image builds, specifically amd64 and arm64, is crucial. This ensures compatibility for your team, whether they're on Apple silicon or Intel machines, and aligns with your production infrastructure.

If you don’t have access to Github Runners or Machines in your CI/CD with both architectures, you might be using emulation, like QEMU and it can be quite slow to build an arm64 image on an amd64 machine.

That’s why you can optimize your builds and use cache to make it faster.

This article details a robust 3-step workflow for achieving efficient multiplatform builds and covers best practices for frontend (FE) projects where static assets should be built only once for amd64.

Why multiplatform builds need special cache handling

Docker's BuildKit and buildx make multiplatform builds possible, but cache storage is architecture-specific.

If you build for both amd64 and arm64 in a single step and push cache to a registry, only one architecture's cache is stored due to cache overwriting. This leads to inefficient builds and missed cache hits on subsequent builds for the other architecture.

3-step workflow for optimized multiplatform caching

To ensure both amd64 and arm64 builds benefit from proper cache, follow this three-step process:

1. Build and cache for arm64 (with --load)

docker buildx build \
 --cache-from=type=registry,ref=image:buildcache-arm64 \
 --cache-to=type=registry,ref=image:buildcache-arm64 \ 
 --platform=linux/arm64 \
 --load .

or for those who use Github Actions:

- name: Build and load cache for arm64
  uses: docker/build-push-action@v6
  with:
    context: .
    platforms: linux/arm64
    load: true
    cache-from: type=registry,ref=image:buildcache-arm64
    cache-to: type=type=registry,ref=image:buildcache-arm64,compression=zstd,mode=max
    env:
      DOCKER_BUILDKIT: 1

This command builds only for arm64, loads the result to the local Docker daemon, and updates the arm64-specific cache in the registry.

2. Build and cache for amd64 (with --load)

docker buildx build \
 --cache-from=type=registry,ref=image:buildcache-amd64 \
 --cache-to=type=registry,ref=image:buildcache-amd64 \ 
 --platform=linux/amd64 \
 --load .

Similarly, this builds for amd64 and updates the amd64-specific cache in the registry.

3. Build final multiplatform image using both caches

docker buildx build \
 --cache-from=type=registry,ref=image:buildcache-arm64 \
 --cache-from=type=registry,ref=image:buildcache-amd64 \
 --platform=linux/amd64,linux/arm64 \
 -t myimage:latest \
 --push .

or for those who use Github Actions:

- name: Build and push image
  uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    platforms: linux/amd64, linux/arm64
    tags: myimage:latest
    cache-from: |
      type=registry,ref=image:buildcache-arm64
      type=registry,ref=image:buildcache-amd64
    env:
      DOCKER_BUILDKIT: 1

This step imports both architecture caches, enabling maximum cache reuse, and pushes the multiplatform image to your registry.

Note: Each cache must be exported to a unique registry reference to avoid overwriting.

Registry cache: best practices

• Separate cache references: Use distinct registry tags for each architecture (e.g., image:buildcache-arm64, image:buildcache-amd64).

• Import multiple caches: The final build should import both caches using multiple --cache-from flags.

• Builder consistency: Perform all steps on the same builder instance to ensure local cache continuity.

Frontend projects: building static assets only once

For frontend projects (e.g., React, Vue, Angular), static assets are architecture-agnostic. There's no need to build them for both amd64 and arm64. Instead, let’s assume the machine in your CI is amd64:

• Force Builder to amd64: Use --platform=linux/amd64 or set DOCKER_DEFAULT_PLATFORM=linux/amd64 to ensure static files are always built on amd64.

• Import Static Files in Final Multiplatform Layer: In your Dockerfile, copy the prebuilt static assets into the final image layer, which is then assembled as multiplatform.

Example:

# Stage 1: Build static assets (amd64 only) 
FROM --platform=linux/amd64 node:18 AS fe-build 
WORKDIR /app 
COPY frontend/ . 
RUN npm ci && npm run build 

# Stage 2: Multi-platform runtime 
FROM nginx:alpine
COPY --from=fe-build /app/build /usr/share/nginx/html

• The fe-build stage always runs on amd64, while the final image can be built for both architectures.

Key takeaways

• Build and cache each architecture independently before building the final multiplatform image.

• Always use unique cache references for each architecture to avoid cache overwriting.

• For frontend projects, build static files only on amd64 and import them into the multiplatform image.

• Use Dockerfile's --platform flag or environment variables to control build architecture.

By following these steps, you ensure fast, reliable, and cache-efficient multiplatform Docker builds for both backend and frontend projects.

In this article we will dive into the standard itself to make sure we understand how JWT works. We will also implement a secure Go server that can issue JWTs and verify them. And finally we will review some best practices for securely using JWTs.

Use Cases

The most common use case for JWT is authorization. Once the user is logged in, each subsequent request will include the JWT, allowing the user to access routes, services, and resources that are permitted with that token. Usually sent in a header, for example “Authorization”.

Note that on this diagram, the Authorization Server and the Resource Server are two separate entities, but technically they could be the same application.

Another use is information exchange. JSON Web Tokens are a good way of securely transmitting information between parties. Because JWTs can be signed, you can be sure the senders are who they say they are.

Format

HEADER.PAYLOAD.SIGNATURE

JWT consists of three concatenated Base64url-encoded strings, separated by dots:

• Header: contains metadata about the type of token and the cryptographic algorithms used to secure its contents.

• JWS payload (set of claims): contains verifiable security statements, such as the identity of the user and the permissions they are allowed.

• JWS signature: used to validate that the token is trustworthy and has not been tampered with. When you use a JWT, you must check its signature before storing and using it.

Payload contains:

• Registered claims. The JWT specification defines seven reserved claims that are not required, but are recommended to allow interoperability with third-party applications. Some of them are: iss (issuer), exp (expiration time), sub (subject), aud (audience)

• Custom claims, created to share information between parties that agree on using them and are neither registered or public claims.

If you want to play with JWT and put these concepts into practice, you can use jwt.io Debugger to decode, verify, and generate JWTs.

Signature

Signature is very important for security, so we can verify that no one tampered with our token data.

In general, JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA (although Auth0 supports only HMAC and RSA). When tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it.

Do not put secret information in the payload or header elements of a JWT unless it is encrypted. Anyone can read claims! Because it' just base64.

Go Server

In Go, the most popular package to work with JWT is golang-jwt/jwt.

go get -u github.com/golang-jwt/jwt/v5

Below is a simple example in a single file of the HTTP server that does the following:

• Signing JWT tokens

• Verify JWT tokens in middleware

• Use RS256 as signing method

• Uses labstack/echo router (my favorite one)

Step 1. PEM keys

There are different signing methods. The most popular ones are HS256 and RS256.

HS256 is symmetric, which requires the decoder to also have the secret private key, which may be an issue.

RSA on the other hand is assymetric, which means that there are two keys: one public key and one private key that must be kept secret.

The most secure practice, is to use RS256 because:

• With RS256, you are sure that only the holder of the private key can sign tokens, while anyone can check if the token is valid using the public key.

• With RS256, if the private key is compromised, you can implement key rotation without having to re-deploy your application or API with the new secret (which you would have to do if using HS256).

Let’s generate our private and public RSA keys:

openssl genrsa -out private_key.pem 2048
openssl rsa -pubout -in private_key.pem -out public_key.pem

We can then load them in our Go Server:

var (
	publicKey  *rsa.PublicKey
	privateKey *rsa.PrivateKey
)

// Load both public (verify) and private (sign) RSA keys
func init() {
	publicKeyData, err := os.ReadFile("./public_key.pem")
	if err != nil {
		log.Fatal(err)
	}
	publicKey, err = jwt.ParseRSAPublicKeyFromPEM(publicKeyData)
	if err != nil {
		log.Fatal(err)
	}

	privateKeyData, err := os.ReadFile("./private_key.pem")
	if err != nil {
		log.Fatal(err)
	}
	privateKey, err = jwt.ParseRSAPrivateKeyFromPEM(privateKeyData)
	if err != nil {
		log.Fatal(err)
	}
}

Make sure you store these PEM keys securely, for example using Kubernetes Secrets.

Step 2. Issuer

Our server will have an open login endpoint to exchange credentials for a JWT. Here we can go to the database and verify the password (we’ll skip this part in the example).

The party that issues the JWT needs to use the private RSA key.

func main() {
	e := echo.New()
	e.Use(middleware.Logger())

	e.POST("/login", login)

	e.Start("127.0.0.1:4242")
}

// We can add custom claims here
type jwtClaims struct {
	jwt.RegisteredClaims
}

func login(c echo.Context) error {
	username := c.FormValue("username")
	password := c.FormValue("password")

	// TODO: implement real auth by checking user in the database
	if username != "package" || password != "main" {
		return echo.ErrUnauthorized
	}

	// Set expiration time (1h)
	claims := &jwtClaims{
		jwt.RegisteredClaims{
			Subject:   username,
			ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour)),
		},
	}

	token := jwt.NewWithClaims(jwt.SigningMethodRS256, claims)

	t, err := token.SignedString(privateKey)
	if err != nil {
		return echo.ErrInternalServerError
	}

	return c.JSON(http.StatusOK, echo.Map{
		"token": t,
	})
}

The result of a successful authentication will be a JWT token that expires in 1 hour. Once the client receives the token, there is no way to invalidate it (expire). To minimize misuse of a JWT, the expiry time is usually kept in the order of a few minutes. Typically the client application would refresh the token in the background using the /refresh route for exanple.

Step 3. Middleware

Once the user is logged in, each subsequent request will include the JWT, allowing the user to access the “/api“ route. Usually done via the “Authorization“ header.

The verify uses the public PEM key.

func main() {
	e := echo.New()
	e.Use(middleware.Logger())

	config := echojwt.Config{
		NewClaimsFunc: func(c echo.Context) jwt.Claims {
			return new(jwtClaims)
		},
		SigningKey:    publicKey,
		SigningMethod: jwt.SigningMethodRS256.Name,
	}

	g := e.Group("/api")
	g.Use(echojwt.WithConfig(config))
	g.GET("/greet", greet)

	e.Start("127.0.0.1:4242")
}

type jwtClaims struct {
	jwt.RegisteredClaims
}

func greet(c echo.Context) error {
	user := c.Get("user").(*jwt.Token)
	claims := user.Claims.(*jwtClaims)
	sub := claims.Subject
	return c.String(http.StatusOK, fmt.Sprintf("hi %s!", sub))
}

The JOSE framework

It’s also important to mention that apart from JWT, there are other very related standards that are usually called a JOSE framework and include:

• JWS: JSON Web Signature, a specification for digitally signing JSON data.

• JWE: JSON Web Encryption, a specification for encrypting JSON data.

• JWK: JSON Web Key, a specification for representing cryptographic keys in JSON format.

• JWT: JSON Web Token, a compact and standardized way of securely transmitting information between parties using JSON data structures.

• JWA: JSON Web Algorithms, a specification for defining cryptographic algorithms used in JWE and JWS.

And you can use this useful package lestrrat-go/jwx that has implementation of these JOSE technologies for Go.

Security Best Practices

• Keep it safe and simple, do not put secret info into JWT. It is because JWTs are not encrypted (only encoded with base64) and can be read by anyone if leaked.

• Embrace HTTPS. TTPS (SSL/TLS) prevents interception and Man-in-the-Middle attacks, ensuring tokens are transmitted securely.

• Use RS256 (asymmetric signing) with public/private key pairs. HS256 is more performant though.

• Carefully store JWT. For example local storage is not the best place, due to security risks like XSS (Cross-Site Scripting) attacks.

Resources

• Source Code

In case you prefer a video format

Introduction

Change Data Capture (CDC) enables real-time tracking of database changes, critical for event-driven systems, analytics pipelines, and synchronizing microservices. This guide walks through implementing PostgreSQL CDC in Go using native logical replication and the pgx driver.

1. Understanding PostgreSQL CDC

PostgreSQL provides CDC via logical replication, which decodes changes from the write-ahead log (WAL) into consumable events (inserts/updates/deletes). Key concepts:

• Replication Slots: Persistent channels for streaming changes.

• Publications: Define which tables to monitor.

• Logical Decoding Plugins: Convert WAL entries to readable formats (e.g., pgoutput, wal2json).

2. Prerequisites

a) PostgreSQL configured for replication

# postgresql.conf  
wal_level = logical  
max_replication_slots = 5

In GCP with cloudSQL, you have to enable some flags:

cloudsql.logical_decoding = on 

b) Replication user

CREATE ROLE repl_user WITH LOGIN REPLICATION PASSWORD 'password';  
GRANT SELECT ON ALL TABLES IN SCHEMA public TO repl_user;

3. Implementing CDC in Go

Preambule:

pglogrepl is a Go library designed for interacting with PostgreSQL’s logical replication protocol. Built atop the github.com/jackc/pgx/v5/pgconn package, it enables Go applications to connect to PostgreSQL databases and process logical replication messages, making it well-suited for building Change Data Capture (CDC) solutions, streaming data pipelines or custom replication clients..

Note: replication=database in the DSN is actually required

import (
   "github.com/jackc/pgx/v5"
   "github.com/jackc/pgx/v5/pgconn"
)

type event struct {
	Relation string
	Columns  map[string]any
	Operation     string
}

func main() {
  conn, _ := pgx.Connect(context.Background(), "postgres://repl_user:password@localhost:5432/dbname?replication=database")

   c := make(chan event)
   ReadAndDefer(context.Background(), conn.PgConn(), c)
}

Step 1: Start replication

func startReplication(ctx context.Context, conn *pgconn.PgConn) error {
	var err error
	
        pubName := os.Getenv("PUBLICATION_NAME")
	// 1. Drop publication if it exists
	if _, err := conn.Exec(ctx, fmt.Sprintf("DROP PUBLICATION IF EXISTS %s;", pubName)).ReadAll(); err != nil {
		return err
	}
        // create the publication for all tables
	if _, err := conn.Exec(ctx, fmt.Sprintf("CREATE PUBLICATION %s FOR ALL TABLES;", pubName)).ReadAll(); err != nil {
		return err
	}

	// 3. create temporary replication slot server
	if _, err = pglogrepl.CreateReplicationSlot(ctx, conn, os.Getenv("SLOT_NAME"), os.Getenv("OUTPUT_PLUGIN"), pglogrepl.CreateReplicationSlotOptions{Temporary: true}); err != nil {
		return err
	}

	pluginArguments := []string{
		"proto_version '1'",
		fmt.Sprintf("publication_names '%s'", pubName),
		"messages 'true'",
	}

	// 4. establish connection
	return pglogrepl.StartReplication(ctx, conn, os.Getenv("SLOT_NAME"), pglogrepl.ParseLSN("0/0"), pglogrepl.StartReplicationOptions{PluginArgs: pluginArguments})
}

Step 2: receive message

func ReadAndDefer(ctx context.Context, conn *pgconn.PgConn, c chan event) error {
    err := startReplication(ctx, conn)
    ...
    relations := map[uint32]*pglogrepl.RelationMessage{}
    typeMap := pgtype.NewMap()

    for {
        msg, err := conn.ReceiveMessage(ctx)

        switch msg := msg.(type) {
	case *pgproto3.CopyData:
	   switch msg.Data[0] {
	   case pglogrepl.PrimaryKeepaliveMessageByteID:
              // Parse primary keep alive message
              // update client XLogPos if necessary
	   case pglogrepl.XLogDataByteID:
	      walLog, err := pglogrepl.ParseXLogData(msg.Data[1:])
	      if err != nil {
	         log.Printf("failed to parse logical WAL log: %v", err)
	      }
	      if err := parseAndDispatchMessage(&walLog, relations, typeMap, c); err != nil {
	          log.Printf("failed to parse and dispatch message: %v", err)
	      }
	}
	default:
	    log.Printf("received unexpected message: %T\n", msg)
	}
    }
}

Step 3: Parse messages

func decodeTextColumnData(mi *pgtype.Map, data []byte, dataType uint32) (any, error) {
   if dt, ok := mi.TypeForOID(dataType); ok {
      return dt.Codec.DecodeValue(mi, dataType, pgtype.TextFormatCode, data)
   }
   return string(data), nil
}


func parseAndDispatchMessage(walLog *pglogrepl.XLogData, relations map[uint32]*pglogrepl.RelationMessage, typeMap *pgtype.Map, c chan event) error {
	msg, err := pglogrepl.Parse(walLog.WALData)
	if err != nil {
		return err
	}
	switch m := msg.(type) {
	case *pglogrepl.RelationMessage:
	    relations[m.RelationID] = m
	case *pglogrepl.InsertMessage:
	    rel, ok := relations[m.RelationID]
	    if !ok {
	        log.Printf("relation %d not found", m.RelationID)
		break
	    }
	    values := map[string]any{}
	    for idx, col := range m.Tuple.Columns {
		colName := rel.Columns[idx].Name
		switch col.DataType {
		case 'n': // null
		    values[colName] = nil
		case 't': // text
		    val, _ := decodeTextColumnData(typeMap, col.Data, rel.Columns[idx].DataType)
		    values[colName] = val
		}
		event := event{
		   Relation: rel.RelationName,
		   Columns:  values,
		   Operation:     "insert",
		}
		c <- event
	case *pglogrepl.UpdateMessage:
            ...
	case *pglogrepl.DeleteMessage:
            ...
	}
	return nil
}

5. Handling Decoded Changes

The pglogrepl library parses raw WAL data into structured messages. For example, an InsertMessage contains:

• RelationID: Table identifier

• Tuple: New row values

• Columns: Metadata about columns

Example Output:

INSERT: {RelationID: 16393, Tuple: [{Column: user_id, Value: 123}, ...]}

Drawbacks

• No automatic schema replication: (ALTER TABLE ..) must be manually applied to subscribers.

• Primary Key requirement: tables must have a primary key or unique index for replication.

• Large objects: BLOB/CLOB are not replicated

• Non-table objects: Views, materialized views and foreign tables are excluded.

• The complexity around handling replication versus the simplicity of other solutions. (tradeoff performance vs maintenance)

Conclusion

PostgreSQL's logical replication paired with Go's pgx provides a robust foundation for building CDC pipelines. By streaming changes directly from the WAL, you achieve low-latency access to database events while minimizing performance overhead.

Resources:

• pgx Documentation

• PostgreSQL Logical Replication Guide

• pglogrepl Example Code

This implementation gives you full control over change processing, making it ideal for building reactive systems that respond instantly to database changes.

Errors are values

I've been using Go for probably 8 years, and it's been my go-to language for most of the projects. I think we can all agree that Go can be verbose sometimes, which attracts some people (like me) but also is a hot topic often. Specifically when it comes to error handling, when the big portion of your codebase might be pure error handling.

What's beautiful in Go about error handling is that errors are values, this simplifies passing them around, modifying them as you want, because as Rob Pike said:

To summarize, a function in Go can return an error, or even multiple:

func Open(name string) (*File, error)

The users of this function can simply check for nil error value:

f, err :=  os.Open("main.zig")
if err !=  nil {
    // process the error
}

// one-liners are also possible
if err := f.Close(); err != nil {
    log.Fatal(err)
}

error type is an interface type, which makes it possible to create custom error types that can hold various data.

type errorString struct {
    s string
}

func (e *errorString) Error() string {
    return e.s
}

func New(text string) error {
    return &errorString{text}
}

That's basically it, there is also an `errors` package that has some handy utilities for working with errors in Go. For example you can check the error type by doing:

if errors.Is(err, fs.ErrNotExist) {
    // process error
}

You can read about error handling in Go in detail here, but as you probably noticed from the title of this article, we wanted to talk about Zig here as well, specifically about its error handling as well.

Few notes on Zig

Heads up, I'm still new to Zig, yes I wrote few dead simple programs, but I'm still learning.

I found some similarities between Go and Zig, as both languages have adopted a design philosophy of simplicity to remove the language out of the way to let you be productive quickly.

Zig, however, stays true to its "no hidden control flow" and requires a programmer to manage the memory allocations manually. I always programmed in languages with GC, so I was used to the fact that allocations happen behind the scenes.

Memory allocation can fail too, and Zig makes allocation failures explicit, so here is our segue into error handling in Zig!

Error handling in Zig: 101

Zig like Go treats errors like values, but does it through a specialized enum that can be created implicitly.

Now, this is where things get interesting. The function below returns an !usize where the exclamation mark ! indicates that this function can return an error. We can either be explicit about the error(s) that our function returns or implicit. In this case, we've taken the implicit route.

fn get_args_count(allocator: std.mem.Allocator) !usize {
    const args = try std.process.argsAlloc(allocator);
    if (args.len < 1) {
        return error.EmptyArgs;
    }

    return args.len;
}

This code listing shows another mechanism of propagating an error up the call stack using the try keyword. We must do that, because std.process.argsAlloc might return an error, and all errors in Zig must be handled or we'll get a compile time error. The equivalent Go code for “const args = try std.process.argsAlloc(allocator);“ would be:

args, err := std.process.argsAlloc(allocator)
if err != nil {
    return nil, err
}

Zig version is definitely more concise. But what if you want to handle the error first? Similar to exception handling in other languages, Zig uses the catch keyword to intercept errors instead of propagating the error up the call stack.

// notice here, our main function can't return an error,
// otherwise its return type would be !void
pub fn main() void {
    const args_count = get_args_count(allocator) catch |err| {
        // do some stuff, maybe log an error
        std.debug.print("invalid input: {}\n", .{err});
        return;
    };
}

By the way, this can be nicely paired with a fallback return using the following 2 forms. Simple fallback value:

const args_count = get_args_count(allocator) catch 0;

Or by using Zig's named blocks:

const args_count = get_args_count(allocator) catch blk: {
    // do some stuff, maybe log an error

    // and then return a result
    break :blk 0;
};

And lastly, we can use if-else-switch construct to handle potential errors more precisely:

if (get_args_count(allocator)) |args_count| {
    std.debug.print("got {d} args\n", .{args_count});
} else |err| switch (err) {
    error.EmptyArgs => {
        std.debug.print("invalid input: no args\n", .{});
    },
    else => {
        std.debug.print("unexpected error: {}\n", .{err});
    },
}

Remember I mentioned that in Zig selectively omitting error handling is not allowed? That's true, but sometimes you want skip error handling in places where it's not really necessary, for example tests or build code, or some quick scripts. For that you can use catch unreachable in functions that do not return an error, but that could possibly panic.

pub fn main() void {
    const args_count = get_args_count(allocator) catch unreachable;
    std.debug.print("got {d} args\n", .{args_count});
}

Zig errors and context

At the end of the day, Zig errors are just enum values and don't contain much information or behaviour, however they do carry (not in the enum itself) a debugging trace which is in theory is accessible at runtime with errorReturnTrace.

Contrary to Zig (as we've seen in Go section), in Go, errors are interfaces, allowing them to be concrete types that can carry additional context.

For instance, when dealing with file operations, a Go function might return an error of type os.PathError. This type includes fields like Op (the operation), Path (the file path) and the underlying system error. And this is very helpful. There was an open issue in Zig to allow for something like that, but it didn't happen and developers can use tagged unions instead to create generic results.

Conclusion

There are different approaches to handle errors and every language does it differently. I personally find that both Go and Zig have the best error handling compared to other languages, that's the reason why I wanted to write this article and give you a tiny taste of Zig's error handling.

Both Go and Zig treat errors as values, emphasizing explicit handling. Zig's approach, with its specialized enum, try, and catch keywords, presents a feature-rich and concise way to manage errors.

While Go's error-as-value using interfaces is straightforward and allows for rich contextual errors (like os.PathError), it can often lead to more verbose code due to explicit if err != nil checks. Despite this verbosity, Go's simplicity makes it easy to grasp.

Let me know in the comments what you think about error handling in these 2 programming languages.

Resources

• Errors are values

• Errors in Zig #1

• Errors in Zig #2

Previously, managing tools like linters (golangci-lint), code generators (mockery, oapi-codegen), or other build utilities often involved manual installation steps (using “go install” or some package manager) outside the project's dependency management, leading to potential version inconsistencies across development environments and CI systems.

I’ve seen it many times when two developers generate different code by using the same command, only because the versions of these tools are different. There are workarounds to “fix“ the versions in fake “tools.go” file, but it’s not enough and felt dirty.

But not anymore! Go 1.24 addresses these challenges directly with the “tool” directive.

Adding a new tool to go.mod

Let’s take oapi-codegen as an example, to add it to our Go project we can simply use “go get -tool“:

go get -tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@latest

Once installed, the tool is available for use via “go tool“. Note, that the first run could be a bit slow because Go will compile the tool if it’s not yet compiled, repeated executions are much faster. Which should not be a big problem for dev tools, but worth mentioning.

go tool oapi-codegen

Note, that oapi-codegen could be installed globally with a different version. Calling “go tool“ without a tool name will list all the tools. Note there are some built-in to Go, so you will always have a few.

The tool will be added to a project and listed in our go.mod file, so all users of our software will understand what tools and what versions are required.

tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen

require (
  github.com/oapi-codegen/runtime v1.1.1
)

By the way, while not a huge deal, I find “go tool toolname“ a bit verbose as you always have to type it. But it doesn’t matter if you use “go:generate“ for example.

Pairing with go:generate

What's also great is that we can leverage that in go:generate annotations, which completely eliminates a manual tool installations. Traditionally, we would rely on the tool being installed globally on the host:

main.go:

// go:generate oapi-codegen openapi.yaml

But now we could use the “tool“:

// go:generate go tool oapi-codegen openapi.yaml

Which will use the correct version pinned in your go.mod file.

# A single command to run all generators!

go generate

Separate go.mod file

It is also possible to use a separate go.mod file for dev tools in case you don’t want to mix it with your regular dependencies or you have some issues with the versions clashing.

go tool -modfile=tools.mod oapi-codegen

Non-Go tools

Only Go tools are supported. However, some projects will likely depend on some non-Go tools, which have to be managed separately.

Conclusion

With the introduction of the “tool” directive in Go 1.24, Go modules gain a much-improved way to handle tool dependencies, similar to “npm run“ in Node.js world.

Resources

• Official docs

Well-structured and documented APIs are pleasure to work with. And nowadays the de-facto standard is OpenAPI, which comes with a good methodology of defining an API interface first, and only then constructing everything around it. Making it easier to understand, implement, and consume those APIs. And standards matter, they allow different teams, regardless of their technology stack, to effectively communicate about and work with the same API.

In this practical guide I want to walk you through all important parts involved in architecting, implementing and consuming the API with help of OpenAPI standard.

Before we dive in, it's helpful to have a basic understanding of the following:

• Go programming language

• RESTful APIs

• JSON/YAML

• Basic command-line usage

What is OpenAPI and OAS?

The OpenAPI Specification (OAS) was originally based on the Swagger 2.0 Specification from SmartBear Software. Later it was moved to the OpenAPI Initiative (OAI), a consortium of industry experts under the Linux Foundation.

The main idea of OpenAPI is to be able to describe in agnostic terms, decoupling them from any specific programming language. Consumers of your API specification do not need to understand the guts of your application or try to learn Lisp or Haskell if that’s what you chose to write it in. They can understand exactly what they need from your API specification, written in a simple and expressive language.

This simple and expressive language is called DSL (domain specific language). It can be written in either JSON or YAML.

The latest version of OAS is v3.1.1 and the specification itself is huge, there are many features and corner cases, but we will try to go through the most important ones.

Architecting the API

It all starts with defining what the API should provide for its consumers and what it is for. While this stage isn't always purely technical, having a sketch of your API design in OAS when gathering requirements gives you a headstart when starting design.

Once the requirements are ready, it's time to open your OpenAPI editor and collaborate with your teammates.

And it's important to understand that it's not only about writing JSON/YAML spec, but actually agreeing on the API design.

It's recommended to follow some API design guide, Google has one for example. So you don't end up having mixed styles like /resourceName/{id} and /resource_name/{id}, inconsistent use of HTTP methods, or unclear resource relationships.

openapi.yaml

The spec of your API starts in the entrypoint document openapi.yaml (recommended but not required name) or openapi.json. I've seen very big openapi.yaml files (50k lines), but it's possible to split your spec into multiple parts. However, this may not work well for some OpenAPI tools as they expect a single file. Google Maps OAS is a good example on how to split the schema, but also comes with a pre-processor to generate a single file.

There are some open source tools to bundle the OAS: swagger-cli (archived) and redocly-cli are great options.

swagger-cli bundle -o _bundle/openapi.yaml openapi.yaml

As I mentioned earlier, the spec is huge, but let's break it into smaller parts. For this post I created a dummy "Smart Home" API, you can see the full spec and code here.

The root object is called OpenAPI Object and has the following structure:

# schema version
openapi: 3.1.1

# docs
info:
  title: Smart Home API
  description: API Specification for Smart Home API
  version: 0.0.1

# optional servers for public APIs
servers:
  - url: "https://..."

# tags are used to group the endpoints
tags:
  - name: device
    description: Manage devices
  - name: room
    description: Manage rooms

# endpoints go here
paths:
  # ...

# reusable objects such as schemas, error types, request bodies
components:
  # ...

# security mechanisms, should correspond to components.securitySchemes
security:
  - apiKeyAuth: []

We defined the skeleton of our schema, but the majority of OpenAPI schema lays in paths and components props.

Paths and Operations

Let's now add few endpoints to our schema. The operations are grouped by paths, so you can have multiple HTTP methods on a single path, for example GET /devices/{deviceId} and DELETE /devices/{deviceId}.

It's a good practice to define all types (request bodies, responses, errors) in components section and reference them instead of manually defining them in paths section. This allows for easier re-use of entities. For example in our API we have a type Device which can be used in many endpoints.

paths:

  # the path has a parameter in it
  /devices/{deviceId}:
    get:
      tags:
        - device
      summary: Get Device
      operationId: getDevice

      parameters:
        - name: deviceId
          in: path
          required: true
          schema:
            $ref: "#/components/schemas/ULID"

      responses:

        "200":
          description: Success
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Device"

        "404":
          description: Not Found
          content:
            application/json:
              schema:
                # use common type for 404 errors
                $ref: "#/components/schemas/ErrorNotFound"

In the spec above we defined two endpoints of our API and referenced the types which we still need to define: Device, ErrorNotFound and ULID. Notice that for deviceId path param we also used a custom type instead of a standard string, which can be helpful in the future in case we want to change the format of our IDs (for example UUID, ULID. integer, etc.).

Notice that each operation has a unique operationId, while it's optional it's very helpful to set one, so then it can be used on the server and client sides.

This is a basic configuration and can be extended further if we want to. For example, when serving this schema in Swagger, it's good to see the examples of our requests (and their variations). We can define it here in responses section, or directly in our components.schemas.

responses:
  "200":
    content:
      application/json:
        examples:
          new_device:
            value: # any value

Schemas

components is an integral part of OAS, it contains the following properties:

• schemas

• responses

• parameters

• requestBodies

• headers

• securitySchemes

• see all here

We could define our Device type like this:

components:
  schemas:
    Device:
      type: object
      properties:
        id:
          $ref: '#/components/schemas/ULID'
        name:
          type: string
      required:
        - id
        - name

But later you may have other types that have name or id fields, so it's recommended to define them separately and combined in final type using allOf:

components:
  schemas:
    WithId:
      type: object
      required:
        - id
      properties:
        id:
          $ref: "#/components/schemas/ULID"

    WithName:
      type: object
      required:
        - name
      properties:
        name:
          type: string

    Device:
      allOf:
        - $ref: "#/components/schemas/WithId"
        - $ref: "#/components/schemas/WithName"

allOf, oneOf and anyOf are very powerful techniques for modelling your OAS.

Extensions

OpenAPI schema can be extended with internal properties that do not affect the schema itself, but are useful for server or client generators. A good example is our ULID type for ids:

ULID:
  type: string
  minLength: 26
  maxLength: 26

  # example is useful for Swagger docs
  example: 01ARZ3NDEKTSV4RRFFQ69G5FAV

  x-go-type: ulid.ULID
  x-go-type-import:
    path: github.com/oklog/ulid/v2

The x- props will be used by Go server generator to use existing Go types for this field instead of generating a new one.

Generate a Go Server

We didn't go through all possible schema properties and just covered the main ones, so people who are not familiar with OAS have a good understanding of this standard. You can read the whole specification here. But now as our schema is ready we can generate a Go server from it.

You can find the full list of generators on opeanapi.tools, there are a lot of them. However, the most popular one for Go servers is oapi-codegen.

oapi-codegen currently doesn’t support OAS 3.1. Issue

ogen does, however.

You can install it via go install:

go install github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@latest

The configuration for the oapi-codegen generator is straightforward. You can either provide command line arguments or specify the same arguments in a yaml configuration file. You can choose which HTTP router to use for the server, where to put the output file, and more. In our case let's use echo router.

# oapi-codegen.yaml

package: api
output: pkg/api/api.gen.go

generate:
  strict-server: true
  models: true
  echo-server: true

We can now generate the server code using the following command:

oapi-codegen --config=oapi-codegen.yaml openapi.yaml

Let's explore now the generated api.gen.go file. It contains the following parts.

Since we enabled strict-server, which will generate code that parses request bodies and encodes responses automatically, the interface that we need to implement is called StrictServerInterface:

type StrictServerInterface interface {

  // List Devices
  // (GET /devices)
  ListDevices(ctx context.Context, request ListDevicesRequestObject) (ListDevicesResponseObject, error)

  // Get Device
  // (GET /devices/{deviceId})
  GetDevice(ctx context.Context, request GetDeviceRequestObject) (GetDeviceResponseObject, error)

}

All our types are also generated:

type ULID = ulid.ULID

type Device struct {
	Id   ULID   `json:"id"`
	Name string `json:"name"`
}

// ...

As well as:

• Code to parse the requests automatically

• Swagger definition

Implementation

What's left for us to do is to create a server using echo, implement the generated interface and glue everything together. We can write the following code in pkg/api/impl.go:

package api

import "context"

type Server struct{}

func NewServer() Server {
	return Server{}
}

func (Server) ListDevices(ctx context.Context, request ListDevicesRequestObject) (ListDevicesResponseObject, error) {
	// actual implementation
	return ListDevices200JSONResponse{}, nil
}

func (Server) GetDevice(ctx context.Context, request GetDeviceRequestObject) (GetDeviceResponseObject, error) {
	// actual implementation
	return GetDevice200JSONResponse{}, nil
}

I skipped the implementation part and just demonstrated how to return the responses, it's quite handy that oapi-codegen generated all possible responses for us.

That leaves us to start the echo server itself. Note that we don't need to write any endpoints manually now, and all request and response parsing is handled for us. Still, we need to validate the requests inside our implementation.

package main

import (
	"oapiexample/pkg/api"

	"github.com/labstack/echo/v4"
)

func main() {
	server := api.NewServer()

	e := echo.New()

	api.RegisterHandlers(e, api.NewStrictHandler(
		server,
		// add middlewares here if needed
		[]api.StrictMiddlewareFunc{},
	))

	e.Start("127.0.0.1:8080")
}

Now when we run our server using go run . we can curl localhost:8080/devices to se the response!

Supported servers

oapi-codegen supports many web frameworks/servers, such as Chi, Fiber, Gin as well as standard net/http.

Swagger UI / Postman

Sometimes it's handy to have Swagger docs shipped together with your API, for testing for example or just as public documentation. oapi-codegen doesn't generate the Swagger UI out of the box but we can have a simple html page that has a Swagger JS which loads our OAS.

You can find the html code for our pkg/api/index.html here.

And then we can use go:embed to embed the static files and add our swagger endpoint:

//go:embed pkg/api/index.html
//go:embed openapi.yaml
var swaggerUI embed.FS

func main() {
	// ...

	// serve swagger docs
	e.GET("/swagger/*", echo.WrapHandler(http.StripPrefix("/swagger/", http.FileServer(http.FS(swaggerUI)))))
}

Now we can visit localhost:8080/swagger/ to see the Swagger UI with our OAS.

Tools like Postman are very popular for API documentation, and it's also possible to import your existing OpenAPI 3.0 and 3.1 definitions into Postman. Postman supports both YAML and JSON formats.

Generate OAS from code

There is also a practice to generate OpenAPI schema from code, especially in typed languages. This approach has been popular, with the main selling point that keeping OpenAPI schema near the code will hopefully mean developers keep it up to date as they work on the code. This is not always the case, which is one of a few reasons this practice is dying out. And I am also not a big fan of that as I haven’t seen a big value in this. Anyway, you can have a look at the following projects: go-swagger, swag, swaggest/rest.

Client Code

As mentioned earlier, OpenAPI is very powerful for collaboration between teams, and all you have to do now is to properly version your schema (see info.version part) and distribute it across the teams. This part can be automated to some extent by packaging your OpenAPI schema and making it available. I've seen that devs use git submodules for that or GitHub actions to publish the version schemas.

Let's assume our client is a web application written in Typescript, which is quite common for web APIs. Again, there are may generators available at opeanapi.tools online but the most popular one is [openapi-typescript.

Here's how you can generate the Typescript code for local or remote schemas:

# Local schema
npx openapi-typescript openapi.yaml -o ./client/schema.d.ts

# Remote schema
npx openapi-typescript https://.../openapi.yaml -o ./client/schema.d.ts

Conclusion

OpenAPI is a de-facto standard for designing, implementing and consuming the REST APIs, so it's crucial to understand how it works. I hope this article has provided a useful introduction to OpenAPI Specification, as well as practical tips and examples for how to use OAS to architect, implement and consume APIs.

Resources

• Watch on YouTube

• Source code

• OpenAPI Initiative

• openapi.tools

• Swagger Editor

• oapi-codegen

Modern applications often rely on data scattered across multiple databases. PostgreSQL’s Foreign Data Wrapper (FDW) empowers you to query and manipulate external databases as if they were native PostgreSQL tables. Combined with pg_cron for scheduling and stored procedures for automation, you can build robust archival pipelines.

In this post we will talk about:

• Setting up FDW to connect to external databases (ex: another PostgreSQL instance).

• Importing foreign schemas.

• Creating archival policies with stored procedures

• Automating archival tasks with pg_cron

What is PostgreSQL FDW?

FDW is an extension that implements the SQL/MED standard, allowing PostgreSQL to interact with external data sources.

Key benefits:

• Unified Query Interface: Run SQL across heterogenous databases (MySQL, MongoDB, Hadoop etc..)

• Real-Time data access: No need for ETL pipelines for simple joins.

• Simplified Archival: Move historical data to cheaper storage while keeping it queryable.

Foreign Data Wrappers

There is a lot of different FDW extensions that will let you connect to a variety of databases (relational, no-sql, files etc..)

• postgres_fdw

  • Connects to other PostgreSQL databases (local or remote)

  • Supports read/write operations

• file_fdw

  • Reads data from flat files (CSV, TSV, etc..)

• mysql_fdw

  • Connects to MySQL/MariaDB databases

  • Supports basic queries and joins

• redis_fdw

  • Query Redis key-value stores

• mongo_fdw

  • Access MongoDB collections

• aws_s3_fdw

  • Read/write data from Amazon S3

There are many others, which you can find here.

Use cases

As you can imagine, there are many use cases possible.

From reading files from your database, to getting cache keys from Redis and augment them from content stored in your PostgreSQL.

The use case we will focus on, is an automatic archival from PostgreSQL to PostgreSQL. But it would also work from PostgreSQL to Hadoop | MySQL | Bigquery etc..

Setting up FDW and connecting to a foreign database

Step 1: Install required extension

-- Enable FDW   
CREATE EXTENSION IF NOT EXISTS postgres_fdw;

Step 2: configure a foreign server (ex: PostgreSQL)

-- Create a foreign server
CREATE SERVER postgres_server FOREIGN DATA WRAPPER postgres_fdw
   OPTIONS (host '{FOREIGN_HOST}', dbname '{FOREIGN_DB_NAME}');

-- Map local PostgreSQL user to foreign server credentials
CREATE USER MAPPING FOR postgres SERVER postgres_server
   OPTIONS (user '{FOREIGN_USERNAME}', password '{FOREIGN_PASSWORD}');

Step 3: Import foreign schema

You might be tempted to import foreign schema from the other postgres into the “public” schema of your current instance, however you can’t have 2 tables with the same name.

That’s why, I would suggest you create a separate schema, you can call it foreign_schema or whatever you wish:

-- Create a new schema to import the tables into
CREATE SCHEMA foreign_schema;

-- Import all tables from the foreign postgreSQL database
IMPORT FOREIGN SCHEMA public FROM SERVER postgres_server INTO foreign_scehma;

-- You can also import only specific tables
IMPORT FOREIGN SCHEMA public LIMIT TO ({TABLE1}, {TABLE2}) FROM SERVER postgres_server INTO foreign_schema;

Now you can query your foreign database:

SELECT * FROM foreign_schema.{TABLE1};

Building an archival policy

Let’s imagine you have a transactions table in both your postgres with the same columns (though it’s not mandatory).

The distant server is used as archive and the current db (the one you are connected to) is your live DB.

CREATE TABLE IF NOT EXISTS transactions (
    id uuid DEFAULT uuid_generate_v4(),
    amount INT DEFAULT 0,
    created_at   TIMESTAMPTZ DEFAULT NOW(),

    PRIMARY KEY (id)
); 

Create a stored procedure to move data

CREATE OR REPLACE PROCEDURE archive_old_transactions()  
LANGUAGE plpgsql  
AS $$  
BEGIN  
  -- Move data older than 1 year to archive  
  INSERT INTO foreign_schema.transactions  
  SELECT * FROM public.transactions  
  WHERE created_at < NOW() - INTERVAL '1 year';  

  -- Delete archived data from main table  
  DELETE FROM public.transactions  
  WHERE created_at < NOW() - INTERVAL '1 year';  
END;  
$$
;  

Automate archival with pg_cron

Step1: Install required extension:

-- Enable pg cron   
CREATE EXTENSION IF NOT EXISTS pg_cron;

Step 2: Schedule the stored procedure to run nightly:

-- Run at 2 AM daily
SELECT cron.schedule(
  'archive_transactions',
  '0 2 * * *',
  'CALL archive_old_transactions()'
); 

Step 3: Create a view to merge both worlds

CREATE VIEW combined_transactions AS (
    WITH remote_data AS (
      SELECT * FROM foreign_schema.transactions
    ),
    local_data AS (
      SELECT * FROM public.transactions
    )
    SELECT * FROM remote_data
    UNION ALL
    SELECT * FROM local_data
);

You will notice that I didn’t simply do a UNION of two tables, I used CTE (Common Table Expressions) because it is crucial for optimizing queries with foreign tables.

Essentially, it containerizes the FDW query, because the query planner will have to ask the foreign database to execute its part, and the clearer this query is, the faster it will be.

Best approach is to:

• Containerize the FDW query to reduce data transfer

• Filter data at the source

• Minimize the returned row count

Indexes & foreign tables

Nothing forces you to have the exact same copy of foreign and local table, neither you have to force foreign keys or equivalent indexes.

It is probably recommended to create specific indexes in your foreign table that will match the query patterns they will be submitted to. Because you don’t query archive data the same way you might query live data.

Materialized Views

The concept of archival is often to reclaim space, but if you were to need faster access to data from both the archive and the live DB then you could use Materialized views to ensure fast queries.

Step 4: Query your view

So if you query transactions from 2 years ago from now, until now. You necessarily have to query both tables.

SELECT * FROM combined_transactions WHERE created_at BETWEEN(NOW() - INTERVAL '2 year', NOW()); 

Best Practices

1. Index Foreign Tables: Improve query performance with indexes on frequently filtered columns.

2. Batch Archival: Use LIMIT in procedures to avoid long locks.

3. Error Handling: Wrap pg_cron jobs in BEGIN...EXCEPTION blocks.

4. Monitor pg_cron: Use cron.job_run_details to track job history.

Conclusion

PostgreSQL FDW transforms your database into a unified gateway for cross-database operations. By combining it with pg_cron and stored procedures, you can automate complex workflows like archival, reporting, and data synchronization without external tools.

Further Reading:

• PostgreSQL FDW Documentation

• pg_cron GitHub Repository

• FDW Wrappers for MongoDB, CSV, etc.

We, developers, spend a lot of time in our terminal. Or maybe we could spend even more, because there are so many great CLI/TUI tools that can boost the developer productivity, or just be fun to use.

This article contains a categorized list of CLIs / TUIs I personally use and those widely adopted in the development community.

Kubernetes

k9s - Kubernetes CLI To Manage Your Clusters In Style!

K9s provides a terminal UI to interact with your Kubernetes clusters. The aim of this project is to make it easier to navigate, observe and manage your applications in the wild. K9s continually watches Kubernetes for changes and offers subsequent commands to interact with your observed resources.

ktop - A top-like tool for your Kubernetes clusters.

Following the tradition of Unix/Linux top tools, ktop is a tool that displays useful metrics information about nodes, pods, and other workload resources running in a Kubernetes cluster.

kubectx - switch between contexts (clusters) on kubectl faster.

kubectx is a tool to switch between contexts (clusters) on kubectl faster.

kubens is a tool to switch between Kubernetes namespaces (and configure them for kubectl) easily.

kubescape - Kubernetes security platform for your IDE, CI/CD pipelines, and clusters.

Kubescape is an open-source Kubernetes security platform that provides comprehensive security coverage, from left to right across the entire development and deployment lifecycle. It offers hardening, posture management, and runtime security capabilities to ensure robust protection for Kubernetes environments. It saves Kubernetes users and admins precious time, effort, and resources.

Containers

ctop - A top-like interface for container metrics.

ctop provides a concise and condensed overview of real-time metrics for multiple containers.

ctop comes with built-in support for Docker and runC; connectors for other container and cluster systems are planned for future releases.

lazydocker - A simple terminal UI for both docker and docker-compose.

Memorising docker commands is hard. Memorising aliases is slightly less hard. Keeping track of your containers across multiple terminal windows is near impossible. What if you had all the information you needed in one terminal window with every common command living one keypress away (and the ability to add custom commands as well). Lazydocker's goal is to make that dream a reality.

dive - A tool for exploring each layer in a docker image.

A tool for exploring a docker image, layer contents, and discovering ways to shrink the size of your Docker/OCI image.

Files/Text

jq - Command-line JSON processor.

jq is a lightweight and flexible command-line JSON processor akin to sed, awk, grep, and friends for JSON data. It's written in portable C and has zero runtime dependencies, allowing you to easily slice, filter, map, and transform structured data.

bat - A cat(1) clone with wings.

bat is an enhanced version of the cat command, written in Rust, integrating syntax highlighting, git integration, and automatic paging.

Its syntax highlighting supports numerous programming and markup languages, making code more readable directly in the terminal. Git integration allows users to see modifications in relation to the index, highlighting the added or changed lines.

ripgrep - Recursively search directories for a regex pattern while respecting your gitignore.

ripgrep is a line-oriented search tool that recursively searches the current directory for a regex pattern. By default, ripgrep will respect gitignore rules and automatically skip hidden files/directories and binary files. (To disable all automatic filtering by default, use rg -uuu.) ripgrep has first class support on Windows, macOS and Linux, with binary downloads available for every release. ripgrep is similar to other popular search tools like The Silver Searcher, ack and grep.

rsync - A fast and extraordinarily versatile file copying tool for both remote and local files.

Rsync uses a delta-transfer algorithm which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. At first glance this may seem impossible because the calculation of diffs between two files normally requires local access to both files.

fd - A simple, fast and user-friendly alternative to find.

fd is a program to find entries in your filesystem. It is a simple, fast and user-friendly alternative to find. While it does not aim to support all of find's powerful functionality, it provides sensible (opinionated) defaults for a majority of use cases.

Git

lazygit - Simple terminal UI for git commands.

Lazygit is a simple terminal UI for git commands, with keybindings for most actions. It aims to make working with git from the terminal easier and more intuitive.

Development

ATAC - A simple API client (postman like) in your terminal.

ATAC is Arguably a Terminal API Client. It is based on well-known clients such as Postman, Insomnia, or even Bruno, but inside your terminal without any specific graphical environment needed.

k6 - A modern load testing tool, using Go and JavaScript.

k6 is a modern load-testing tool, built on our years of experience in the performance and testing industries. It's built to be powerful, extensible, and full-featured. The key design goal is to provide the best developer experience.

httpie - modern, user-friendly command-line HTTP client for the API era.

HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI interaction with web services as human-friendly as possible. HTTPie is designed for testing, debugging, and generally interacting with APIs & HTTP servers. The http & https commands allow for creating and sending arbitrary HTTP requests. They use simple and natural syntax and provide formatted and colorized output.

asciinema - Terminal session recorder.

asciinema (aka asciinema CLI or asciinema recorder) is a command-line tool for recording terminal sessions.

Unlike typical screen recording software, which records visual output of a screen into a heavyweight video files (.mp4, .mov), asciinema recorder runs inside a terminal, capturing terminal session output into a lightweight recording files in the asciicast format (.cast).

Networking

doggo - Command-line DNS Client for Humans.

doggo is a modern command-line DNS client (like dig) written in Golang. It outputs information in a neat concise manner and supports protocols like DoH, DoT, DoQ, and DNSCrypt as well.

gping - Ping, but with a graph.

gping is an excellent tool for network, system, devops engineers and for anyone looking to visualize their ping output. It's best utilized in environments where readability and visual communication of ping results is essential.

Workstation

tmux - A terminal multiplexer.

tmux is a terminal multiplexer. It enables a number of terminals to be created, accessed, and controlled from a single screen.

tmux may be detached from a screen and continue running in the background, then later reattached.

zellij - A terminal workspace with batteries included.

Zellij is a workspace aimed at developers, ops-oriented people and anyone who loves the terminal. Similar programs are sometimes called "Terminal Multiplexers".

Zellij is designed around the philosophy that one must not sacrifice simplicity for power, taking pride in its great experience out of the box as well as the advanced features it places at its users' fingertips.

btop - A monitor of resources.

Resource monitor that shows usage and stats for processor, memory, disks, network and processes.

Conclusion

These CLIs/TUIs should work well in any modern terminal, I personally use Ghostty currently and it works great.

There's a huge amount of CLIs/TUIs out there, and we can't list them all (though we tried).

With growing amount of code editors and programming languages this became the classic M*N complexity problem.

But then Microsoft introduced the Language Server Protocol (LSP) as a solution to the problem above, which elegantly transforms this M*N complexity into a more manageable M+N situation.

The LSP was initially driven by the needs of VS Code

• LSP separates language servers from code editors (language clients). By making language servers independent processes dedicated to language understanding, the LSP enables any editor to utilize a standard language server. Which means that a single standard language server can be used by all editors.

• This interoperability is achieved through a defined set of standard messages and procedures that govern communication between language servers and editors. The LSP defines the format of the messages sent using JSON-RPC between the development tool and the language server.

Subscribe now

Language Server Features

The list of features may vary for each individual language server, but usually they provide the following functionalities:

• Auto-completion

• Go to definition/declaration

• Find references

• Code formatting

• Diagnostics

• Documentation

• etc.

For example, here you can see the list of editor features that gopls (Go Language Server) provides.

And here you can see the full LSP specification for available features.

How does LSP work?

The Language Server Protocol is built upon JSON-RPC. It specifically uses JSON RPC 2.0. You can think of JSON-RPC as a remote procedure call protocol that uses JSON for data encoding.

In a nutshell it works like this. First, the editor establishes a connection with the language server, then as the developer types code, the editor sends incremental changes to the language server. It then sends back insights like: code completion suggestions, diagnostics.

Let’s see one real example for auto-completion. The request from Language Client (editor) for this case will be:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "textDocument/completion",
  "params": {
    "textDocument": {
      "uri": "file:///home/alex/code/test/main.go"
    },
    "position": {
      "line": 35,
      "character": 21
    }
  }
}

As you can see it sends the information about current cursor position and the buffer file. Let's break it down:

• ID: The client sets this field to identify the request uniquely. Once the request is processed, it will return a response with the same request ID so that the client can match which response is for what request.

• Method: A string containing the name of the method to be invoked.

• Params: The parameters to be passed to the method. This can be structured as an array or an object.

Language server can access this file, analyze it and respond with suggestions:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "isIncomplete": false,
    "items": [
      {
        "label": "Println",
        "kind": 3,
        "insertText": "Println(${1:format}, ${2:a ...interface{}})$0",
        "insertTextFormat": 2,
        "detail": "func Println(a ...interface{}) (n int, err error)",
        "documentation": "Println formats ..."
      },
      // ... other items
    ]
  }
}

Language Server for Go

The most popular and commonly used language server for Go is gopls. It is used by many editors, for example by Visual Studio Code Go extension. Previously, there was another popular Language Server for Go by the Sourcegraph team called go-langserver, but this is no longer under active maintenance.

Many editors install gopls Language Server automatically if it’s not present on the host machine, but you can install it manually as well:

go install golang.org/x/tools/gopls@latest

gopls also provides an experimental CLI interface:

$ gopls references ./test.go:35:8

/Users/alex/code/test/test.go:55:37-51

Editors

Usually, you do not have to start the Language Server yourself as your editor is doing it for you. It also only starts the language server if a specific file has been opened. You can confirm it by checking the processes your editor has created, for example:

ps aux| grep gopls

Conclusion

Thanks to the Language Server Protocol, advanced coding capabilities are becoming universally accessible across various programming languages and coding environments.

It’s good to know how your code editors work, therefore it’s beneficial to have an understanding of this widely used technology called LSP.

LSP is a win for both language providers and tooling vendors!

Subscribe now

Resources

• Language Server Protocol

• gopls

In this post we want to show another alternative in case you use GitHub Actions as your CI platform, which became the most popular CI/CD solution at the moment.

It’s called Service Containers, and we realised that, unfortunately, not so many developers are aware of it.

In this hands-on tutorial we want to demonstrate how to create a GitHub Actions workflow for integration tests with external dependencies (MongoDB and Redis), using the demo Go application we wrote previously, as well as review the pros and cons of GitHub Service Containers.

Subscribe now

What are Service Containers?

Service Containers are Docker containers that offer a simple and portable way to host dependencies like databases (MongoDB in our example), web services, or caching systems (Redis in our example) that your application needs within a workflow. This article focuses on integration tests, however there are many other possible applications. Service containers can also be used to run supporting tools required by your workflow, such as code analysis tools, linters, or security scanners.

Why not Docker Compose?

Sounds similar to services in Docker Compose, right? Because it is.

However, while you could technically use Docker Compose within a GitHub Actions workflow by installing Docker Compose and running the docker-compose up, service containers provide a more integrated and streamlined approach specifically designed for the GitHub Actions environment.

Also, while they are similar, they solve different purpose.

• Docker Compose is good when you need to manage a multi-container application on your local machine or a single server. best suited for long-living environments.

• Service Containers are ephemeral and exist only for the duration of a workflow run, and defined directly within your GitHub Actions workflow file.

Also, the feature set of service containers (at least as of now) is more limited compared to Docker Compose, so be ready to discover some potential bottlenecks, we will cover some of them at the end of this article.

Job runtime

You can run GitHub jobs directly on a runner machine or in a Docker container (by specifying container property). The second option simplifies the access to your services by using labels you defined in the services section.

Run directly on a runner machine.

.github/workflows/test.yaml

jobs:
  integration-tests:
    runs-on: ubuntu-24.04

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017

    steps:
      - run: |
          echo "addr 127.0.0.1:27017"

Or run in a container (Chainguard Go Image in our case):

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:latest

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017

    steps:
      - run: |
          echo "addr mongo:27017"

You can also omit the host port, so the container port will be randomly assigned to a free port on the host. You can then access the port using the variable.

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:1.23

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017/tcp

    steps:
      - run: |
          echo "addr mongo:${{ job.services.mongo.ports['27017'] }}"

Readiness Healthcheck

Prior to running the job steps that connect to our provisioned containers we sometimes need to make sure that the services are ready. It’s possible to do so by specifying docker create options such as health-cmd.

This is very important, otherwise the services may not be ready when we start accessing them in the steps section.

In a case of MongoDB and Redis these will be:

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017/27017
        options: >-
          --health-cmd "echo 'db.runCommand("ping").ok' | mongosh mongodb://localhost:27017/test --quiet"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

Private Container Registries

In our example we use public images from Dockerhub, however it’s possible to use private images as well and pass the credentials:

services:
  private_service:
    image: ghcr.io/org/service_repo
    credentials:
      username: ${{ secrets.registry_username }}
      password: ${{ secrets.registry_token }}

Sharing data between services

You can use volumes to share data between services or other steps in a job. You can specify named Docker volumes, anonymous Docker volumes, or bind mounts on the host. However, it’s not directly possible to mount the source code as a container volume. Open discussion

volumes:
  - /src/dir:/dst/dir

Golang Integration Tests

Now as we can provision our external dependencies, let’s have a look at how to run our integration tests in Go. We will do it in the steps section of our workflow file.

We will run our tests in a container which uses Chainguard Go image, which means we don’t have to install/setup Go. In case you want to run your tests directly on a runner machine, you need to use setup-go Action.

You can find the full source code with tests and this workflow here.

.github/workflows/integration-tests.yaml

name: "integration-tests"

on:
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:latest

    env:
      MONGO_URI: mongodb://mongo:27017
      REDIS_URI: redis://redis:6379

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017
        options: >-
          --health-cmd "echo 'db.runCommand("ping").ok' | mongosh mongodb://localhost:27017/test --quiet"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

    steps:
      - name: Check out repository code
        uses: actions/checkout@v4

      - name: Download dependencies
        run: go mod download

      - name: Run Integration Tests
        run: go test -tags=integration -timeout=120s -v ./...

To summarize:

1. We run our job in a container with Go (container)

2. We spin up 2 services: MongoDB and Redis (services)

3. We configure healthchecks to make sure our services are “Healthy” when we run the tests (options)

4. Standard code checkout

5. Run the Go tests

Once the Action is completed (it took ~1 min for this example), all the services will be stopped and orphaned so we don’t need to worry about that.

Personal Experience & Limitations

We’ve been using service containers for running backend integration tests at BINARLY for some time, and they work great. However, the initial workflow creation took some time and we encountered the following bottlenecks:

• It’s not possible to override or run custom command in an action service container (as you would do in Docker Compose using command property). Open pull request

• It’s not directly possible to mount the source code as a container volume. Open discussion

Conclusion

GitHub service containers is a great option to scaffold the ephemeral testing environment by configuring it directly in your GitHub workflow. With configuration somehow similar to Docker Compose it’s easy to run any containerised application and to communication with it in your pipeline, making sure that GitHub runners take care of shutting everything down upon completion.

If you use Github Actions, this approach works extremely well as it is specifically designed for the GitHub Actions environment.

Subscribe now

Resources

• Source Code

• GitHub Documentation

That's where fuzz testing or fuzzing comes to the rescue.

Subscribe now

What is Fuzz Testing

Fuzzing is an automated software testing technique that involves inputting a large amount of valid, nearly-valid or invalid random data into a computer program and observing its behavior and output. So the goal of fuzzing is to reveal bugs, crashes and security vulnerabilities in source code you might not find through traditional testing methods.

The Fuzz Testing in Go video which I made a few years ago shows a very simple example of the Go code that may work well unless you provide a certain input:

func Equal(a []byte, b []byte) bool {
  for i := range a {
    // can panic with runtime error: index out of range.
    if a[i] != b[i] {
      return false
    }
  }

  return true
}

Fuzzing technique would easily spot this bug by bombarding this function with various inputs.

It's a good practice to integrate fuzzing into your team’s software development lifecycle (SDLC) as well. For example, Microsoft uses fuzzing as one of the stages in its SDLC, to find potential bugs and vulnerabilities.

Fuzz Testing in Go

There are many fuzzing tools that have been available for a while such as oss-fuzz for example, but since Go 1.18 fuzzing was added to Go's standard library and is part of the regular testing package since it is a kind of test. It can also be used together with the other testing primitives which is nice.

The steps to create a fuzz test in Go are the following:

1. In a _test.go file create a function that starts with Fuzz which accepts *testing.F

2. Add corpus seeds using f.Add() to allow fuzzer to generate the data based on it.

3. Call fuzz target using f.Fuzz() by passing fuzzing arguments which our target function accepts.

4. Start the fuzzer using regular go test command, but with the –fuzz=Fuzz flag

Note, the fuzzing arguments can only be the following types:

• string, byte, []byte

• int, int8, int16, int32/rune, int64

• uint, uint8, uint16, uint32, uint64

• float32, float64

• bool

A simple fuzz test for the Equal function above may look like this:

// Fuzz test
func FuzzEqual(f *testing.F) {

  // Seed corpus addition
  f.Add([]byte{'f', 'u', 'z', 'z'}, []byte{'t', 'e', 's', 't'})

  // Fuzz target with fuzzing arguments
  f.Fuzz(func(t *testing.T, a []byte, b []byte) {
    // Call our target function and pass fuzzing arguments
    Equal(a, b)
  })
}

By default, fuzz tests run forever, so you either need to specify the time limit or wait for fuzz tests to fail. You can specify which tests to run using --fuzz argument.

go test --fuzz=Fuzz -fuzztime=10s

If there are any errors during the execution, the output should look similar to this:

go test --fuzz=Fuzz -fuzztime=30s
--- FAIL: FuzzEqual (0.02s)
    --- FAIL: FuzzEqual (0.00s)
        testing.go:1591: panic: runtime error: index out of range
    Failing input written to testdata/fuzz/FuzzEqual/84ed65595ad05a58
    To re-run:
    go test -run=FuzzEqual/84ed65595ad05a58

Notice that the input for which fuzz test has failed are written into a file in testdata folder and can be re-played by using that input identifier.

go test -run=FuzzEqual/84ed65595ad05a58

The testdata folder can be checked into the repository and be used for regular tests, because fuzz tests can also act as regular tests when executed without --fuzz flag.

Fuzzing HTTP Services

It's also possible to fuzz test the http services by writing a test for your HandlerFunc and using the httptest package. Which can be very useful to test the whole HTTP service, not only the underlying functions.

Let's now introduce a more real example such as an HTTP Handler that accepts some user input in the request body and then write a fuzz test for it.

Our handler accepts JSON request with limit and offset fields to paginate some static mocked data. Let's define the types first.

type Request struct {
  Limit  int `json:"limit"`
  Offset int `json:"offset"`
}

type Response struct {
  Results    []int `json:"items"`
  PagesCount int   `json:"pagesCount"`
}

Our handler function then parses the JSON, paginates the static slice and returns a new JSON in response.

func ProcessRequest(w http.ResponseWriter, r *http.Request) {
  var req Request

  // Decode JSON request
  if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
  }

  // Apply offset and limit to some static data
  all := make([]int, 1000)
  start := req.Offset
  end := req.Offset + req.Limit
  res := Response{
    Results:    all[start:end],
    PagesCount: len(all) / req.Limit,
  }

  // Send JSON response
  if err := json.NewEncoder(w).Encode(res); err != nil {
    http.Error(w, err.Error(), http.StatusInternalServerError)
    return
  }

  w.WriteHeader(http.StatusOK)
}

As you may have already noticed, this function doesn't handle slice operations quite well and can easily panic. Also it can panic if it tries to divide by 0. It's great if we can spot it during the development or using only unit tests, but sometimes not everything is visible to our eye, and our handler may pass the input to other functions and so forth.

Following our FuzzEqual example above, let's implement a fuzz test for the ProcessRequest handler. The first thing we need to do is to provide the sample inputs for the fuzzer. This is the data that the fuzzer will use and modify into new inputs that are tried. We can craft some sample JSON request and use f.Add() with []byte type.

func FuzzProcessRequest(f *testing.F) {
  // Create sample inputs for the fuzzer
  testRequests := []Request{
    {Limit: -10, Offset: -10},
    {Limit: 0, Offset: 0},
    {Limit: 100, Offset: 100},
    {Limit: 200, Offset: 200},
  }

  // Add to the seed corpus
  for _, r := range testRequests {
    if data, err := json.Marshal(r); err == nil {
      f.Add(data)
    }
  }

  // ...
}

After that we can use httptest package to create a test HTTP server and make requests to it.

Note: Since our fuzzer can generate invalid non-JSON requests, it's better just to skip them and ignore with t.Skip(). We can also skip BadRequest errors.

func FuzzProcessRequest(f *testing.F) {
  // ...

  // Create a test server
  srv := httptest.NewServer(http.HandlerFunc(ProcessRequest))
  defer srv.Close()

  // Fuzz target with a single []byte argument
  f.Fuzz(func(t *testing.T, data []byte) {
    var req Request
    if err := json.Unmarshal(data, &req); err != nil {
      // Skip invalid JSON requests that may be generated during fuzz
      t.Skip("invalid json")
    }

    // Pass data to the server
    resp, err := http.DefaultClient.Post(srv.URL, "application/json", bytes.NewBuffer(data))
    if err != nil {
      t.Fatalf("unable to call server: %v, data: %s", err, string(data))
    }

    defer resp.Body.Close()

    // Skip BadRequest errors
    if resp.StatusCode == http.StatusBadRequest {
      t.Skip("invalid json")
    }

    // Check status code
    if resp.StatusCode != http.StatusOK {
      t.Fatalf("non-200 status code %d", resp.StatusCode)
    }
  })
}

Our fuzz target has a single argument with a type []byte that contains the full JSON request, however you can change it to have multiple arguments.

Everything is ready now to run our fuzz tests. When fuzzing http servers, you may need to adjust the amount of parallel workers, otherwise the load may overwhelm the test server. You can do that by setting -parallel=1 flag.

go test --fuzz=Fuzz -fuzztime=10s -parallel=1

And as expected we will see the following errors uncovered.

go test --fuzz=Fuzz -fuzztime=30s
--- FAIL: FuzzProcessRequest (0.02s)
    --- FAIL: FuzzProcessRequest (0.00s)
        runtime error: integer divide by zero
        runtime error: slice bounds out of range

We can also see the fuzz inputs in the testdata folder to see which JSON contributed to this failure. Here is a sample content of the file:

go test fuzz v1
[]byte("{"limit":0,"offset":0}")

To fix that issue we can introduce input validation and default settings:

if req.Limit <= 0 {
  req.Limit = 1
}

if req.Offset < 0 {
  req.Offset = 0
}

if req.Offset > len(all) {
  start = len(all) - 1
}

if end > len(all) {
  end = len(all)
}

With this change the fuzz tests will run for 10 seconds and exit without an error.

Conclusion

Writing fuzz tests for your HTTP services or any other methods is a great way to detect hard-to-find bugs. Fuzzers can detect hard-to-spot bugs that happen for only some weird unexpected input.

It's amazing to see that fuzzing is a part of built-in testing library, making it easy to combine with regular tests. Note: prior to Go 1.18 developers used go-fuzz, which is a great tool for fuzzing as well.

Subscribe now

Resources

• Source Code

• Fuzz Testing in Go

• Go Fuzzing