Golang optimizations for high‑volume services
Lessons from a Postgres → Elasticsearch pipeline
Intro
Building services that sit on top of a Postgres replication slot and continuously stream data into Elasticsearch is a great way to get low‑latency search without hammering your primary database with ad‑hoc queries. But as soon as traffic ramps up, these services become a stress test for Go’s memory allocator, garbage collector, and JSON stack.
This post walks through optimizations applied to a real-world service that:
Connects to a Postgres replication slot
Transforms and enriches the change events
Uses Elasticsearch’s bulk indexer to index and delete documents
The constraints: the service cannot stop reading from the replication slot for long (or Postgres disk will grow), and it cannot buffer unbounded data in memory (or Go’s heap will). The goal is to keep latency and memory stable under sustained high volume.
