Bridging Backend and Data Engineering: Communicating Through Events
Building a Unified System: Event-Driven Approach to Backend and Data Engineering Communication.
Introduction
In the rapidly evolving landscape of software development, the need for seamless communication between backend services and data engineering pipelines has become paramount.
Traditional methods, such as REST APIs for backend services and batch processing for data pipelines, often fall short in providing the real-time, scalable, and flexible solutions required by modern applications.
This is where event-driven architecture (EDA) comes into play, offering a robust framework for integrating these disparate systems through asynchronous event communication.
However, if you did not build those systems with an event-driven architecture in mind from day one, what can you do?
For small teams, it can also be daunting and time-consuming to think about, implement, and maintain such systems.
A hybrid solution
One easy way to reach some kind of event-driven platform without going all the way in is simply to set up a Pub/Sub system (e.g., GCP Pub/Sub, Amazon SQS, Redis Pub/Sub, RabbitMQ, etc.).
This involves setting up producers that send messages through a topic and consumers that read them via a subscription.
From there, the hard part is defining a standard way of expressing events that come through your system, whether they are ETL pipeline events or backend service events.
You could define one topic to start with, let’s call it “platform_events”, where any service would broadcast its events in a standardized format.
Let’s take an e-commerce example with buyers, orders, products, and merchants that is deployed on GCP and uses GCP Pub/Sub.
You gather your backend and data engineering teams, and all of them agree on adding optional attributes (merchant_id, buyer_id, event_name), and a JSON-marshalled payload.
From there, any service can subscribe to that unique topic to receive a copy of those events. Each service can have its own router in place that quickly reads the event name to decide whether to skip or consume it.
Notice that subscriptions decide which events to process and which simply to ignore.
