Redis streams: a different take on event-driven

A different take on event driven.

Redis Streams are a powerful data structure in Redis designed for handling real-time, high-throughput event data. They combine the simplicity of an append-only log with advanced features for event consumption and processing, making them ideal for use cases like event sourcing, sensor data collection, and notification systems.

What are Redis Streams ?

A Redis stream acts as an append-only log where each new entry is assigned a unique, time-based ID. Unlike traditional logs, Redis Streams support random access in O(1) time and enable complex consumption patterns, such as consumer groups for distributed processing. This makes them suitable for scenarios where you need to record, syndicate, and process events in real time.

Key characteristics:

• Append-only: Once data is written, it cannot be modified.

• Immutable entries: Each entry is a set of key-value pairs with a unique ID.

• Real-time syndication: Multiple consumers can process events as they arrive.

• Trimming and retention: Streams can be trimmed to prevent unbounded growth.

---

Cut Code Review Time & Bugs in Half (Sponsor)

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo's!

---

How to use

Single client

For basic usage, you interact with Redis Streams using a set of core commands:

• XADD: Add a new entry to a stream. Each entry consists of field-value pairs, and Redis generates a unique ID if you use the * wildcard.

• XRANGE: Retrieve a range of entries by specifying start and end IDs, making it easy to access historical data or iterate over the stream.

• XLEN: Get the number of entries in a stream—useful for monitoring or batch processing.

Example workflow:

XADD mystream * field1 value1 field2 value2
XLEN mystream
XRANGE mystream - +

The - and + special IDs mean respectively the minimum ID possible and the maximum ID possible inside a stream, so the XRANGE command will just return every entry in the stream.

Consumer groups

For scalable processing, Redis Streams support consumer groups. This allows multiple consumers to cooperatively process messages from the same stream, with Redis distributing entries among them.

How it works:

• Consumer group: A named group coordinating multiple consumers.

• Consumers: Each consumer reads a subset of entries, ensuring parallelism.

• Acknowledgement: Consumers must acknowledge (XACK) processed entries. Acknowledgements are tied to a consumer group and its consumers.

Example workflow:

XGROUP CREATE mystream mygroup $
XREADGROUP GROUP mygroup consumer1 STREAMS mystream >
XACK mystream mygroup <entry-id>

Handling failures and message replays

One of the strengths of Redis Streams is the ability to replay messages in case of incidents or failures. Since messages are not deleted automatically after acknowledgment, you can reprocess them if needed.

Auto-claiming and message management:

• XAUTOCLAIM: Automatically transfer ownership of pending messages that have been idle for too long to another consumer, ensuring that no message is left unprocessed if a consumer crashes.

• Manual deletion: Messages are not deleted upon acknowledgment; you must use XDEL to remove them. This gives you control but also requires you to manage retention, typically after all consumer groups have processed the message.

Trade-offs:

• Manual cleanup: You need to implement logic to delete messages once they're no longer needed, as Redis does not do this automatically even if all consumer groups have acknowledged a message.

• Replay capability: The upside is that you can replay messages without recreating payloads, simply by reading from the stream history as needed.

Practical considerations

• Memory management: Use XTRIM or the MAXLEN option with XADD to cap stream size and prevent unbounded memory growth.

• Visibility: Inspect pending messages with XPENDING to monitor which messages are yet to be processed or acknowledged by consumers.

Redis Streams provide a flexible, high-performance foundation for event-driven architectures. By leveraging consumer groups, auto-claiming, and careful retention management, you can build scalable, resilient systems that can recover from failures and replay events as needed, all with the speed and simplicity Redis is known for.

Also, it’s nice to see Redis to be open-source (AGPL) again.

Cut Code Review Time & Bugs in Half (Sponsor)

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo's!