An event-driven platform for a client
I designed a client's event-driven backend in Rust: consumers on Valkey Streams, payloads on S3, end-to-end tracing, and an operator console for the people who run it.
Client work under NDA: the client, its data and internal details are left out on purpose.
A client runs its workloads through an event-driven backend. Work arrives as events, payloads live on S3, and every event has to be processed exactly once in effect — under load, with retries — while the people operating the platform need to see what happened when something goes wrong.
- 01Async Rust consumers on Valkey Streams that read, handle and acknowledge events, with payloads on S3.
- 02Tracing and logging through OpenTelemetry into Grafana, Loki and Tempo, so a single event can be followed across services.
- 03A new operator console — Spring Boot APIs and a React UI — for the people who run the platform day to day.
- 04Everything shipped as small, reviewable diffs under senior code review.
- 01
Acknowledge after the side effect, not after the read
I came from a request/response world, where a call is answered and forgotten. On an event bus I treated “delivered” and “processed” as the same thing. They are not: the gap between them is where retries, duplicates and ordering bugs live.
Engineering choiceMake each consumer an explicit small state machine — read, handle, acknowledge — and acknowledge only once the work has actually happened. Retries stopped being edge cases and became part of the design. The cost is one I accept on purpose: if a consumer stops after the work but before the acknowledgement, the event comes back. Delivery is at least once, so every handler has to be safe to run twice.// delivered is not processed let message = stream.read().await?; if worker.handle(&message).await? { stream.ack(message.id).await?; } - 02
Observability before trust
An event bus fails quietly: a message that was never processed can look exactly like one that was.
Engineering choiceTraces and logs went in through OpenTelemetry, into Grafana, Loki and Tempo, before more was built on top of the bus — so every event can be followed end to end, and problems are found from evidence rather than guesses. - 03
Choose with the client, not after the fact
This is someone else's production system. A design that looks clean to me and wrong to the people who run it is not a good design, and I don't see the cost of a choice as clearly as they do.
Engineering choiceI put the options on the table with them: what to build now, what can wait, and what fails if we guess. We pick one, then I write it. The technical decisions above came out of those conversations, not from a ticket thrown over a wall.
Delivery is not processing. Once the transitions are explicit and observable, an event-driven system becomes something you can reason about, and explain to the people who depend on it. The same is true of the choices: they are easier to defend when the client was in the room when we made them.
- Rust
- Tokio
- Valkey Streams
- S3
- OpenTelemetry
- Grafana
- Loki
- Tempo
- Spring Boot
- React