Architecture · 9 min read
Event-Driven Architecture: The Tradeoffs Nobody Tells You About
Event-driven systems unlock real engineering capabilities — but they come with a specific set of complexity costs that need to be understood before adoption.
The Appeal — and the Reality
Event-driven architecture has genuine engineering advantages: services are decoupled by design, producers do not need to know about consumers, workloads can be scaled independently, and the system naturally builds an audit trail of everything that has happened.
These are real. The pattern is not hype. But the tradeoffs are also real, and they are often understated in the enthusiasm for the benefits.
This is a practical accounting of what adopting event-driven architecture actually costs — and when those costs are worth paying.
Eventual Consistency Is Not a Property You Configure — It Is One You Design For
In a request-response system, when a write returns 200, the data is there. In an event-driven system, a write to the event stream means the event has been accepted for processing — not that it has been processed.
The consumer might not have processed the event yet. It might be behind. It might have failed and be retrying. The downstream state might not reflect the event for seconds, or minutes, or in failure scenarios, longer.
This changes how you build everything downstream of the event stream. Any read that must reflect a write the user just made cannot simply query the database — you need either read-your-writes semantics (tracking which events have been consumed per user) or a different data model for write-path queries.
It also changes user experience design. If a user creates a record and immediately navigates to a list view, they may not see it. This is not a bug — it is a property of the architecture. But it must be accounted for, either in the data model or in the UX.
Exactly-Once Is Harder Than It Sounds
Kafka provides at-least-once delivery semantics. If a consumer processes an event and then crashes before committing its offset, it will process the event again on restart.
Exactly-once processing — ensuring each event produces its effect exactly once regardless of failure — requires either:
- Idempotent consumers: the effect of processing an event twice is the same as processing it once. Implement this by tracking event IDs and skipping already-processed events.
- Transactional outbox: for events that must trigger a database write, write to an outbox table in the same database transaction as the business write, then have a separate process publish from the outbox to Kafka. This couples event publication to the database commit.
- Kafka transactions: Kafka supports transactions across producers and consumers within the Kafka ecosystem, but not across external databases.
None of these are simple. Each adds operational complexity. The choice depends on the failure tolerance of the specific event type.
Schema Evolution Is a Breaking Change Problem
In a request-response system, changing an API payload is a negotiation between one producer and one consumer — or at most a few. In an event-driven system, a schema change to an event affects every consumer of that event — some of which you may not control.
Schema registries (Confluent Schema Registry, AWS Glue Schema Registry) enforce compatibility rules: backward-compatible, forward-compatible, or fully compatible. They prevent schema changes that break existing consumers from being published.
But schema governance is more than tooling. It is a process: event schemas are contracts. Breaking changes require consumer coordination. New required fields cannot be added without a migration path for consumers that were not built to handle them.
Without this discipline, event-driven architectures accumulate schema debt that eventually produces production incidents when a consumer that was not updated encounters an event shape it was not designed for.
Observability in Event-Driven Systems Is a First-Class Concern
In a synchronous system, a failing request produces an error response. The caller knows something went wrong, immediately, and the stack trace is available in the same request context.
In an event-driven system, failure is asynchronous. An event is published. The consumer fails to process it. The failure is logged in the consumer's process. The user who triggered the original action that produced the event has no idea.
Distributed tracing becomes non-optional. Event headers must carry trace context. The trace must propagate from the HTTP request that published the event through every consumer that processes it. Without this, debugging a failure that spans a producer and three consumers requires correlating logs from multiple services by timestamp — which is slow, error-prone, and miserable during incidents.
Dead-letter queues are the minimum for fault tolerance: events that cannot be processed after a configured number of retries are moved to a DLQ, where they can be inspected, corrected, and reprocessed. Monitor DLQ depth as a critical operational metric. A growing DLQ is a production incident in progress.
When the Tradeoffs Are Worth It
Event-driven architecture is the right choice when:
Services need to be independently deployable and scalable at different rates. An order processing service that handles 50,000 events/hour and a notification service that handles the same volume but is much less compute-intensive should scale independently. Event-driven decoupling makes this natural.
The business domain is naturally event-shaped. Payment processed, order placed, inventory updated — these are events in the business sense. Modelling them as events in the technical architecture aligns technical structure with business reality.
Audit trails are a first-class requirement. Event streams are immutable logs. Every state change is recorded as an event with a timestamp and a producer identity. This is an inherent capability of the architecture, not an add-on.
Consumer workloads have variable latency tolerance. Some consumers need to process events immediately (fraud scoring, real-time pricing). Others can batch process (reporting, analytics, email notifications). Event-driven architecture accommodates both in the same system without coupling their operational characteristics.
When these characteristics match your system, the tradeoffs are worth accepting and designing around. When they do not — when your system is fundamentally request-response shaped, when consistency requirements are strict, when team capacity to manage operational complexity is limited — introducing Kafka does not simplify the system. It adds infrastructure complexity that provides no architectural benefit for your specific use case.
Event-driven architecture is a powerful pattern. It is also one that rewards honest analysis of whether the benefits justify the costs for the specific system being built. The worst outcome is adopting it because it is popular, rather than because the problem it solves matches the problem you have.