Data Modeling for Event Sourcing: Handling Projections and Schema Evolution in Distributed Systems

Event sourcing has emerged as a powerful architectural pattern in modern distributed systems, offering an immutable audit trail and enabling complex temporal queries. However, implementing it effectively introduces significant challenges, particularly when it comes to managing projections and handling schema evolution. Unlike traditional CRUD applications where data is mutated in place, event sourcing requires you to reconstruct state from a stream of events, making the separation of commands, events, and projections critical.

The Core Challenge: Immutable Events vs. Mutable Projections

In an event-sourced system, the "source of truth" is the sequence of events. These events must remain immutable to preserve the integrity of the system's history. However, the projected views—the read models that your frontend or API consumes—must evolve as business requirements change. This dichotomy creates a unique engineering problem: how do you update your read models without rewriting history or causing data inconsistency during updates?

The solution lies in treating projections as first-class citizens in your data model. They are not mere database tables but distinct components that must be versioned, migrated, and potentially rebuilt.

Handling Schema Evolution in Events

One of the most persistent issues in event sourcing is schema drift. If your application evolves to include new fields in an event, how do you ensure backward compatibility with older services that haven't been updated yet?

The golden rule is: Never change the name or type of an existing event property. Always add new properties. If you must change a property, create a new event type. To facilitate this, your event schema should include a version number.

Consider a simple order creation event. In its initial state, it might look like this:

{
  "eventId": "123e4567-e89b-12d3-a456-426614174000",
  "aggregateId": "order-001",
  "timestamp": "2023-10-27T10:00:00Z",
  "type": "OrderCreated",
  "version": 1,
  "customerId": "cust-123",
  "totalAmount": 150.00,
  "currency": "USD"
}

Later, if you need to track the shipping address at the time of order creation, you should not modify the existing event. Instead, you might introduce a new event type or add a nullable field while maintaining support for the old structure in your deserialization logic. Using a library like Newtonsoft.Json or System.Text.Json with ignoreDataMember attributes can help manage these variations gracefully.

Designing Robust Projections

Projections are the mechanism by which we transform immutable events into queryable read models. They should be idempotent and isolated from the domain logic that produces events.

A well-designed projection handler should:

1. Subscribe to specific event types.
2. Transform the event data into a format suitable for the read database (often a relational or document store).
3. Handle missing or out-of-order events gracefully.

Here is a conceptual example of a projection handler in C#:

public class OrderTotalProjection : IHandleEvents<OrderCreated>, IHandleEvents<OrderCancelled>
{
    private readonly IDocumentSession _session;

    public OrderTotalProjection(IDocumentSession session)
    {
        _session = session;
    }

    public void Handle(OrderCreated @event)
    {
        var projection = new OrderProjection
        {
            OrderId = @event.AggregateId,
            Status = "Open",
            Total = @event.TotalAmount
        };
        
        // Upsert logic to ensure idempotency
        _session.Store(projection); 
    }

    public void Handle(OrderCancelled @event)
    {
        var projection = await _session.LoadAsync<OrderProjection>(@event.AggregateId);
        if (projection != null)
        {
            projection.Status = "Cancelled";
            projection.Total = 0;
            _session.Store(projection);
        }
    }
}

Migrating Projections in Production

When the schema of your read model changes—for example, adding a shippingAddress field to the OrderProjection—you cannot simply run a database migration. You must also rebuild the projection data from the event stream. This is typically done by:

1. Deploying the new projection code alongside the old one.
2. Running a background worker that replays all events for the affected aggregates.
3. Updating the read database with the new schema.
4. Switching the API to query the new view once the backfill is complete.

Conclusion

Data modeling for event sourcing requires a shift in mindset. You are no longer designing tables for storage; you are designing streams for history and views for access. By strictly separating event evolution from projection updates and ensuring projections are idempotent and versioned, you can build distributed systems that are not only resilient to change but also maintain a clear, auditable history of all business activities. Embrace the complexity, and your system will reward you with unparalleled flexibility and traceability.