Write the code first. Let it tell you what the book says.

Back in December 2024 I asked here whether anyone would be interested in a book about Event Sourcing and CQRS. I had a six page table of contents and a lot of doubt about whether I wanted to commit the time.

Well, I committed the time. Two years of it.

And I did one thing differently from what I planned, which turned out to matter more than anything else: I built the implementation first and let the code tell me what the book should say.

Sounds like a small decision, right? It is not. It rewrote chapters.

Let me show you one.

Snapshotting looks simple until you build it

Everybody describes snapshotting the same way. Your event stream gets long, replaying it on every load gets slow, so you periodically save the aggregate’s state and on the next load you restore that and replay only what came after.

That description is correct. It is also useless if you are the one writing the code.

I found four things while implementing it that no diagram tells you. Three of them ended up in the chapter.

The trigger is boundary math, not a modulus

You want to snapshot every fifty events. So you check whether the version is a multiple of fifty, right? version % 50 == 0. Done.

No. And here is the part that should worry you: it fails silently.

One command appends three events. Your stream goes from version 49 to version 52. It steps right over 50 and never lands on it. The check never fires, the snapshot never happens, and nobody tells you anything, because a missing snapshot looks exactly like an aggregate that has not hit the threshold yet. The load still works. It is just slow. You find out six months later when somebody asks why one aggregate takes 400 ms to load.

What you want is to ask whether the append moved the stream into a new bucket:

postVersion / interval > preVersion / interval

Integer division, evaluated across the append instead of at a single point. A multi-event append that jumps a boundary captures once, at the post-append version.

Simple fix. But I did not see it until I wrote the test that appends three events at once.

Do not upcast your snapshots

Events get upcast. Event shapes change, you write an upcaster to lift the old shape to the new one, and you maintain that chain forever, because events are immutable and you cannot go back and rewrite history.

So when the snapshot shape changes, you upcast the snapshot too. Right?

Wrong, and I am glad I found this in code rather than in print. A snapshot is a cache. The aggregate can always rebuild it from events. So when the shape changes, let the old snapshot read as a miss, rebuild from history, and capture a fresh one at the current shape the next time you cross a boundary.

Upcasting snapshots buys you nothing that a discard does not already give you, and it costs you a second chain of upcasters to keep correct next to your event upcasters. Two lineages that have to agree with each other, forever, where zero would do.

Put the schema version in the WHERE clause, by the way. Do not read the row and compare in memory. A snapshot at a shape you cannot read should never cross the wire in the first place.

Guard the restore or corrupt your writes

This one bit harder.

RestoreFrom(snapshot, version) seats state onto your aggregate and sets its version. That version is the concurrency token your next append checks against.

Now let somebody call that on an aggregate that has already applied events, or is holding uncommitted work. You just overwrote the token. The next append goes out with an expected version describing a state the aggregate is not in, and a stale write lands as if it were current.

In an event-sourced system! Where the whole promise is that the log tells you the truth!

So the restore throws unless the aggregate is pristine. Loud failure right at the seam, instead of silent corruption three layers downstream that you find out about in production.

Your speedup test cannot be a stopwatch

Obvious test: load with a snapshot, load without, assert the first is faster.

I have the scar on this one. The same code passed five out of five on a 32-core machine and starved on a shared 4-core CI runner. The scheduler widened the window and the timing budget stopped meaning anything at all.

Snapshotting does not save you milliseconds. It saves you replays. So test replays. Seed a stream past two boundaries, load through the snapshotting repository over a store that records where the read started, and assert two things: the read began at the snapshot’s version, and it replayed strictly fewer events than the full stream.

Machine independent, and it measures what the pattern is actually for.

This happened across the whole book

The snapshotting chapter is not the chapter I outlined. Neither is the one on correlation tracing. Neither is the one on event versioning, and that one moved the most.

Every single time the implementation and the manuscript disagreed, the implementation won and the chapter got rewritten. Not once did I look at working code and decide my prose had been right all along.

And I think this is the thing nobody tells you about writing on architecture. Any pattern you can draw on a whiteboard has a layer underneath it where the real decisions live. Interval math. Concurrency tokens. What your test can honestly assert on a CI runner you do not control. You cannot write about that layer if you have not been down there, and readers can tell.

So here is my advice to anybody thinking about writing a technical book: build it first. All of it. Ship the code, run it, break it, and let it tell you what your chapters should say.

The manuscript is at 449 pages across 18 chapters now, with a production-grade reference implementation in .NET running on PostgreSQL, SQL Server, KurrentDB and DynamoDB behind one contract.

What would you want to see from a book like this? I am still listening.

Serverless Microservices Online Courses

Update 04/03/2019: I have completed the FREE course: “Why you need serverless microservices, yesterday!“. Enroll for FREE!
At the moment, I’m working on four online courses in the following order:

 

Why You Need Serverless Microservices, Yesterday, FREE Course

WhyYouNeedServerlessMicroservices_960x520

In this course I will walk you through the many benefits of creating serverless microservices instead of the traditional node / instance approach including the use of containers. There are more than enough things to worry about when you want to create a new cloud system or transform a legacy system to operate in the cloud.

From a business point of view, there are huge benefits in going serverless rather than instance based (including containers). A very large jump in business agility can be achieved through focusing on the problems and opportunities rather than the technical jungle of traditional computing solutions.

From a technical point of view, it is almost nirvana where you can eliminate many points of failures in the architecture.

 

“How To Build An Event Store”, Paid Course

eventstore_960_520_darker

Your event store is the heart of an event sourced system. The event store is the source of truth for all business events. It needs to be able capture and replay all domain events in your system reliably and with great performance.

In this course, I will walk you through building an event store that you can re-use in your own projects. In addition, I will walk you through building a read model that allows you to query domain events from the read model.

I will also go through why building your own event store has many more advantages over using a third-party event store.

We will be building the event store in AWS DynamoDB but you can apply the design to a traditional RDMS SQL storage just as well. I will go over the pros and cons in doing so.

 

“Architecting And Designing Event Based Microservices”, Paid Course

ArchitectAndDesign

Learn the details on how to design and architect event based microservices using Domain Driven Design (DDD), Event Storming, CQRS, and EventSourcing techniques. I will show you how best combine many principles, patterns, and techniques to create an architecture with as few points of failures as possible and still deliver a great solution.

What you will learn can be applied to cloud based systems but also to traditional, on-premise systems. The benefits are great in either environments.

Whether you are designing a new system in a greenfield environment or transforming a legacy system, I will show tips & tricks that you can use depending which type of project you are in.

 

“Implementing A Serverless Microservice in AWS”, Paid Course

Code_960x520

Learn the details on how to implement a serverless microservice in AWS using Domain Driven Design (DDD), CQRS, and EventSourcing techniques.

We will build a fully functioning billing system in AWS using Visual Studio and C# .NET Core. Even if you are not familiar with C# and .NET Core, you will learn a lot of practical tips on using the different AWS services including building fully automated CI/CD pipelines.

 

I’m still working on these courses but I’m planning on publishing “Why You Need Serverless Microservices, Yesterday” and “How To Build An Event Store” first.

My New YouTube Channel

I just created my new YouTube channel “Creating Great Software”. Have a look.

I will be publishing videos about creating great software including serverless computing in AWS. In addition, I will be publishing my STRONG opinions about the state of the software industry from time to time.

I’m super excited about this new channel and I’m looking forward in seeing your feedback. See you there!

Event Store in AWS DynamoDB

Update 05/27/2019: How to build an event store masterclass” is now available. Learn how to build an event store using C#. NET Core, DynamoDB, MySQL for read models, and more.

In the past few weeks, I have been working on creating an event store in AWS DynamoDB and AWS S3. I use an event store for the domain driven design (DDD) concept in that system. Specifically, one of my systems uses CQRS and Event Sourcing (which is awesome, btw).

The idea for the event store started when I wanted to create my own event store for several reasons. I played with EventStore from Greg Young available at https://eventstore.org/ I encountered errors when I tested EventStore and I expected 100% functionality without issues. Besides, I did not want to babysit another persistence mechanism. I just do not have time for that.

I have created an event store before that was based entirely on Redis. That had worked great and it was super-fast. I used https://redislabs.com/ service to allow zero-maintenance of a fully clustered Redis solution. This has been working for some time now. The only problem was that it can have inconsistencies when it comes to running many applications nodes that use the Redis event store. Long story short, this has to do with complex timing, concurrency violations, etc.

So, I thought there must be a better way. I want zero administration headaches ideally, but I want to take advantage of the consistency capability of the underlying storage mechanism. I want the event store to be a service to the application itself that runs within the same process of the application. So, no need to babysit a separate event store cluster. I want the event store conceptually living side by side within the application.

If I can only take advantage of the consistency of the persistent mechanism, the application can then handle error conditions etc. accordingly based on what the use cases are.

After some experimenting and doing quiet a few load tests using http://loader.io, I can now say that the Event Store in DynamoDB has been born. My first 10,000 user / min load test has passed with flying colors and reached API response times of less than 10 ms with sustained rates of 170 clients / second. All this was running on a single cheap t2.medium instance. DynamoDB had to increase the throttling automatically to take this kind of load. This is one of those cool features, btw.

The event store I created in DynamoDB uses two concepts, Aggregates and Change Sets. Aggregates are the aggregates in your DDD system but only a handful of meta data. Change sets are one ore more domain events that are created when your DDD system processes a command.

I know I can get much higher numbers even by adding additional nodes and doing further tweaking. I would have to do much more load testing of course and tweaking but so far this has been a tremendous success and I’m super excited to take advantage of DynamoDB.

Anyways, I wanted to share this information. Maybe in the in future I can go into much more details. Time is the only problem I have really.

Creating IDs with CQRS and Event Sourcing in Java and .NET

I’m currently working on a cloud-based system that uses microservices with domain driven design and CQRS and Event Sourcing in Java. I love C# and .NET but I decided to do this in Java for several reasons. This could have been done in C# just as well. In fact, this system is a mix of some C# microservices and Java microservices. The important thing is that we are using the same concepts that are applicable when doing DDD and CQRS and Event Sourcing.

Anyways, I wanted to put down my thoughts on what I have been thinking about since last week and I hope this might be useful to anyone who came across the same question. So, my question since last week was this:

“Who creates an entity id in DDD when doing CQRS and Event Sourcing?”

Well, the way I have been doing this was as follows:

  1. When an application service executes a use case via one of its methods, it can create a new entity via a new immutable value object. All my entities’ ids are immutable value objects.
  2. Sometimes, a factory in the domain model creates new entities. Again, the entities’ ids are created via immutable value objects.
  3. Sometimes, a domain service creates new entities with immutable value objects.

This is all fine and good. I started to look at Greg Young‘s sample C# project at his Github here. For this who do not know him, he is a fantastic speaker and mentor for CQRS and Event Sourcing. Check out his videos, papers, and blog posts. Excellent material. He has a free 6 hour video here. And, while you are at it, check out his video about: “7 Reasons Why DDD Projects Fail“.

What poked my interest was this particular line in his CreateInventoryItem command code:

public class CreateInventoryItem : Command {
    public readonly Guid InventoryItemId;
    public readonly string Name;

    public CreateInventoryItem(Guid inventoryItemId, string name)
    {
        InventoryItemId = inventoryItemId;
        Name = name;
    }
}

If you look closely on line 5, you will see that the entity id inventoryItemId is being passed as part of the CreateInventoryItem command. This is a small but important detail.

I had an interesting thread discussion with Greg Young about this here. Greg raised a few good questions:

  1. What about when you want to send 3 commands?
  2. How will you start returning ids from commands?
  3. What if a command creates 3 entities?
  4. How will the domain get those ids back to the client?
  5. My UI is creating an account then the next step is managing some details. How does the UI go from one step to the next without knowing the account id?

As you can see, there are several good reasons why a client would want to create the ids instead of the domain model. Another reason I would want to add to this is that if the domain would create the ids, it would violate the goal of keeping track of all state changes in domain events with the commands as the initiators. In my opinion:

In a CQRS based system with Event Sourcing, commands carry all attributes that are required to execute the command including the creation of identities.

So, a client can create a command with ids that are based on Guids/Uuids and then send these creation commands with those ids. This allows you to do things such as persisting the commands and being able to replay those commands as well with 100% accuracy because the domain model with arrive every time to a deterministic state when you replay the domain events.

I like this very much and it provides a lot of freedom with inductive UIs / task based UIs.