Have you heard? Clickhouse is winning the observability wars!
Years ago I predicted that columnar storage would remake observability. What I didn't see coming: vendors would build it, nerf it, and sell a worse version back to you as "Datadog, but cheaper"
We interrupt our regularly scheduled series of posts on AI norms and values to bring you this incredible piece from Mat Duggan, “Why Clickhouse Is Winning the Observability Wars,” which you should go read right now. Go! I’ll wait.
Mat starts by describing the experience every developer starts with, logging on the command line — “the observability equivalent of a first kiss, that ruins you for everything after” — followed by the descent into hell as four services becomes forty, becomes four hundred, and the impossibility of satisfying stakeholders in engineering, data, customer support, and executives, who all want different impossible things before breakfast.
As Mat says, at 1TB a day, every modern observability stack is fine. Pick something and be productive. But at 10TB/day, they all become unmanageable — all, he says, except Clickhouse.
(My favorite part of the rant, because I am a bad person and/or an SRE, is where he meticulously details exactly how Elastic, LGTM, and Datadog fall apart at scale. This is a boy with scar tissue, respect.)
Few observability tools can handle 10TB/day
It’s hardly news that data tools tend to start off easy to run, but require teams to run at scale. Clickhouse, though — let’s hear Mat gush:
“Every other observability backend I’ve worked with mutates as it grows... ClickHouse at 10 TB a day looks like ClickHouse at 1 TB a day with more shards. That’s it. That’s the pitch. That’s the whole reason I’m writing this.”
Mat admits that Clickhouse has a small tax up front, but worth it since a tiny bit of upfront effort buys you SO MUCH EASE down the road, as you scale indefinitely, with high cardinality data and no messy schema lock-ins or performance cliffs.
Wow, this is starting to sound like a Clickhouse ad, isn’t it? Why am I boosting Clickhouse's reputation like this?
Because it isn't just fucking Clickhouse, you numbnuts.
Observability backed by columnar storage is a different class of tool than the three pillars
Mat’s piece isn’t actually about who is “winning” the observability wars, it’s about how Mat just discovered what it's like using observability powered by a columnar storage engine, and it is blowing his everloving mind.
And it should. It is genuinely better, and this is genuinely exciting!
And we have been talking about this for the past ten years.1 How is this still news to people?
We wrote about this at length in the first edition of “Observability Engineering”, devoting over a tenth of the book to this topic. When writing the second edition, we invited Clickhouse to contribute a guest chapter for the same reason — THIS IS WHAT OBSERVABILITY SHOULD FEEL LIKE IN TWENTY FUCKING TWENTY-SIX.
I don’t mean to be all “I told you so”, but look! what a difference! this shit makes! if it can even make a grizzled logs warrior like Mat fall so madly in love. 💕
Why haven’t you heard about this?
I have a theory why it is that people still don’t get this, even people like Mat who are clearly steeped in the space.
My theory is that anything one vendor says gets written off as self-interested marketing. It isn’t until multiple vendors link arms and say the same thing together that people sit up and note that the landscape really has changed.
I always assumed that would happen naturally once more observability companies were built on columnar storage. We would align on some shared technical vocabulary due to our keen shared interest in clarifying the landscape for buyers, to help them understand how much better it is. The upstarts vs the incumbents. The shiny new way vs the kludgey, slow, painful old way.
Well, I was half right. Every observability company founded post-2019 has been built on top of a columnar store, did you know that? But not only have they not linked arms with us, they keep insisting there’s nothing all that new or different about what they’re building. They’re all selling “Datadog, but cheaper.”
Why are newer observability vendors nerfing their own products and obscuring the difference?
As a consequence, most of these newer observability tools are, despite being built on columnar storage engines, not that different from Datadog.
Same old three pillars. Same old problems with cardinality and scale, give or take. They are doing this because the weakness that they sense in Datadog is all price.
The weakness I sense in Datadog is all product.
Yes, it’s cheaper to run on columnar storage. But lower price is a consequence of better architecture, and honestly, it’s one of the least interesting consequences. Infra logs and metrics are commodities, but observability for your own products, your own code? That should be an investment.
Everybody thinks Datadog is expensive because they choose to be expensive. This is true in part. But they also don’t really have a choice. Their business model has grown up around a thirty-year-old architecture, and they are fully locked in to it.
Will AI be enough to finally shake off the death grip of the three pillars?
As AI starts accounting for a larger share of workloads, people are beginning to realize that the trace is all that matters. The trace — or wide, structured canonical logs, one or the other — is all you need.
One powerful, context rich data set is worth more than the sum of its pillars, because relationships are what makes data valuable. Slice it up into pillars, and you destroy that value for good.
One way or another, people are gonna figure this out, and that poses an enormous threat to Datadog’s business model, which relies on storing (and charging for) the same data over and over and over and over again in different formats, then storing (and charging for) links between datasets, and all the attendant fees and costs.
Or maybe people won’t figure it out. God knows I never thought the swindle could go on this long.
Is Clickhouse winning the observability wars? God I hope so
Clickhouse has used the term “observability 2.0” two times by my count, making them the only other vendor who has. Why is Clickhouse willing to link arms, kinda, but no one else has?
My theory is because they are a database company, not an observability company, and therefore less attached to marketing themselves as a cheaper Datadog. Unlike most observability vendors, they have decided to prioritize building something that solves customer problems over building something easy to market and sell.
Clickhouse, if you’re reading this, I would love to coordinate more. Call me.
So yeah. If you want to run your own logs clusters, you should use Clickhouse. If you don’t want to DBA your own clusters or deal with the overhead (which scales linearly, as Mat points out, but is nontrivial), you should use Honeycomb.
If you’re still limping along on tools designed twenty years ago, maybe you too should give Clickhouse -- or Honeycomb -- a shot.
It really is a whole new world. Just ask Mat Duggan.
~charity
Further references (far from comprehensive):
A post on the cost crisis in observability, which tackles the three pillars architecture vs the columnar-backed o11y 2.0 model.
We started calling columnar storage-backed o11y “Observability 2.0“ in 2023.
One long twitter rant, and another, and another. (A LinkedIn rant for variety’s sake.)
“Why Observability Requires A Distributed Column Store”, by Alex Vondrak, a personal favorite
Here’s one from 2018 on the technical differentiators in observability powered by columnar stores
Another post from 2020 on using columnar stores instead of the three pillars
“There is only one key difference between observability 1.0 and 2.0”, and it is unified columnar storage
I vented about vendors “o11ywashing” their old three pillars tools in 2025
Sam Stokes talking about our columnar storage engine at Strangeloop 2017
Jessica Kerr talking about how we serverless’d our columnar database at Strangeloop 2023