First posted on 2021-08-18 at https://www.honeycomb.io/blog/how-much-should-my-observability-stack-cost
What should one pay for observability? What should your observability stack cost? What should be in your observability stack?
How much observability is enough? How much is too much, or is there such a thing?
Is it better to pay for one product that claims (dubiously) to do everything, or twenty products that are each optimized to do a different part of the problem super well?
It’s almost enough to make a busy engineer say “Screw it, I’m spinning up Nagios”.
(Hey, I said almost.)
All of these service providers can give you sticker shock when you begin investigating them. The biggest reason is always that we aren’t used to considering the price of our own time. We act like it’s “free” to just take an hour and spin something up … we don’t count the cost of maintenance, context switching, and opportunity costs of not using the time to build something of business value. Which is both understandable and forgivable, as a starting point.
Considerably less forgivable is the vagueness–and sometimes outright misdirection and scare tactics–some vendors offer around pricing. It’s not ok for a business to optimize for revenue at the expense of user experience. As users, we have the right to demand transparency and accurate information. As vendors, we have the responsibility to provide it. Any pricing scheme that doesn’t align with best practices and users’ interests will be a drag on reputation and growth.
The core question, rarely addressed outright, is: how much should you pay? In this post I’ll talk about what your observability costs include, and in the next post, what you should consider including in your “observability stack”.
But I’ll give you the answer to your question right off the bat: you should probably spend 20-30% of infra costs on observability.
O11y spend should be 20-30% of infra spend
Rule of thumb: your observability spend should come to 20-30% of your infra spend. (I’ve seen 10% a few times from reasonable-seeming shops, but they have been edge cases and outliers. I have also seen 50% or more, but again, outliers.)
Full disclosure: this isn’t based on any particular science. It’s just based on my experience of 15+ years working in operations engineering, talking to other engineers and managers, and a couple of informal Twitter polls to satisfy my own curiosity.
Nevertheless, it’s a pretty solid rule. There are exceptions, but in general, if you’re spending less than 20%, you’re “saving money” at the expense of engineering time, or being silently dragged underwater by a million little time leaks and quality of service issues — which you could eliminate completely with a bit of investment.
Consider the person who told me proudly that his o11y spend was just 1-3%. (He meant the PagerDuty bill and Pingdom checks, actually.) He wasn’t counting the dedicated hardware for their ELK cluster (80k/month), or the 2-3 extra engineers they had to recruit, train and hire (250-300k/year apiece) to run the many open source tools they got for “free”.
And ultimately, it didn’t meet their needs very well. Few people knew how to use it, so they leaned on the “observability team” to craft custom views, write scripts and ETL one-offs, and serve as the institutional hive mind and software usability tutors. They could have used better tools, ones under active development by large product teams. They could have used that headcount to create core business value instead.
Engineers cost money
Engineers are expensive. Recruiting them is hard. The good ones are increasingly unwilling to waste time on unnecessary labor. This manager was “saving” maybe a million dollars a year (he mentioned a vendor quote of less than 100k/month)–but spending a couple million more than that in less-visible ways.
Worse, he was driving his engineering org into the ground by wasting so much of their time and energy on non-mission-critical work, inferior tooling, one-offs, frustrating maintenance work, etc, all of which had nothing to do with their core business value.
If you want to know if an org hires and retains good engineers, you could do worse than to ask the question: “What tools do you use, and why?”
- Good orgs use good tools. They know engineering cycles are their scarcest and most valuable resource, and they want to train maximum firepower on their core business problems.
- Mediocre orgs use mediocre tools, have no discipline or consistency around adoption and deprecation, and leak lost engineering cycles everywhere.
So back to our rule of thumb: observability amounting to 20-30% of total spend is where most shops should fall. This refers to cloud-native infrastructure, using third-party services to instrument and monitor code, with the basics covered — resource utilization graphs, end to end checks, paging, etc.
So, what do I need in my “observability stack”?
What are the basics? Well, obviously “it depends”. It depends on your requirements, your components, your commitments, your budget, sunk costs and skill sets, your teams, and most expensive of all — customer expectations and the cost of violating them. You should think carefully about these things and try to draw a straight line from the business case to the money you spend (or don’t spend). And don’t forget to factor in those invisible human costs.
dashboards, but I have heard a few people refer to them as “dynamic dashboards”.)
software engineers don’t have that kind of systems experience or intuition…and they shouldn’t have to.
graphs that were relevant to some long-ago situation, without context or history, without showing their work. Sometimes you’ll spot the exact scenario, and — huzzah! — the number you shout is correct! But when it comes to unknown scenarios, the odds are not in your favor.
dashboards. You cannot drill down from errors to endpoints to error strings; for that, you’d need a wide structured data blob per request. Those might in fact be two or three separate outages or anomalies happening at the same time, or just the tip of the iceberg of a much larger event, and your hasty assumptions might extend the outage for much longer than was necessary.
had — naming specific hosts, etc — which just creates clutter and toil. This is how your dashboards become that graveyard of past outages.
If what you have is “nothing”, even shitty dashboards are far better than no dashboards. But shitty dashboards have been the only game in town for far too long. We need more vendors to think about building for queryability, explorability, and the ability to follow a trail of breadcrumbs. Modern systems are going to demand more and more of this approach.
a job where your core responsibilities are writing and speaking. I don’t know. All I know is that if I’m the manager of that team, my responsibility and loyalty is to the well-functioning of that team, so we need to have a conversation and clarify what’s happening.
I cannot honestly understand why this is controversial. If you were hired as a database engineer, and you spent the year doing mostly DEI work, how does it make sense to promote you to the next level as a database engineer? That’s not setting you up for success at the next level at ALL. If this was agreed upon by you and your manager, I can see giving you glowing reviews for the period of time spent on DEI work, as long as your dbeng responsibilities were gracefully handed off to someone else (and not just dropped), but not promoting you for it.
negotiate a job that has a LOT of icing — but you should negotiate, no surprises. DEI work is absolutely valuable to companies, because having a diverse workforce is a competitive advantage and increasingly a hiring advantage. But that work isn’t typically budgeted under engineering, it comes from G&A
managers who were good at setting structure and boundaries, deeply encouraging, gave tons of chances and great constant feedback, tried every which way to make it work … and the employee just wasn’t interested. They volunteered for every social committee and followed every butterfly that fluttered by. They just weren’t into the work.
A surprising number of people thought I was writing this as advice specifically to and for women, and got mad about that. Nope, sorry. It was not a gendered thing at all. I’ve seen this happen pretty much equally with men and women. I had planned on writing three or four anecdotes, using multiple fake names and genders, but the first one turned out long so I stopped.
that was less easy to nitpick. But I didn’t want anyone to see themselves in it, so I didn’t want it to be too recognizable or realistic. I just wanted a placeholder for “Person who does lots of great stuff at work”, and that’s how you got Susan.
proposals… typical glue work. If you have a manager that thinks you are only doing your One Job when you are heads down writing code with your headphones on, that’s a Really Bad Manager.
their best for each other, with comparatively low levels of assholery, sociopaths, and other bad actors. If your situation is very unlike mine, I can understand why my advice could seem blinkered, naive, and even harmful. Please use your own judgment.
situations.
Susan also runs three employee resource groups, mentors other women in tech both internally and external to the company, and spends a lot of time recruiting candidates to come work here. She never turns down a request to speak at a meetup or conference, and frequently writes blog posts, too. She is extremely responsive on chat, and answers all the questions her coworkers have when they pop into the team slack. Susan has a high profile in the community and her peer reviews are always sprinkled lavishly with compliments and rave reviews from cross-functional coworkers across the company.
these are qualitatively different from the obligation you have to your core job. If you did only your core responsibilities and none of the extras, your job should not be in any danger. But if you don’t do the core parts, no matter how many extras you do, eventually your job probably will be in danger. Explain to your coworkers that you need to hit the pause button on electives; they’ll understand. They should respect your maturity and foresight.
engineer. Maybe you just aren’t into the work anymore (if you’ve been there a while), or maybe you don’t know how to get started (if you’re new). Maybe it’s even as simple as mentioning it to your manager and reshuffling your responsibilities a bit. But don’t assume the problem will solve itself.
CI/CD pipeline, culminating with an automatic deploy to production. This should happen once per mergeset, never batching multiple engineers’ diffs in a run, and it should be over and done with in 15 minutes or less with no human intervention.
more and more time addressing those pathologies instead of making forward progress for the business. Longer diffs. Manual deploy processes. Bunching up multiple engineers’ diffs in a single deploy, then spending the rest of the day trying to figure out which one was at fault for the error.
Some people eventually make peace with this, but many never do. No shame in that.
