“Observability Engineering”: a book so nice, we wrote it twice (xpost)

“In which Martin Fowler gives me exactly the right advice at exactly the wrong time, we assemble a rogues gallery of technical contributors, and the second edition is off to print.”

Last week I got to meet Martin Fowler for the first time in person. This was an exciting moment for me. Martin ranks high on my personal pantheon; he is, so far as I can tell, hardly ever wrong.

Martin Fowler, me, and Nathen Harvey, at the happy hout for Gergely’s Pragmatic Summit! Not pictured: my glass of wine, Nathen’s box of fucks to give
After letting me know that he “only hugs on this side of the Rockies” (noted), the topic of writing books came up, to which he drawled,

“The second edition should always be shorter. I always made my second editions shorter. Shorter books are better books.”

God dammit, Martin.. Now you tell me. 🙈

In completely unrelated news, our last few chapters for “Observability Engineering, 2nd edition” go out to tech reviewers this week! This puts us on track for dead tree publication in June, although chapters will be available earlier for O’Reilly subscribers as well as behind an email gate on the Honeycomb site.

What’s different about the second edition?

Almost everything. The only chapters that carry over some material are the ones on sampling, retriever (columnar store), and a smattering of the SLO stuff—maybe 10% all told? And we’ve added a monstrous amount of new material.

So no, it will not be shorter than the first edition. It is almost twice as long.[1] (Sorry!)

The first edition was a spaghetti mess

Books, as I understand, are like children; if you made them, you are not allowed to say you aren’t proud of them.

So fine, I won’t say it. But I think we can all privately agree that the first edition was a bit of a hot mess.

No shade on my wonderful co-authors, Liz and George, or our O’Reilly editors, or myself for that matter. We did our best, but now, with the clarity of hindsight, it’s easy to see all the ways the ground was shifting under us as we wrote.

When we started the book in 2018, Honeycomb was the only observability company, and our definition of observability—high cardinality, high dimensionality, explorability—was the only definition. By the time the book came out in 2021, everyone was rebranding their products as observability, Gartner had waded into the fray.. it was a mess.

Perhaps the mature thing to do would be to have gone back and rewritten the book in light of the evolving market definition. But while I won’t speak for my co-authors, after 3.5 years, I was pretty fucking desperate to be done.

Artist’s rendering of the traditional authorial glow of pride, joy and deep satisfaction upon completing any book manuscript

I swore I would never go through that again. And when O’Reilly first approached us about writing a second edition, my first reaction was blind panic.

The second edition has a clearer mission

But once my lizard brain calmed down, I realized two things. Number one, it absolutely needed to be written; number two, I definitely wanted to help write it.

SO MUCH has changed. SO MUCH needs saying. When we met up in June to pull together a new outline, it seemed to just flow out of us.

A few of the many things that were not at all clear in 2018, but are crystal clear today:

  • Who we are writing for (software engineers)
  • What they are doing (instrumenting their code and analyzing it in production, with and without AI)
  • What observability means to analysts and the market at large (literally anything to do with software telemetry)
  • The integrations game is over, and OpenTelemetry has won
  • Most companies still don’t have real observability. And they don’t know it. 😕

I am excited and incredibly grateful for the opportunity to take a second whack at this book in the era of AI. Not how I thought I’d feel, but I will take it.

The first edition of “Observability Engineering” was translated into Japanese, Korean, and Chinese (I believe it’s Mandarin?).

We brought Austin Parker on as a fourth co-author very early, with a special emphasis on topics related to OpenTelemetry and AI.

We also invited a number of people we admire to contribute in a variety of formats… guest chapters, use cases, stories, embedded advice, and more:

  • Jeremy Morrell on how to instrument your code effectively
  • Hanson Ho and Matt Klein on observability for mobile and frontend
  • Kesha Mykhailov and Darragh Curran from Intercom on fast feedback loops and developing with LLMs
  • Dale McDiarmid on how to use Clickhouse for observability workloads
  • Rick Clark on the mechanics of driving organizational transformation in order to build and learn at the speed of AI
  • Frank Chen, a returning champion from our first edition, wrote about ontologies for your instrumentation chain
  • Phillip Carter wrote about eval pipelines and instrumenting LLMs
  • Mat Vine has a case study about moving ANZ from thresholds to SLIs/SLOs
  • Mike Kelly on managing telemetry pipelines for fun and profit
  • Hugo Santos on how to instrument your CI/CD pipelines
  • Peter Corless made our chapter on “Build vs Buy (vs Open Source)” immensely better and more well-rounded

What a fucking list, huh? 🙌

Truly, this book is a veritable rogues gallery of engineers and companies we look up to (including some of our own direct competitors 😉). The one thing all these people have in common (besides being great writers with a unique perspective, and people who are willing to return our emails) is that we share a similar vision for observability and the future of software development.

Spotted this week: Nathen Harvey, walking around, giving out fucks by the handful.

In addition to the sections written for software engineers on “Instrumentation Fundamentals” and “Analysis Workflows”, both with and without AI, we have a section on “Observability Use Cases” and another on “Technical Deep Dives”, which lets us cover even more ground.

Which brings us to the last section, the one that I personally signed up to write.

Part 6: “Observability Governance”

When we met in June, I successfully pitched Liz and George on adding one final section: “Observability Governance”. Unlike the rest of the book, these chapters would be written for the observability team, or the platform engineering team, or whoever is wrestling with problems like cost containment and tool migrations.

I sketched out a few ideas and started writing. July passed, August, September…I was cranking out one governance chapter per month, right on track, planning to wrap up well before November.

In September, halfway through my last chapter, I reached out to the internet for advice. “Are you an experienced software buyer? I could use some help.

The response was ✨tremendous✨; my inbox swelled with interesting stories, bitter rants, lessons learned, and practical tips from engineers and executives alike.

But when I tried to finish the chapter, my engine stalled out. I could not write. I kept doggedly blocking off time on my calendar, silencing interruptions, staring at drafts, writing and rewriting, trying every angle. Four weeks passed with no progress made.

Five weeks. Six.

Cliffhanger!

Tomorrow I’ll publish the second half of this story, in which the due date for my chapters comes and goes, and I end up throwing away everything I had written and starting over from scratch. Good times!

 

[1] If we ever write a third edition, I swear on the lives of my theoretical children that it will be MUCH shorter than this one.

“Observability Engineering”: a book so nice, we wrote it twice (xpost)

Got opinions on observability? I could use your help (once more, with feeling)

Last month I dropped a desperate little plea for help in this space, asking people to email me any good advice and/or strong opinions they happened to have on the topic of buying software.

I wasn’t really sure what to expect — desperate times, desperate measures — but holy crap, you guys delivered. To the many people who took the time to write up your experiences and expertise for me, and suffer through rounds of questions and drafts: ✨thank you✨. And thank you, too, to those of you who forwarded my queries along to experts in your network and asked for help on my behalf.

I learned a LOT about buying software and managing vendor relationships in the process of writing this. Honestly, this chapter is shaping up to be one of the things I’m most excited about for the second edition of the book.

Why I’m excited about the software buying chapter (& you should be too)

I’m imagining you reading this with a skeptical expression and an arched eyebrow. “Really, Charity…‘how to buy software’ doesn’t exactly suggest peak engineering prowess.”

Au contraire, my friends. I’ve come to believe that vendor engineering is one of the subtlest and most powerful practical applications of deep subject matter expertise, and some of the highest leverage work an engineer can do. How often do you get to make decisions that leverage the labor of hundreds or thousands of engineers per year, for fractions of pennies on the dollar? How many of the decisions you make will have an impact on every single engineer you work with and their ability to do their jobs well, as well as the experience of every single customer?

If you think I’m hyperventilating a bit, nah; this is entry level shit. In the book, I tell the story of the best engineer I ever worked with, and how I watched him alter the trajectory of multiple other companies, none of which he was working for, buying from, or formally connected to in any way — in the space of a few conversations. It upended my entire worldview about what it can look like for an engineer to wield great power.

Doing this stuff well takes both technical depth and technical breadth, in addition to systems thinking and knowledge of the business. It is one of the only ways a staff+ engineer can acquire and develop executive-level communication, strategy, and execution skills while remaining an individual contributor.

I’ve been wanting to write about this for YEARS. Anyway — ergh! — I’m rambling now. That was not what I came here to talk about, I’m just excited. Back to the point.

My second (and final) round of questions

I got so much out of your thoughtful responses that I thought I’d press my luck and put a few more questions out to the universe, before it’s too late.

These questions speak to areas where I worry that my writing may be a little weak or uninformed, or too far away from the world where people are using the “three pillars” model (aka multiple pillars or o11y 1.0) and happy about it. I don’t know many (any??) of those people, which suggests some pretty heavy selection bias.

I don’t expect anyone to answer all the questions; if one or two resonate with you, write about those and ignore the rest. If there’s something I didn’t ask that I should have asked, answer that. Something I’ve written in the past that bugged you that you hope I won’t say again? Tell me! We are almost out of time ⌛ so gimme what you got. 🙌

On migrations:

📈 Have you ever migrated from one observability vendor to another? If so, what did you learn? What was the hardest part, what took you by surprise? What do you wish you could go back in time and tell your self at the start?

📈 If you ran (or were involved in) a large scale migration or tool change… how did you structure the process? Like, was it team by team, service by service, product by product? Did you have a playbook? What did you do to make it fun or push through organizational inertia? How long did it take?

On managing costs for the traditional three pillars:

📈 For orgs that are using Datadog, Grafana, Chronosphere, or another traditional three pillars architecture.. How would you describe your approach to cutting and controlling costs? Pro tips and/or comprehensive strategy.

📈 Alternately, if there are particular blog posts with advice you have followed and can personally vouch for, would you send me a link?

📈 How do you guide your software engineers on which data to send to which place — metrics, logs, traces, errors/exceptions, profiling, etc? How do you manage cardinality? How do you work to keep the pillars in sync, or are there any particular tips and tricks you have for linking / jumping between the data sources?

📈 How many ongoing engineering cycles does it take to manage and maintain costs, once you’ve gotten them to a sustainable place?

On managing costs at massive scale:

(Especially for people who work at a large enterprise, the kind with multiple business units, but others welcome too!):

  • Do you use tiers of service for managing costs? How do you define those?
  • How do new tools get taken for a spin? (Like, sometimes there is an office of the CTO with carte blanche to try new things and evaluate them for the rest of the org)
  • How do you use telemetry pipelines?

Observability teams (quick poll):

📈 If you have an observability team, how big is it? What part of the org does it report up into? Roughly how many engineers does that team support?

📈 If you don’t have an observability team — and you have more than, say, 300 engineers — who owns observability? Platform? SRE? Other?

A grab bag:

📈 Build vs Buy: If you built your own observability tool(s)…. What were the reasons? What does it do? Would you make the same decision today?

📈 OpenTelemetry: If your team has weighed the pros and cons of adopting OTel and ultimately decided not to, for technical or philosophical reasons (i.e. not just “we’re too busy”) — what are those reasons?

📈 Instrumentation: what do you do to try and remove cognitive overhead for engineers? How much have you been able to make automatic and magical, and where has the magic failed?

📈 Consolidation: I would love to hear any thoughts on tool consolidation vs tool proliferation. Is this primarily driven by execs, or do technical users care too? Is it driven by cost concerns, usability, or something else?

edited on 2025-10-15 to add… oh crap, one last question:

📈 Open source: Are you using open source observability tools, and if so, are these your primary tools or one piece of a comprehensive tooling strategy? If the latter, could you describe that strategy for me?

Send it to me in an email

Please send me your opinions or answers in an email, to my first name at honeycomb dot io, with the subject line “Observability questions”.

If I end up cribbing from your material, it okay for me to print your name? (As in, “thanks to the people who informed my thinking on this subject, abc xyz etc”). I will not mention your employer or where you work, don’t worry.

If you send it to me more than a week from now, I probably won’t be able to use it. Augh, I wish I had thought of this in JUNE!!! #ragrets

✨THANK YOU✨

I know this is an incredibly time consuming thing to ask of someone, and I can’t express how much I appreciate your help.

P.S. Yes, the title is absolutely a reference to the Buffy musical. Hey, I had to give you guys something fun to read along with my second bleg in less than a month (do people still say “bleg”??).

6 Musical Episodes of TV Shows That Deserve an Encore

P.P.S. Grammar quiz of the day: should my title read “opinions ABOUT observability” or “opinions ON observability” ??

GREAT QUESTION — and, as it turns out, the preposition you choose may reveal more than you realized.

“About” is used to introduce a topic or subject in a broad, vague, or approximate sense, while “on” is used to signal more detailed, specific, formal or serious subject matter (as well as physical objects). “Let’s talk about dinner” vs “she delivered a lecture on why AI is trying to kill babies.”

Or as Xander says, “To read makes our English speaking good.”

The earth is doomed,
~charity

Got opinions on observability? I could use your help (once more, with feeling)

Are you an experienced software buyer? I could use some help.

If it seems like I’ve been relatively quiet lately on social media and my blog, that’s because I have. Liz, Austin, George and I have been busy toiling away on the second edition of “Observability Engineering” ever since April or May. I personally have been trying to spend 75-80% of my time on the book since May.

Have I been successful in that attempt? No. But I’m trying. Progress is being made. Hopefully just a few more weeks of drafting and we’ll be on to edits, and on to your grubby little paws by May-ish.

The world has changed A LOT since we wrote the first edition, in 2019-2022. Do you know, the phrase “observability engineering teams” doesn’t even occur in the first edition of the book? Try and search — it can’t be found! Even the phrase “observability teams” doesn’t pop up til near the end, and when it does, we are referring to those few teams that choose to build their own observability tools from scratch.

These days, observability engineering teams are everywhere. Which is why we are adding a whole new section, a sizable one, called “Observability Governance.” The governance section will have a bunch of chapters on topics like how to staff these teams, where they should fit in the org chart, how to buy good tools, how to integrate them, how to manage costs, how to make the business case up the chain to senior execs, how to manage schemas and semantic conventions at scale, and much much more.

The problem

The problem is, I’ve never really bought software. Not like this. I’ve never even worked at a  truly large, software-buying enterprise tech company. So I am not well equipped to give good advice on questions like:

  • How do you shop around for options?
  • What are some signs you may need to suck it up and change vendors?
  • What does a good POC (proof of concept) look like?
  • Who are your stakeholders? What are their concerns?
  • How do you drive consensus when millions of dollars (and the work experience of thousands of engineers) are on the line? What does ‘consensus’ even mean in that context?
  • What are the primary considerations should you take into account when making a decision? What are secondary considerations?

I’m looking for the kind of advice that a principal engineer who has done this many times might give a staff engineer who is doing it for the first time. Or that a VP who’s done this many times might give a director who is doing it for the first time.

Can you help?

This is me wearing Leia buns and projecting a unicorn-shaped rainbow bat signal out into the sky for help. Do you have any advice for me? What guidance would you give to the readers of the second edition of this book?

Please send your advice to me in an email, addressed to my first name at honeycomb dot io, with the subject line: “Buying Software”. Include any relevant context about how large the company or engineering org is, and what your role in purchasing was.

I may respond with more questions, or reply and ask if you are able to talk synchronously. But I will not quote anything you send me without first asking your permission and getting a signed release. I will not mention ANY vendors by name, good or bad.

I am not fishing for honeycomb customers or buyers, I assume most of you haven’t tried honeycomb and don’t care about it and that is fine. This is not a Honeycomb project, this is an O’Reilly writing project. I just want to gather up some good advice on buying software and funnel it back out to good engineers.

Can you help? Your industry needs you! <3

 

 

Are you an experienced software buyer? I could use some help.

The Hierarchy Is Bullshit (And Bad For Business)

My friend Molly has had an impressive career. She got a job as a software engineer after graduating from college, and after kicking ass for a year or so she was offered a promotion to management, which she accepted with relish. Molly was smart, driven, and fiercely ambitious, so she swiftly clambered up the ranks to hold director, VP, and other shiny leadership roles. It took two decades, an IPO and a vicious case of burnout before she allowed herself to admit how much she hated her work, and how desperately she envied (guess who??) the software engineers she worked alongside. Turns out, all she ever really wanted to do was write code every day. And now, to her dismay, it felt too late.

Why did it take Molly so long to realize what made her happy? I personally blame the fucking hierarchy.

The Hierarchy Lie

The “Big Lie” of hierarchy is that your organizational structure is a vertical tree from the CEO on down, where higher up is always better.

Of course any new grad is going to feel that way, on the heels of 15-20 years spent going through school year by year, grade by grade, measuring success via good grades and teacher approval. The early years of professional life are a similar blend of hard work, leveling up and basic skills acquisition. (They got Molly hopped on the leveling treadmill before she even had a chance to become a real adult, in other words. 😍)

But by the time you are fully baked as a senior contributor, maybe 7-8 years in, your relationship to levels and ladders should undergo a dramatic shift. At some point you have to learn to tune in to your own inner compass. What draws you in to your work? What fuels your growth and success?

Being an adult means not measuring yourself entirely on other people’s definition of success. Personal growth might come in the guise of a big promotion, but it also might look like a new job, a different role, a swing to management or back, becoming well-known as a subject matter expert, mentoring others, running an affinity group, picking up new skill sets, starting a company, trying your hand at consulting, speaking at conferences, taking a sabbatical, having a family, working part time, etc. No one gets to define that but you.

You have a thirty- or forty-year adult life and career in front of you. What the hell are you going to do with all that time and space??

Your career is not one mad sprint to the finish line

Literally nobody’s career looks like a straight line, going up, up up and to the right, from intern to CEO (to a coffin).

One of the most exhausting things about working at Facebook was the way engineering levels feltLiterally no one's career, ever. like a hamster wheel, where every single quarter you were expected to go go go go go, do more do more, scrape up ever more of your mortal soul to pour in more than you could last quarter — and the quarter before that, and before that, in ever-escalating intensity.

It was fucking exhausting, yo. Life does not work that way. Shit gets hilly.

The strategy for a fulfilling, lifelong career in tech is not to up the ante every interval. Nor is it to amass more and more power over others until you explode. Instead:

  1. Train yourself to love the feeling of constantly learning and pushing your boundaries. Feeling comfortable is the system blinking orange, and it should make you uneasy.
  2. Follow your nose into work that lights you up in the morning, work you can’t stop thinking about. If you’re bored, do something else.
  3. Say yes to opportunities!! Intensity is nothing to be afraid of. Instead of trying to cap your speed or your growth, learn to alternate it with recovery periods.
  4. If you aren’t sure what to do, make the choice that preserves or expands future optionality. Remember: Most startups fail. Will you be okay with your choices if (& when) this one does too?

Why do people climb the ladder? “Because it’s there.” And when they don’t have any other animating goals, the ladder fills a vacuum.

But if you never make the leap from externally-motivated to intrinsically-motivated, this will eventually becomes a serious risk factor for your career. Without an inner compass (and a renewable source of joy), you will struggle to locate and connect with the work that gives your life meaning. You will risk burnout, apathy and a serious lack of fucks given..

The times I have come closest to burnout or flaming out have never been when I was working the hardest, but when I cared the least. Or when I felt the least needed.📈📉💔

A disturbing number of companies would rather feel in control than unclench and perform better

But hey! Lack of inner drive isn’t the ONLY thing that drives people to climb the ladder. Plenty of companies fuck this up too, all on their lonesome. Let’s talk about more of the ways that companies mess up the workplace! Like by disempowering the people doing the work and giving all the power to managers, thereby forcing anyone who wants a say in their own job become one.

The way we talk about work is riddled with hierarchical, authoritarian phrases: “She was my superior”, “My boss made me do it”, “I got promoted into management”, and so on.

There are plenty of industries where line workers are still disempowered cogs and power structures are hierarchical and absolute (like flipping burgers at McDonalds, or factory line work). There are even software companies still trying to make it work in command-and-control mode, to whom engineers are interchangeable monkeys that ship story points and close JIRA tasks.

But if there’s one thing we know, it’s that for industries that are fueled by creativity and innovation, command-and-control leadership is poison. It stifles innovation, it saps initiative, it siphons away creativity and motivation and caring.

Studies also show that the more visible someone’s power is, the less likely anyone is to give them honest feedback.[2]

Companies that don’t learn this lesson are unlikely to win over the long run. Engineering is a deeply creative occupation, and authoritarian environments are toxic for creativity and people’s willingness to share information.

Hierarchy is just a data structure

The basic function of a hierarchy is to help us make sense of the world, simplify information, and make decisions. Hierarchy lets us break down enormous projects — like “let’s build a rocket!”, or “let’s invade the moon!” — into millions of bite size decisions and tasks, and this is how progress gets made.

A certain amount of authority is invested into the hierarchy model. If you are responsible for delivering a unit of work, the company needs to make sure you have enough resources and decision-making ability to do so. This is what we think of as the formal power structure [1], and there is nothing wrong with that. It’s what makes the system work.

The problem starts when we stop thinking of hierarchy as a neutral data structure — a utilitarian device for organizing groups and making decisions — and start projecting all kinds of social status and dominance onto it.

A sensitivity to social dominance is wired deep, deep into our little monkey brains. It’s what tells us we deserve more power, leverage, pride, influence, and autonomy — and simply have more value — than those below us. It’s what tells us those above us are better, stronger and more deserving than we are, and that we owe them our respect and deference.

It also tells us “if you lose status, YOU MIGHT DIE” 😱😱😱 which is why we may react to a perceived loss of status with a sting that seems astonishingly extreme and overwrought, even to ourselves, yet somehow impossible to shrug off.

hierarchies tend to get mixed up with social dominance

In general, it is better to pursue roles and growth based on the affirmative (what it is you want to learn, grow or do more of) than the negative (what you want to avoid, evade or stop doing). Your motivation systems don’t kick in to gear when you are feeling “lack of pain” — the system doesn’t work that way. They kick in when you get interested.

And if you are sick of doing something or being treated a certain way, chances are everyone else will hate it, too. Who wants to work at a company where all the shit rolls downhill?

Hierarchies have stuck around for one very good reason: because they work. Hierarchies are simple, intuitive, and allow large numbers to collaborate with low cognitive overhead. Unfortunately, most hierarchies become entwined with status and dominance markers, which can bring enormous downsides. At their worst, they can suck the literal life out of work, reducing us all to glum little cogs obeying orders.

We aren’t getting rid of hierarchy anytime soon. But we can use culture and ritual to gently untangle them from dominance, and we can choose to interpret formal power as a service function instead of a dictatorship. This frees people up to choose their work based on what makes them feel fulfilled, instead of their perceived status. (Also helpful? Flatter pay bands. 😛)

Good managers do not dictate and demand, they nurture, develop, and inspire. The most important roles in the company aren’t held by managers; they are all the little leaf nodes  busily building the product, supporting users, identifying markets, writing copy, etc. The people doing the work are why we exist as a company; all the rest is, with considerable due respect, overhead.

How to drain your hierarchy of social dominance

When it comes to hierarchy and team structure, there are the functional, organizational aspects (mostly good) and the social dominance parts (mostly bad). With that in mind, there are plenty of smaller things we can do as a team to remind people that we are equal colleagues, simply with different roles.

  • Be conscious of the language you use. Does it reinforce dominance and hierarchy? (Step one: stop calling management “a promotion”🥰)
  • De-emphasize trappings of power. The more you refer to someone’s formal power, the less likely anyone is to give them critical feedback or question them.
  • Push back against common but unhelpful practices, like “a manager should always make more money than the people who report to them.” Really? Why??
  • Are there opportunities for career advancement as an IC, or only as a manager? Everyone should have the ability to advance in their career.
  • Do your own dishes, everyone.
  • Practice visualizing the org chart upside down, where managers and execs support their teams from below rather than topping them from above. (I was going to write a whole post about this, then discovered other people have been doing that for the past decade. 🤣)

And then there is the big(ger) thing we can (and must!) do, in order to 1) make people go into management for the right reasons, 2) help senior IC roles remain attractive to highly skilled creative and technical contributors, and 3) encourage everybody to make career decisions based on curiosity, growth, and what’s best for the business, instead of turf and power grabs. Which is:

Practice transparency, from top to bottom

Share authority, decision-making and power

Technical contributors own technical decisions

Most people who go in to management don’t do it out of a burning desire to write performance reviews. They do it because they are fed the fuck up with being out of the loop, or not having a say in decisions over their own work. All they want is to be in the room where it happens, and management tends to be the only way you get an invite.

EVERY company says they believe in transparency, but hardly any of them are, by my count. Transparency doesn’t mean flooding people with every trivial detail, or freaking them out with constant fire drills. It does mean being actively forthcoming about important questions and matters which are happening or on the horizon…often before you are fully comfortable with it. Honestly, if you never feel any discomfort about your level of transparency, you probably aren’t transparent enough.

People do better work with more context! You’re equipping them with information to better understand the business problems and technical objectives, and thereby unleashing them and their creativity to help solve them. You’re also opening yourself up to questioning and sanity checks — which may feel uncomfortable, but 🌞sunlight is sanitizing🌞 — it is worth it.

Some practical tips for transparency

At Honeycomb, we present the full board deck after every board meeting in our all hands, and take questions. When we’re facing financial uncertainty, we say so, along with our working plan for dealing with it. We also do org-level updates in all hands, once per quarter per org. Each org presents a snapshot to the company of how they are doing, but we ask that no more than 2/3 of the presentation be about their successes and triumphs, and 1/3 of their material be about their failures and misses. Normalize talking about failure.

Being transparent isn’t about putting everyone on blast; it’s about cultivating a habit of awareness about what might be relevant to other people. It’s about building systems of feedback, updates and open questioning into your culture. This can be scary, so it’s also about training yourselves as a team to handle hard news without overreacting or shooting the messenger. If you always tell people what they want to hear, they’ll never trust you. You can’t trust someone’s ‘yes’ until you hear their ‘no’.

Transparency is always a balance between information and distraction, but I think these are healthy internal rules of thumb for management:

  1. If anyone has further questions or wants to know more details than what was shared, they are free to ask any manager or exec, who will willingly answer more fully, up to the boundaries of privacy or legal reasons. As employees, they have a right to know about the business they are part of. A right — not a privilege, which can be revoked on a whim.
  2. When making internal decisions about e.g. salary bands, individual exceptions to formal policy, etc, ask each other … if this decision were to leak, could we justify our reasoning with head held high? If you would feel ashamed, or if you really don’t want people to find out about it, it’s probably the wrong decision.

Some practical tips for distributing power

Power flows to managers by default, just like water flows downhill. Managers have to actively push back on this tendency by explicitly allocating powers and responsibilities to tech leads and engineers. Don’t hoard information, share context generously, and make sure you know when they would want to tap in to a discussion. Your job is not to “shield” them from the rest of the org; your job is to help them determine where they can add outsize value, and include them. Only if they trust you to loop them in will they feel free to go heads down and focus.

Wrap your senior ICs into planning and other leadership activities. Decisions about sociotechnical processes (code reviews, escalation points, SLI/SLOs, ownership etc) are usually better owned by staff+ engineers than anyone on the management track. Invite a couple of your seniormost engineers to join calibrations — they bring a valuable perspective to performance discussions that managers lack.

Demystify management. Blur the lines between people managers and engineers; delegate ownership and accountability for some important projects to ICs. Ask every engineer about their career interests, and if management is on the list, find opportunities for them to practice and improve at managerial skills — mentoring, interviewing, onboarding, etc.

Adults don’t like being told what to do

People do phenomenal work when they want to do it, when they are creatively and emotionally engaged at the level of optimum challenge, and when they know their work matters. That’s where you’ll find your state of flow. That is where you’ll do your best work, which is also the best way to get promoted and make durable advances in your career.

Not, ironically, by chasing levels and titles for their own sake. ☺️

People want to be challenged. They want you to ask them to step up and take responsibility for something hard. They want to be needed, and they want to have agency in the doing of it. Just like you do.

Oh yeah, back to Molly …

Molly, who I mentioned at the beginning, joined Honeycomb five years ago as a customer success exec. After realizing she wanted to go back to engineering, she switched to working our support desk to build up her technical chops while she practiced writing code on the side. She has now been working as a software engineer on the product team for over two years, and she is ✨rocking it.✨ It is NEVER too late. 🙌

<3 charity

p.s. Molly also says, “don’t waste time at bad companies, whether you’re climbing the ladder or not!” 🥂

 

[1] Formal power is only one kind of power, and in some ways it is the weakest, because it doesn’t belong to you. It belongs to the company and is only loaned out for you to wield on its behalf. (You don’t carry the innate ability to fire people along with you after you stop being an engineering manager, for example.) Formal powers are limited, enumerated, and functional. You don’t get to use them for any reason other than furthering the goals of the org, or else it is literally an abuse of power.

Formal power is fascinating in another way, too: which is that your formal power is seen as legitimate only if you ~basically always wield it in the ways everyone already expects you to. You can make a surprising call only so often; you can straight up overrule the wishes of your constituents extremely rarely. If you use your formal power to do things that people disagree with or don’t support, without taking the time to persuade them or create real consensus, you will squander your credibility and good faith unbelievably fast.

[2] I am not going to bother rustling up lots of links and citations, because I expect most of this falls into the voluminous category of “shit you already knew”. But if any of it sounds surprising to you, here are some classic reference works:

Flow, by Mihaly Csikszentmihalyi
Drive, by Dan Pink
The Culture Code: Secrets of Successful Groups, by Daniel Coyle
A Lapsed Anarchist’s Guide to Being a Better Leader, by Ari Weinzweig

[3] The scientific literature suggests that dominating instincts tend to emerge in more overtly hostile environments. Make of that what you will, I guess.

 

Some other writing I have done on this topic, or topics adjacent …

The Engineer/Manager Pendulum
The Pendulum or the Ladder
If Management Isn’t a Promotion, then Engineering isn’t a Demotion
Twin Anxieties of the Engineer Manager Pendulum
Things to Know About Engineering Levels
Advice for Engineering Managers who want to Climb the Ladder
On Engineers and Influence
Is There a Path Back from CTO to Engineer?

The Hierarchy Is Bullshit (And Bad For Business)

Quarantine Reading Queue on the “Tiger King” Phenomenon

Last Wednesday I walked into my living room and saw three gay rednecks in hot pink shirts being married as a “throuple” on a TV screen at close range, followed by one of the grooms singing a country song about a woman feeding her husband’s remains to her tigers.

I could not look away.  What the fuck.

If you too have been rubbernecking the Tiger King — at any range — I have a book that will help you make sense of things: “Blood Rites: On The Origins and History of the Passions of War“, by Barbara Ehrenreich[1].  I re-read it last night, and here is my book report.


throuple

 

In Blood Rites, Ehrenreich asks why we sacralize war.  Not why we fight wars, or why we are violent necessarily, but why we are drawn to the idea of war, why we compulsively imbue it with an aura of honor and noble sacrifice.  If you kill one person, you’re a murderer and we shut you out from society; kill ten and you are a monster; but if you kill thousands, or kill on behalf of the state, we give you medals and write books about you.

And it’s not only about scale or being backed by state power.  The calling of war brings out the highest and finest experiences our species can know: it sings of heroism and altruism, of discipline, self-sacrifice, common ground, a life lived well in service; of belonging to something larger than one’s self.  Even if, as generations of weary returning soldiers have told us, it remains the same old butchery on the ground, the near-religious allure of war is never dented for long in the popular imagination.

What the fuck is going on?  bloodrites

Ehrenreich is impatient with the traditional scholarship, which locates the origin of war in some innate human aggression or turf wars over resources.  She is at her dryly funniest when dispatching feminist theories about violence being intrinsically male or “testosterone poisoning”, showing that the bloodthirstiest of the gods have usually been feminine.  (Although there are fascinating symmetries between girls becoming women through menstruation, and boys becoming men through … some form of culturally sanctioned ritual, usually involving bloodshed.)

Rather, she shows that our sacred feelings towards blood shed in war are the direct descendents of our veneration of blood shed in sacrifice — originally towards human sacrifice and other animal sacrifice, in a reenactment of our own ever-so-recent role inversion from prey to predator.  Prehistoric sacrifice was likely a way of exerting control over our environment and reenacting the death that gave us life through food.

In her theory, humans do not go to war because we are natural predators. Just the blink of an eye ago, on an evolutionary scale, humans were not predators by any means: we were prey.  Weak, blind, deaf, slow, clawless and naked; we scrawny, clever little apes we were easy pickings for the many large carnivores who roamed the planet.  We scavenged in the wake of predators and worshiped them as gods.  We are the nouveaux riche of predators, constantly re-asserting our dominance to soothe our insecurities.

We go to war not because we are predators, in other words, but because we are prey — and this makes us very uncomfortable!  War exists as a vestigial relic of when we venerated the shedding of blood and found it holy — as anyone who has ever opened the Old Testament can attest.  It was not until the Axial Age that religions of the world underwent a wholesale makeover into a less bloody, more universalistic set of aspirations.  ashes

When I first read this book, years ago, I remember picking it up with a roll of the eyes.  “Sounds like some overly-metaphorical liberal academic nonsense” or something like that.  But I was hooked within ten pages, my mind racing ahead with even more evidence than she marshals in this lively book.  It shifted the way I saw many things in the world.

Like horror movies, for example.  Or why cannibalism is so taboo.  How Jesus became the Son of God, the Brothers’ Grimm, the sacrament of Communion.  The primal fear of being food still resonates through our culture in so many sublimated ways.

And whether what you’re watching is “Tiger King” or the Tiger-King-watchers, it will make A LOT more sense after reading this book too.

Stay safe and don’t kill each other,

charity

IMG_6288

 

[1]  Ehrenreich is best known for her stunning book on the precariousness of the middle class, “Nickel and Dimed”, where she tried to subsist for a year only on whatever work she could get with a high school education.  Ehrenreich is a journalist, and this is a piece of science journalism, not scientific research; yet it is well-researched and scrupulously cited, and it’s worth noting that she has a PhD in biology and was once a practicing scientist.

 

 

Quarantine Reading Queue on the “Tiger King” Phenomenon