How “Engineering-Driven” Leads to “Engineering-Supremacy”

Honeycomb has a reputation for being a very engineering-driven company. No surprise there, since it was founded by two engineers and our mission involves building an engineering product for other engineers.

We are never going to stop being engineering-driven in the sense that we are building for engineers and we always want engineers to have a seat at the table when it comes to what we build, and how, and why. But I am increasingly uncomfortable with the term “engineering-driven” and the asymmetry it implies.

We are less and less engineering-driven nowadays, for entirely good reasons — we want this to be just as much of a design-driven company and a product-driven company, and I would never want sales or marketing to feel like anything other than equal partners in our journey towards revolutionizing the way the world builds and runs code in production.

It is true that most honeycomb employees were engineers for the first few years, and our culture felt very engineer-centric. Other orgs were maybe comprised of a person or two, or had engineers trying to play them on TV, or just felt highly experimental.

But if there is one thing Christine and I were crystal clear on from the beginning, it’s this:

✨WE ARE HERE TO BUILD A BUSINESS✨

Not just the shiniest, most hardcore tech, not just the happiest, most diverse teams. These things matter to us — they matter a lot! But succeeding at business is what gives us the power to change the world in all of these other ways we care about changing it.

If business is booming and people are thirsty for more of whatever it is you’re serving, you pretty much get a blank check for radical experiments in sociotechnical transformation, be that libertarian or communitarian or anything in between.

If you don’t have the business to back it up, you get fuck-all.

Not only do you not get shit, you risk being pointed out as a Cautionary Tale of “what happens if $(thing you deeply care about) comes true.” Sit on THAT for a hot second. 😕

So yes, we cared. Which is not to say we knew how to build a great business. We most certainly did not. But both of us had been through too many startups (Linden Lab, Aardvark, Parse, etc) where the tech was amazing, the people were amazing, the product was amazing … and the business side just did not keep up. Which always leads to the same thing: heartbreak and devastation.

✨IF you can’t make it profitable✨
✨your destiny will inevitably be✨
✨taken out of your hands✨
✨and given to someone else.✨

So Christine and I have been repeating these twin facts back and forth to each other for over six years now:

  1. Honeycomb must succeed as a revenue-generating, eventually profitable business.
  2. We are not business experts. Therefore we have to make Honeycomb a place that explicitly values business expertise, that places it on the same level as engineering expertise.

We have worked hard to get better at understanding the business side (her more than me 🙃) but ultimately, we cannot be the domain experts in marketing or sales (or customer success, support, etc).

What we could do is demonstrate respect for those functions, bake that respect into our culture, and hire and support amazing business talent to run them with us.

On being “engineering-driven”

Self-described “engineering-driven” companies tend to fall into one of two traps. Either they alienate the business side by pinching their nose and holding business development at arm’s length (“aaahhh, i’m just an engineer! I have no interest in or capacity for participating in developing our marketing voice or sales pitch decks, get off me!”), or they act like engineering is a sort of super-skillset that makes you capable of doing everybody else’s job better than they possibly could. As though those other disciplines and skill sets aren’t every bit as deep and creative and challenging in their own right as developing software can be.

For the first few years we really did use engineers for all of those functions. We were trying to figure out how to build and explain and sell something new, which meant working out these things on the ground every day with our users. Engineer to engineer. What resonated? What clicked? What worked?

So we hired just a few engineers who were interested in how the business worked, and who were willing to work like Swiss Army knives across the org. We didn’t yet have a workable plan in place, which is what you need in order to bring domain experts on board, point them in a direction, and trust them to do what they do best in executing that plan.

Like I said, we didn’t know what to do or how to do it. But at least we knew that. Which kept us humble. And translated into a hard, fast rule which we set early on in our hiring process:

We DO NOT hire engineers who talk shit about sales and marketing.

If I was interviewing an engineer and they made any alienating sort of comment whatsoever about their counterparts on the business side, it was an automatic no. Easy out. We had a zero tolerance policy for talking down down about other functions, or joking, even for being unwilling to perform other business functions.

In retrospect, I think this is one of the best decisions we ever made.

Hiring engineers who respected other functions

We leaned hard into hiring engineers who asked curious questions about our business strategy and execution. We pursued engineers who talked about wanting to spend time directly with users, who were intrigued by the idea of writing marketing copy to help explain concepts to engineers, and who were ready, willing and able to go along on sales calls.

Once we finally found product-market fit, about 2-3 years ago, we stopped using engineers to play other roles and started hiring actual professionals in product, design, marketing, sales, customer success, etc to build and staff out their organizations. That was when we first started building the business for the longer term; until we found PMF, our event horizon was never more than 1-3 months ahead of “right now”.

(I’ll never forget going out to coffee with one of our earlier VPs of marketing, shortly after she was hired, and having her ask, in bemusement: “Why are all these engineers just sitting around in the #marketing channel? I’ve never had so many people giving opinions on my work!” 🙃)

Those early Swiss Army knife engineers have since stepped back, gratefully, into roles more centered around engineering. But that early knee-jerk reaction of ours established an important company norm that pays dividends to this day.

Every function of the business is equally challenging, creative, and worthy of respect. None of us are here to peacock; our skill sets serve the primary business goals. We all do our jobs better when we know more about how each other’s functions work.

These days, it’s not just about making sure we hire engineers who treat business counterparts like equals. It’s more about finding ways to stimulate the flow of information cross-functionally, creating a hunger for this information.

Caring about the big picture is a ✨learnable skill✨

You can try to hire people who care about the overall business outcomes, not just their own corner of reality, and we do select for this to  some degree — for all roles, not just engineers.

But you can also foster this curiosity and teach people to seek it out. Curiosity begets curiosity, and every single person at Honeycomb is doing something interesting. We all want to succeed and win together, and there’s something infectious and exciting about connecting all the dots that lead to success and reflecting that story back to the rest of the company.

For example,

    1. Every time we close a deal, a post gets written up and dropped into the announcements slack by the sales team. Not just who did we close and how much money did we make, but the full story of that customer’s interaction with honeycomb. How did they hear of us? Whose blog posts, training sessions, or office hours did they engage with? Did someone on the telemetry pull a record-fast turnaround on an integration they needed to get going? What pains did we solve effectively for them as a tool, and where were the rough edges that we can improve on in the future?

      The story is often half a page long or more, and tags a dozen or more people throughout all parts of the company, showing how everyone’s hard work added up and materially contributed to the final result.

 

    1. We have orgs take turn presenting in all hands — where they’re at, what they’ve built, and the impact of their contributions, week after week. Whether that’s the design team talking about how they’ve rolled out our new design system and how it is going to help everyone in the company experiment more and move more quickly, or it’s the people team showing how they’ve improved our recruiting, interviewing, and hiring processes to make people feel more seen, welcomed, and appreciated throughout the process.

      We expect people to be curious about the rest of the company. We expect honeybees to be interested in, excited about and celebratory of each other’s hard work. And it’s easy to be excited when you see people showing off work that they were excited to do.

 

    1. We have a weekly Friday “demo day” where people come and show off something they’ve built this week, rapid-fire. Whether it’s connecting to a mysql shell in the terminal to show off our newly consistent permissioning scheme, or product marketing showing off new work on the website. Everybody’s work counts. Everybody wants to see it.

 

  1. We have a #love channel in slack where you can drop in and tag someone when you’re feeling thankful for how much they just made your day better. We also have a “Gratefuls” section during all hands, where people speak up and give verbal props to coworkers who have really made a difference in their lives at work.

We have always attracted engineers who care about the business, not just the technology and the culture. As a result, we have consistently recruited and retained business leaders who are well above our weight class — our investors still sometimes marvel at the caliber of the business talent we have been able to attract. It is way above the norm for developer tools companies like ours.

“Engineering-driven” can be a mask for “engineering-supremacy”

Because the sad truth is that so many companies who pride themselves on being “engineering-driven” are actually what I would call more “engineering-supremacist”. Ask any top-tier sales or marketing leader out there about their experiences in the tech industry and you’ll hear a painful, rage-inducing list of times they were talked down to by technical founders, had their counsel blown off or overridden, had their plans scrapped and their budgets cut, and every other sort of disrespectful act you can imagine.

(I am aware that the opposite also exists; that there are companies and cultures out there that valorize and glorify sales or marketing while treating engineers like code monkeys and button pushers, but it’s less common around here. In neither direction is this okay.)

This isn’t good for business, and it isn’t good for people.

It is still true that engineering is the most mature and developed organization at the company, because it has been around the longest. But our other orgs are starting to catch up and figure out what it means to “be honeycomby” for them, on their own terms. How do our core values apply to the sales team, the developer evangelists, the marketing folks, the product people? We are starting to see this play out in real time, and it’s fascinating. It is better than forcing all teams to be “engineering-driven”.

Business success is what makes all things possible

We are known for the caliber of our engineering today. But none of that matters a whit if you never hear about us, or can’t buy us in a way that makes sense for you and your team, or if you can’t use the product, or if we don’t keep building the right things, the things you need to modernize your engineering teams and move into the future together.

When looking towards that future, I still want us to be known for our great engineering. But I also want us to be a magnet for great designers who trust that they can come here and be respected, for great product people who know they can come here and do the best work of their life. That won’t happen if we see ourselves as being “driven” by one third of the triad.

Supremacy destroys balance. Always.

And none of this, none of this works unless we have a surging, thriving business to keep the wind in our sails.

~charity

How “Engineering-Driven” Leads to “Engineering-Supremacy”

Why I hate the phrase “breaking down silos”

We hear this phrase constantly: “I worked at breaking down silos.” “We need to break down silos.” “What did I do in my last role? I broke down silos.”

It sets my fucking teeth on edge.

What is a ‘silo’, anyway? What specifically wasn’t working well, and how did you solve it; or how was it solved, and what was your contribution to the solution? did you just follow orders, or did you personally diagnose the problem, or did some of your suggestions pan out?

Solutions to complex problems rarely work on the first go, so … what else did you try? How did you know it wasn’t working, how did you know when to abandon earlier ideas? It’s fiendishly hard to know whether you’ve given a solution enough time to bake, for people to adjust, so that you can even evaluate whether it works better or worse off than before.

Communication is not magic pixie dust

Breaking down silos is supposed to be about increasing communication, removing barriers and roadblocks to collaboration.

But you can’t just blindly throw “more communication” at your teams. Too much communication can be just as much of a problem and a burden as too little. It can distract, and confuse, and create little eddies of information that is incorrect or harmful.

The quantity of the communication isn’t the issue, so much as the quality. Who is talking to whom, and when, and why? How does information flow throughout your company? Who gets left out? Whose input is sought, and when, and why? How can any given individual figure out who to talk to about any given responsibility?

Every time you say "break down silos", I want to "break down your face." | News EcardWhen someone says they are “breaking down silos”, whether in an interview, a panel, or casual conversation, it tells me jack shit about what they actually did.

cliches are a substitute for critical thinking

It’s just like when people say “it’s a culture problem”, or “fix your culture”, or “everything is about people”. These phrases tell me nothing except that the speaker has gone to a lot of conferences and wants to to sound cool.

If someone says “breaking down silos”, it immediately generates a zillion questions in my mind. I’m curious, because these problems are genuinely hard and people who solve them are incredibly rare.

Unfortunately, the people who use these phrases are almost never the ones who are out there in the muck and grind, struggling to solve real problems.

When asked, people who have done the hard labor of building better organizations with healthy communication flows, less inefficiency, and alignment around a single mission — people who have gotten all the people rowing in the same direction — tend to talk about the work.

People who haven’t, say they were “breaking down silos.”

Why I hate the phrase “breaking down silos”

Software deploys and cognitive biases

There exist some wonderful teams out there who have valid, well thought through, legitimate reasons for enforcing “NO FRIDAY DEPLOYS” week in and week out, for not hooking CI/CD up to autodeploy, and for not shipping one person’s changes at a time.

And then there are the reasons most people have.

Bad decisions, and the biases they came from

 

We’re humans. 💜  We leap to conclusions with the wetware we have doing the best it can based on heuristics that feel objectively true, but are ultimately just emotional reactions based on past lived experience. And then we retroactively enshrine those goofy gut feelings with the language of noble motive and moral values.

“I tell people not to deploy to production … because I care so deeply about my team and their ability to have a quiet weekend.”

Barf. 🙄  That’s just like saying you tell your kid not to brush his teeth at night, because you care SO DEEPLY about him and his ability to go to bed calm and happy.

Once the retcon engine in your brain gets running, it comes up with all sorts of reasons. Plausible-sounding reasons! But every single argument of the items in the list above is materially false.

Deploy myths are never going away for good; they appeal to too many of our cognitive biases. But what if there was one simple thing you could do that would invert many of these cognitive biases and cause people to grapple with the question in a new way? What if you could kickstart a recalculation?

My next post will pick up right here. I’ll tell you all about the One Simple Trick you can do to fix your deploys and set you on the virtuous path of high-performing teams.

Til then, here’s what I’ve previously written on the topic.

 

Footnotes

 

Availability bias: The tendency to overestimate the likelihood of events with greater “availability” in memory, which can be influenced by how recent the memories are or how unusual or emotionally charged they may be.

Continued influence effect: The tendency to believe previously learned misinformation even after it has been corrected. Misinformation can still influence inferences one generates after a correction has occurred.

Conservatism bias: The tendency to revise one’s belief insufficiently when presented with new evidence.

Default effect: When given a choice between several options, the tendency to favor the default one.

Dread aversion: Just as losses yield double the emotional impact of gains, dread yields double the emotional impact of savouring

False-uniqueness bias: The tendency of people to see their projects and themselves as more singular than they actually are.

Functional fixedness: Limits a person to using an object only in the way it is traditionally used

Hyperbolic discounting: Discounting is the tendency for people to have a stronger preference for more immediate payoffs relative to later payoffs. Hyperbolic discounting leads to choices that are inconsistent over time – people make choices today that their future selves would prefer not to have made, despite using the same reasoning

IKEA effect: The tendency for people to place a disproportionately high value on objects that they partially assembled themselves, such as furniture from IKEA, regardless of the quality of the end product

Illusory truth effect: A tendency to believe that a statement is true if it is easier to process, or if it has been stated multiple times, regardless of its actual veracity.

Irrational escalation: The phenomenon where people justify increased investment in a decision, based on the cumulative prior investment, despite new evidence suggesting that the decision was probably wrong. Also known as the sunk cost fallacy

Law of the instrument: An over-reliance on a familiar tool or methods, ignoring or under-valuing alternative approaches. “If all you have is a hammer, everything looks like a nail”

Mere exposure effect: The tendency to express undue liking for things merely because of familiarity with them

Negativity bias: Psychological phenomenon by which humans have a greater recall of unpleasant memories compared with positive memories

Non-adaptive choice switching: After experiencing a bad outcome with a decision problem, the tendency to avoid the choice previously made when faced with the same decision problem again, even though the choice was optimal

Omission bias: The tendency to judge harmful actions (commissions) as worse, or less moral, than equally harmful inactions (omissions).

Ostrich effect: Ignoring an obvious (negative) situation

Plan continuation bias: Failure to recognize that the original plan of action is no longer appropriate for a changing situation or for a situation that is different than anticipated

Prevention bias: When investing money to protect against risks, decision makers perceive that a dollar spent on prevention buys more security than a dollar spent on timely detection and response, even when investing in either option is equally effective

Pseudocertainty effect: The tendency to make risk-averse choices if the expected outcome is positive, but make risk-seeking choices to avoid negative outcomes

Salience bias: The tendency to focus on items that are more prominent or emotionally striking and ignore those that are unremarkable, even though this difference is often irrelevant by objective standards

Selective perception bias: The tendency for expectations to affect perception

Status-quo bias: If no special action is taken, the default action that will happen is that the code will go live. You will need an especially compelling reason to override this bias and manually stop the code from going live, as it would by default.

Slow-motion bias: We feel certain that we are more careful and less risky when we slow down. This is precisely the opposite of the real world risk factors for shipping software. Slow is dangerous for software; speed is safety. The more frequently you ship code, the smaller the diffs you ship, the less dangerous each one actually becomes. This is the most powerful and difficult to overcome of all of our biases, because there is no readily available counter-metaphor for us to use. (Riding a bike is the best I’ve come up with. 😔)

Surrogation: Losing sight of the strategic construct that a measure is intended to represent, and subsequently acting as though the measure is the construct of interest

Time-saving bias: Underestimations of the time that could be saved (or lost) when increasing (or decreasing) from a relatively low speed and overestimations of the time that could be saved (or lost) when increasing (or decreasing) from a relatively high speed.

Zero-risk bias: Preference for reducing a small risk to zero over a greater reduction in a larger risk.

Software deploys and cognitive biases

Why every software engineering interview should include ops questions

I’ve fallen way behind on my blog posts — my goal was to write one per month, and I haven’t published anything since MAY. Egads. So here I am dipping into the drafts archives! This one was written in April of 2016, when I was noodling over my CraftConf 2016 talk on “DevOps for Developers (see slides).”

So I got to the part in my talk where I’m talking about how to interview and hire software engineers who aren’t going to burn the fucking house down, and realized I could spend a solid hour on that question alone. That’s why I decided to turn it into a blog post instead.

Stop telling ops people to code better, start telling SWEs to ops better

Our industry has gotten very good at pressing operations engineers to get better at writing code, writing tests, and software engineering in general these past few years. Which is great! But we have not been nearly so good at pushing software engineers to level up their systems skills. Which is unfortunate, because it is just as important.

Most systems suffer from the syndrome of running too much software. Tossing more software into the heap is as likely to cause more problems as often as it solves them.

We see this play out at companies stacked with good software engineers who have built horrifying spaghetti messes of their infrastructure, and then commence paging themselves to death.

The only way to unwind this is to reset expectations, and make it clear that

  1. you are still responsible for your code after it’s been deployed to production, and 
  2. operational excellence is everyone’s job.

Operations is the constellation of tools, practices, policies, habits, and docs around shipping value to users, and every single one of us needs to participate in order to do this swiftly and safely.

Every software engineering interviewing loop should have an ops component.

Nobody interviews candidates for SRE or ops nowadays without asking some coding questions. You don’t have to be the greatest programmer in the world, but you can’t be functionally illiterate. The reverse is less common: asking software engineers basic, stupid questions about the lifecycle of their code, instrumentation best practices, etc. 

It’s common practice at lots of companies now to have a software engineer in the loop for hiring SREs to evaluate their coding abilities. It should be just as common to have an ops engineer in the loop for a SWE hire, especially for any SWE who is being considered for a key senior position. Those are the people you most rely on to be mentors and role models for junior hires. All engineers should embrace the ethos of owning their code in production, and nobody should be promoted or hired into a senior role if they don’t.

And yes, that means all engineers!  Even your iOS/Android engineers and website developers should be interested in what happens to their code after they hit deploy.  They should care about things like instrumentation, and what kind of data they may need later to debug their problems, and how their features may impact other infrastructure components.

You need to balance out your software engineers with engineers who don’t react to every problem by writing more code. You need engineers who write code begrudgingly, as a last resort. You’ll find these priceless gems in ops and SRE.

ops questions for software engineers

The best questions are broad and start off easy, with plenty of reasonable answers and pathways to explore. Even beginners can give a reasonable answer, while experts can go on talking for hours.

For example: give them the specs for a new feature, and ask them to talk through the infrastructure choices and dependencies to support that feature. Do they ask about things like which languages, databases, and frameworks are already supported by the team? Do they understand what kind of monitoring and observability tools to use, do they ask about local instrumentation best practices?

Or design a full deployment pipeline together. Probe what they know about generating artifacts, versioning, rollbacks, branching vs master, canarying, rolling restarts, green/blue deploys, etc. How might they design a deploy tool? Talk through the tradeoffs.

Some other good starting points:

  • “Tell me about the last time you caused a production outage. What happened, how did you find out, how was it resolved, and what did you learn?”
  • “What are some of your favorite tools for visibility, instrumentation, and debugging?
  • “Latency seems to have doubled over the last 6 hours. Where do you start looking, how do you start debugging?”
  • And this chestnut: “What happens when you type ‘google.com’ into a web browser?” You would be fucking *astonished* how many senior software engineers don’t know a thing about DNS, HTTP, SSL/TLS, cookies, TCP/IP, routing, load balancers, web servers, proxies, and on and on.

Another question I really like is: “what’s your favorite API (or database, or language) and why?” followed up by “… and what are the worst things about it?” (True love doesn’t mean blind worship.)

Remember, you’re exploring someone’s experience and depth here, not giving them a pass-fail quiz. It’s okay if they don’t know it all. You’re also evaluating them on communication skills, which is severely underrated by most people but is actually as a key technical skill.

Signals to look for

You’re not looking for perfection. You are teasing out signals for things like, how will this person perform on a team where software engineers are expected to own their code? How much do they know about the world outside the code they write themselves? Are they curious, eager, and willing to learn, or fearful, incurious and begrudging?

Do they expect networks to be reliable? Do they expect databases to respond, retries to succeed? Are they offended by the idea of being on call? Are they overly clever or do they look to simplify? (God, I hate clever software engineers 🙃.)

It’s valuable to get a feel for an engineer’s operational chops, but let’s be clear, you’re doing this for one big reason: to set expectations. By making ops questions part of the interview, you’re establishing from the start that you run an org where operations is valued, where ownership is non-optional. This is not an ivory tower where software engineers can merrily git push and go home for the day and let other people handle the fallout

It can be toxic when you have an engineer who thinks all ops work is toil and operations engineering is lesser-than. It tends to result in operations work being done very poorly. This is your best chance to let those people self-select out.

You know what, I’m actually feeling uncharacteristically optimistic right now. I’m remembering how controversial some of this stuff was when I first wrote it, five years ago in 2016. Nowadays it just sounds obvious. Like table stakes.

Hell yeah. 🤘

Why every software engineering interview should include ops questions

How Much Should My Observability Stack Cost?

First posted on 2021-08-18 at https://www.honeycomb.io/blog/how-much-should-my-observability-stack-cost

What should one pay for observability? What should your observability stack cost? What should be in your observability stack?

How much observability is enough? How much is too much, or is there such a thing?

Is it better to pay for one product that claims (dubiously) to do everything, or twenty products that are each optimized to do a different part of the problem super well?

It’s almost enough to make a busy engineer say “Screw it, I’m spinning up Nagios”.

(Hey, I said almost.)

All of these service providers can give you sticker shock when you begin investigating them. The biggest reason is always that we aren’t used to considering the price of our own time.  We act like it’s “free” to just take an hour and spin something up … we don’t count the cost of maintenance, context switching, and opportunity costs of not using the time to build something of business value.  Which is both understandable and forgivable, as a starting point.

Considerably less forgivable is the vagueness–and sometimes outright misdirection and scare tactics–some vendors offer around pricing. It’s not ok for a business to optimize for revenue at the expense of user experience. As users, we have the right to demand transparency and accurate information.  As vendors, we have the responsibility to provide it.  Any pricing scheme that doesn’t align with best practices and users’ interests will be a drag on reputation and growth.

The core question, rarely addressed outright, is: how much should you pay? In this post I’ll talk about what your observability costs include, and in the next post, what you should consider including in your “observability stack”.

But I’ll give you the answer to your question right off the bat: you should probably spend 20-30% of infra costs on observability.

O11y spend should be 20-30% of infra spend

Rule of thumb: your observability spend should come to 20-30% of your infra spend. (I’ve seen 10% a few times from reasonable-seeming shops, but they have been edge cases and outliers. I have also seen 50% or more, but again, outliers.)

Full disclosure: this isn’t based on any particular science.  It’s just based on my experience of 15+ years working in operations engineering, talking to other engineers and managers, and a couple of informal Twitter polls to satisfy my own curiosity.

Nevertheless, it’s a pretty solid rule. There are exceptions, but in general, if you’re spending less than 20%, you’re “saving money” at the expense of engineering time, or being silently dragged underwater by a million little time leaks and quality of service issues — which you could eliminate completely with a bit of investment.

Consider the person who told me proudly that his o11y spend was just 1-3%. (He meant the PagerDuty bill and Pingdom checks, actually.) He wasn’t counting the dedicated hardware for their ELK cluster (80k/month), or the 2-3 extra engineers they had to recruit, train and hire (250-300k/year apiece) to run the many open source tools they got for “free”.

And ultimately, it didn’t meet their needs very well. Few people knew how to use it, so they leaned on the “observability team” to craft custom views, write scripts and ETL one-offs, and serve as the institutional hive mind and software usability tutors.  They could have used better tools, ones under active development by large product teams.  They could have used that headcount to create core business value instead.

Engineers cost money

Engineers are expensive. Recruiting them is hard. The good ones are increasingly unwilling to waste time on unnecessary labor. This manager was “saving” maybe a million dollars a year (he mentioned a vendor quote of less than 100k/month)–but spending a couple million more than that in less-visible ways.

Worse, he was driving his engineering org into the ground by wasting so much of their time and energy on non-mission-critical work, inferior tooling, one-offs, frustrating maintenance work, etc, all of which had nothing to do with their core business value.

If you want to know if an org hires and retains good engineers, you could do worse than to ask the question: “What tools do you use, and why?”

  • Good orgs use good tools. They know engineering cycles are their scarcest and most valuable resource, and they want to train maximum firepower on their core business problems.
  • Mediocre orgs use mediocre tools, have no discipline or consistency around adoption and deprecation, and leak lost engineering cycles everywhere.

So back to our rule of thumb: observability amounting to 20-30% of total spend is where most shops should fall. This refers to cloud-native infrastructure, using third-party services to instrument and monitor code, with the basics covered — resource utilization graphs, end to end checks, paging, etc.

So, what do I need in my “observability stack”?

What are the basics? Well, obviously “it depends”. It depends on your requirements, your components, your commitments, your budget, sunk costs and skill sets, your teams, and most expensive of all — customer expectations and the cost of violating them. You should think carefully about these things and try to draw a straight line from the business case to the money you spend (or don’t spend). And don’t forget to factor in those invisible human costs.

 

How Much Should My Observability Stack Cost?

Notes on the Perfidy of Dashboards

The other day I said this on twitter —

… which stirred up some Feelings for many people. 🙃  So I would like to explain my opinions in more detail.

Static vs dynamic dashboards

First, let’s define the term. When I say “dashboard”, I mean STATIC dashboards, i.e. collections of metrics-based graphs that you cannot click on to dive deeper or break down or pivot. If your dashboard supports this sort of responsive querying and exploration, where you can click on any graph to drill down and slice and dice the data arbitrarily, then breathe easy — that’s not what I’m talking about. Those are great. (I don’t really consider them dashboards, but I have heard a few people refer to them as “dynamic dashboards”.)

Actually, I’m not even “against” static dashboards. Every company has them, including Honeycomb. They’re great for getting a high level sense of system functioning, and tracking important stats over long intervals. They are a good starting point for investigations. Every company should have a small, tractable number of these which are easily accessible and shared by everyone.

Debugging with dashboards: it’s a trap

What dashboards are NOT good at is debugging, or understanding or describing novel system states.

I can hear some of you now: “But I’ve debugged countless super-hard unknown problems using only static dashboards!” Yes, I’m sure you have. If all you have is a hammer, you CAN use it to drive screws into the wall, but that doesn’t mean it’s the best tool. And It takes an extraordinary amount of knowledge and experience to be able to piece together a narrative that translates low-level system statistics into bugs in your software and back. Most software engineers don’t have that kind of systems experience or intuition…and they shouldn’t have to.

Why are dashboards bad for debugging? Think of it this way: every dashboard is an answer to a question someone asked at some point. Your monitoring system is probably littered with dashboards, thousands and thousands of them, most of whose questions have been long forgotten and many of whose source data streams have long since gone silent.

So you come along trying to investigate something, and what do you do? You start skimming through dashboards, eyes scanning furiously, looking for visual patterns — e.g. any spikes that happened around the same time as your incident. That’s not debugging, that’s pattern-matching. That’s … eyeball racing.

if we did math like we do dashboards

Imagine you’re in a math competition, and you get handed a problem to solve. But instead of pulling out your pencil and solving the equation, step by step, you start hollering out guesses.

“27!”
“19992.41!”
“1/4325!”

That’s what flipping through dashboards feels like to me. You’re riffling through a bunch of graphs that were relevant to some long-ago situation, without context or history, without showing their work. Sometimes you’ll spot the exact scenario, and — huzzah! — the number you shout is correct! But when it comes to unknown scenarios, the odds are not in your favor.

Debugging looks and feels very different from flipping through answers. You ask a question, examine the answer, and ask another question based on the result. (“Which endpoints were erroring? Are all of the requests erroring, or only some? What did they have in common?”, etc.)

You methodically put one foot in front of the other, following the trail of bread crumbs, until the data itself leads you to the answer.

The limitations of metrics and dashboards

Unfortunately, you cannot do that with metrics-based dashboards, because you stripped away the connective tissue of the event back when you wrote the metrics out to disk.

If you happened to notice while skimming through dashboards that your 404 errors spiked at 14:03, and your /payment and /import endpoints started erroring at 14.03, and your database started returning a bunch of mysql errors shortly after 14:00, you’ll probably assume that they’re all related and leap to find more evidence that confirms it.

But you cannot actually confirm that those events are the same ones, not with your metrics dashboards. You cannot drill down from errors to endpoints to error strings; for that, you’d need a wide structured data blob per request. Those might in fact be two or three separate outages or anomalies happening at the same time, or just the tip of the iceberg of a much larger event, and your hasty assumptions might extend the outage for much longer than was necessary.

With metrics, you tend to find what you’re looking for. You have no way to correlate attributes between requests or ask “what are all of the dimensions these requests have in common?”, or to flip back and forth and look at the request as a trace. Dashboards can be fairly effective at surfacing the causes of problems you’ve seen before (raise your hand if you’ve ever been in an incident review where one of the follow up tasks was, “create a dashboard that will help us find this next time”), but they’re all but useless for novel problems, your unknown-unknowns.

Other complaints about dashboards:

They tend to have percentiles like 95th, 99th, 99.9th, 99.99th, etc. Which can cover over a multitude of sins. You really want a tool that allows you to see MAX and MIN, and heatmap distributions.

A lot of dashboards end up getting created that are overly specific to the incident you just had — naming specific hosts, etc — which just creates clutter and toil. This is how your dashboards become that graveyard of past outages.

The most useful approach to dashboards is to maintain a small set of them; cull regularly, and think of them as a list of starter queries for your investigations.

Fred Hebert has this analogy, which I like:

“I like to compare the dashboards to the big display in a hospital room: heartbeat, pressure, oxygenation, etc. Those can tell you when a thing is wrong, but the context around the patient chart (and the patient themselves) is what allows interpretation to be effective. If all we have is the display but none of the rest, we’re not getting anywhere close to an accurate picture. The risk with the dashboard is having the metrics but not seeing or knowing about the rest changing.”

In conclusion

Dashboards aren’t universally awful. The overuse of them just encourages sloppy thinking, and static ones make it impossible for you to follow the plot of an outage, or validate your hypotheses. 🤒  There’s too many of them, and not enough shared consensus. (It would help if, like, new dashboards expired within a month if nobody looked at them again.)

If what you have is “nothing”, even shitty dashboards are far better than no dashboards. But shitty dashboards have been the only game in town for far too long. We need more vendors to think about building for queryability, explorability, and the ability to follow a trail of breadcrumbs. Modern systems are going to demand more and more of this approach.

Nothing < Dashboards < a Queryable, Exploratory Interface

If everyone out there who slaps “observability” on their web page also felt the responsibility to add an observability-enabling interface to their tool, one that would let users explore and identify unknown-unknowns, we would all be in a far better place. 🙂

 

 

 

 

 

Notes on the Perfidy of Dashboards

Engineering Manager Archetypes and Career Paths

Cross-posted from leaddev.com

Exploring seven common leadership scenarios.

If you’re looking for a very traditional breakdown of engineering manager archetypes – tech lead manager, team manager, director, and so forth – you can’t do better than Will Larson’s post, and I won’t try. I enthusiastically endorse everything he says, especially about Tech Lead Management roles being mostly a trap.

This post aims to capture a different set of archetypes: the common inflection points in a senior engineering manager’s career trajectory, what contributes to these, and some recommended next steps.

I’m of the opinion that career ‘planning’ is mostly overrated. There are just so many variables that go into what makes you feel happy, challenged, and content over the course of your life, and trying to predict how you’ll feel more than two or three years down the line is a fool’s game. You might get married, have kids, deal with immigration status or work visas, or family or health issues that suddenly throw a wrench in things. The company you work at might get acquired, go public, make some bad hires, get pummeled by changing market conditions or an economic downturn, or reorg you under somebody you absolutely loathe. The plum role you want might go to someone else. You might hitch your career to a technology that takes off like a rocket, or sputters out quietly, or suffers a series of community scandals.

Everything fails sooner or later, and most fail sooner rather than later.

And then there’s the fact that people who do plan their careers tend to plan in terms of hierarchy. They are eager to ‘make management’. Then they’re hungry to manage managers. They read ‘30 under 30’ lists and obsessively track which of their peers has climbed the ladder to director or vice president first, and their heart swells in tune with their headcount growth.

Few of us are completely immune to the siren song of such external signifiers of success. (Everybody’s mom congratulates them when they move into management – this shit runs deep in our culture.) But for most of us, nothing breaks the spell like spending time in those upper-hierarchical roles.

When you’re an engineer, it seems like managers hold all the power. Two or three years into a managerial role, you should be thoroughly cured of that illusion – aware of the constraints managers operate under and the multiple stakeholders they serve, but also newly appreciative of the powers and freedoms held by engineers, and attuned to how wielding influence and marshaling results may have little if anything to do with formal decision-making powers.

So, if you’re an ambitious person who wants a long, happy, rewarding career in technology, what’s the alternative?

Self-knowledge. Never stop working on self-awareness, being honest with yourself and others. Try to understand what truly matters to you – is it status and public acclaim? Is it spending half your life in the wilderness? Do you have a singular passion for compilers? Whatever it is, expect it to shift over the years.

  1. Do set goals for yourself – know what you want to achieve over the next few years, and have a hazy idea of where your nose is pointing five years from now.
  2. Whenever you don’t feel strongly compelled to make a particular move, act to preserve or expand optionality.

Don’t get comfortable. Once you’ve been doing any job for two or three years, it’s time for a change.

Let’s examine some archetypes and what to do if one of these applies to you. Keep in mind that these are just rules of thumb, and rules are meant to be broken. If your heart is pulling you in one direction, by all means, follow it.

The junior engineer who became an early manager

If you have worked as an engineer for less than seven-ish years before becoming a manager, you’re in risky territory. You’ll lose your fluency very quickly, and find it much harder to go back to engineering than if you had waited until you were more solidly senior. This can feel like a big compliment – and it is! – but this is your long-term career we’re talking about, not theirs, and too often those who progress early in their career aren’t warned about the downsides and the loss of optionality.

You’ll probably be fine as long as you stay at that same company, given that you know the systems well and management is all about relationships, but you may struggle when you try to find your next job, and you may also struggle finding someone willing to hire you as an engineer when you’re rusty. Technical skills buy credibility.

It’s worth noting that this seems to disproportionately happen to women, who are tapped for management because they are perceived as having better social skills, and then suffer a double penalty when they are rusty and judged ‘not technical’. If you’re a young woman, be extra cautious – make sure you ardently guard and grow your technical credibility.

The novice manager who wants to go back to engineering, six months in

When you’re deciding whether to be a manager or not, think of it as a tour of duty that will last for two to three years. Can you commit to that? If not, this probably isn’t the right time for you to try it out. If you aren’t sure, or you don’t feel like you have any idea what the job entails, don’t accept. Find ways to level up at managerial skills without taking on full responsibility for a team.

This might include taking ownership over a hiring loop – defining the role, writing a posting, designing the interview loop, training the interviewers, screening the candidates, adopting an intern or two (or even running the intern program), updating or open-sourcing your engineering job ladder, subbing for a manager while they’re out on paternity leave for a few months – any number of things. It’s okay to say no, and far better than saying yes, then backing out.

Why? First and foremost, it’s really disruptive to the team, and not fair to them. Secondly, you need to be prepared to shelve your own opinion of your work for the next two years and just keep doggedly plugging away, doing your best to learn and taking loads of feedback, without worrying too much about whether you personally think you’re doing a good job or not. You can’t base your self-esteem on how well you think you’re doing, because frankly, you’re not to be trusted. You have to gain a lot of experience and rewire a lot of neural pathways before you can once again trust your own inner voice on the topic of your job performance.

Management isn’t a promotion, it’s a change of career. And if you aren’t up for resetting the dial to ‘NOOB’ for a couple years, with all of the ups and downs, anxieties and discombobulations, then you aren’t up for management.

The manager who wants to go back to engineering, but only temporarily

One thing I hear surprisingly often from people is that they’re anxious that they’ll never get another shot at management if they step away from it – to take a new job at a new company, for instance, or to solidify their senior engineering skills. But they like management! – and are afraid of getting ‘stuck’ back in engineering.

It’s true: there are no guarantees in life. But in my experience, this fear is particularly overblown. There is a chronic shortage of good engineering managers, especially ones who genuinely enjoy the work. It’s pretty obvious when someone has the engineering manager’s skill set in their tool box; it permeates how you do your job, how you interact with your coworkers, and how you get things done. This means that it’s only a matter of time before you get tapped again. (And again.)

Engineers who have been managers tend to bat the question away over and over again – job after job, year after year. It takes more effort not to be a manager once you’re capable of it.

The manager who was forced into it and wants out…has wanted out for years

This is a tricky one, because it comes in two different guises. Sometimes it’s the person who got pushed into management unwillingly, or had a really rough time of it for a while, or maybe they have never worked in an environment where management was spoken of respectfully. For whatever reason, they still openly grump and groan about it despite the fact that they have actually come around and quite like the job now, and don’t actually want to go back to engineering after all.

Other times, the person got pushed into management unwillingly and genuinely dislikes it. Perhaps they were better as an engineer than they are as a manager, or maybe they simply never grew to love the management role enough to compensate for what they miss about engineering. If they’ve been complaining about it for years, and haven’t done anything about it, there are a few possible reasons: 1) they are genuinely getting pressured by their own boss and the organization to stay in their current role 2) they can’t bring themselves to give up the salary, the control, or the perceived status 3) there’s some other fear holding them back – for example, maybe they’re afraid it’s been too long and they’re too rusty.

In the first case, you should stay a manager, by all means, but own up to the fact that you enjoy it and find it rewarding. Even if you didn’t initially choose the gig, you’re choosing it now, so shut the hell up about being forced into it. Nobody wants to report to a manager who doesn’t want to be there, or is doing the job begrudgingly. Your complaining is toxic, and makes it impossible for you to coach others to have a healthy relationship with their roles.

In the latter case, if you genuinely don’t enjoy your job, then get the hell out. If it’s been years, then you have all the data you need. Your boss has no place forcing you into doing something you hate. You’re never going to do as well at a job you don’t enjoy as one that you do, so ultimately it’s a career-limiting move to stay where you are. Think long and hard about the distorting role that hierarchy has played in your life, and find a place where you are respected and valued for doing what you love.

Personally, I think the worst reason of all for not wanting to give up management is because it means giving up control. How do you think everyone else on your team feels? The good news is that you are ideally positioned to go back to engineering and model what a healthy power balance should look like between management and engineering – to demand transparency and model autonomy and ownership as a senior engineer.

The manager who has been managing for several years … now what?

If you’re a line manager who has been doing the job for around five years, and you feel pretty confident in your craft … what’s next?

That’s a great question. As you know, your technical credibility is grounded to some extent in your ability to do the same work as your team. So one question to ask yourself is, how wobbly are you on that front? Some managers manage to keep a hand in (but out of the critical path) the whole time; most managers do not. There is no right or wrong answer here, only an honest assessment of where you’re at. If you’re rusty, and you want to keep doing line management, it’s probably time to go back to the well for at least six months work as an engineer.

(This could be a GREAT opportunity to swap places with someone who’s questioning whether or not to be a manager full time.)

That’s definitely the choice that preserves the most optionality for you career-wise. It ensures you will still be hireable as an engineer or an engineering manager anywhere, regardless of how much technical knowledge the interview requires (and it does run the gamut, up to and including places that make you work as an engineer for six months before assuming your managerial role).

But most managers are at least slightly curious about what it would be like to move up the chain, or expand their managerial repertoire. This might be: managing teams where you are not the technical domain expert, taking on more of a hybrid product manager role, managing multiple teams, spinning off one or more subteams and managing those managers in addition to your primary team, or managing managers as a director.

Here your choices will likely be constrained by opportunities, which means that the company you work for matters A LOT. At a company with hundreds of engineers and high growth, there’s usually a nonstop trickle of reorgs, reteaming, and other opportunities opening up. If you work at a smaller company (or a place with less growth) and your bosses show no interest in leaving soon, you may need to switch companies to seek out those opportunities.

If this is important to you, you may want to consider switching earlier rather than later. You should pay extra attention to managing ‘out’ and ‘up’; building close relationships with your peers and those above you, in other words. Don’t be afraid to open your mouth and state your goals, and ask for your manager’s advice in getting there. People don’t get randomly tapped for these opportunities so much as they share their goals and are then grown into those positions over time.

The engineer who wants to be a manager, but hasn’t had the opportunity

The most demystifying thing about management is actually being a manager. To those who haven’t had the chance yet – or worse, who have seen colleague after colleague tapped for opportunities while being repeatedly overlooked– that can be cold comfort, or even infuriating to hear. If you want a shot at management, and haven’t had one yet: why not? What should you do or try differently?

Every circumstance is different, but I can offer some suggestions.

First of all, my personal belief is that in an ideal world, any senior engineer who wants to give management a try (and who demonstrates sufficient self-awareness, emotional intelligence, and respect for their colleagues), should get a shot at it. I also believe managers and engineers should both have parallel career ladders, should make equal salaries across their bands, and that technical leadership should be the purview of technical contributors. Leadership is not synonymous with management, and the more we can drain the relationships of hierarchy, the better people will be equipped to find the work that they love and find most fulfilling.

I live in the real world, so I know that most places don’t operate this way (although I believe the number is growing).

I also think that engineers tend to systematically underestimate and undervalue their own power, and they fail to inhabit and flex the power they should have. This leaves a vacuum, so managers step in to fill by default.

All that said. First, look at your company. How many opportunities are there for new managers, really? Is it too small, or not growing fast enough, or do you work in a specialty niche? Most companies aren’t bristling with new opportunities; if this is important to you, you should go work for a company that is growing fast, and you should state your ambition from the outset – yes, to the recruiter or hiring manager in the interview process.

Second, make friends with your manager and the other managers, and eagerly absorb as many managerial tasks or responsibilities as you can. Be more concerned with learning the skills and gaining the experience than with the title itself.

Third, work on your communication skills, written and spoken. Consider doing some public speaking. Run a workshop for your coworkers in something you know well. Take good meeting notes, write great technical project docs.

Fourth, ask your manager (and any other trusted senior people you work with) what you’re missing. Make it clear that you want to hear honest feedback, even if it hurts. If there’s something holding you back that you don’t know about, and someone is willing to tell you, that’s an incredible gift.

Fifth, educate yourself about diversity and inclusion issues. Stand up for others on your team who are getting talked over or having their ideas stolen. Be a leader among your peers, be someone who is willing to do what is right despite the temporary social costs.

The senior manager, director or vice president who daydreams about being an engineer again

Very often, people who were in the right place at the right time and climbed the ladder rapidly, get to the top and realize they feel restless and empty inside – they miss the intrinsic stimulation of writing code, solving problems, delighting users. Very few of them have the courage to go back.

It’s a true fact: the higher you climb the ladder, the further removed you are from the work that brings most people intrinsic joy, that feels real. Some manage to find dopamine hits in their work as a director or a vice president, but the dose is usually fainter and always on a delay.

It’s also a true fact that you’d have to go from being atop the monkey pyramid to being just another individual contributor, slinging code in the mines with the rest of them. That’s a lot for an ego to take. And it’s rough to pick up the skills again if you’ve been ten years or more away from writing code on a daily basis. Lots of people who are in their 30s, 40s or 50s feel like they have lost the mental agility to do so and therefore it might not even be an option anymore.

I don’t have an enormous sample size, but what I do have suggests the above is bullshit.

Consider my friend Molly Stamos, who was a software engineer from 1997-2001, then rapidly climbed the corporate ladder working in product, as a director, then as a vice president. She actually joined Honeycomb as our vice president of customer success, but she was burnt, and admitted to herself shortly after that she deeply missed software engineering and that was where her heart still lay. She worked in support for a while and picked up tasks from the backlog, and formally joined the engineering team in 2020, almost 20 years after she stopped coding full time. She’s freaking great at it and says she’s never been happier.

Be like Molly. Life’s too short to be miserable at work.

And P.S. most of the prestige is in your head. If you go from being an exec to being a senior engineer, I guarantee you that lots of people, especially other engineers, will actually respect you tons more.

Engineering Manager Archetypes and Career Paths

Career Development For Engineering Managers

(Cross-posted from leaddev.com)

What comes to mind when you hear the words, ‘career development for managers’?

If you’re like me, it’s one of two things: either obsessively climbing the ladder or those awful mandatory ‘trainings’ that get hastily assembled. (Otherwise known as, ‘How can we have hired so many brilliant engineers, yet none of them seem to have any idea what appropriate professional behavior looks like – and how long till we get sued?!’)

No one would claim that we have things like engineering roles, levels, and career progression locked down, but it all looks quite mature and well-understood compared to the Wild Wild West that is engineering management. Engineering at least has some basic skills (data structures, algorithms etc.) that one can study and structure an interview around, whereas for managers those basic skills (communication, relationships) come from ‘being human’.

To be fair, you can (and should!) work very hard on being a better human. All of us should – our entire life! Skills like communication, empathy, discipline and self-mastery, emotional self-regulation, self-awareness, and self-acceptance are a huge part of being an effective leader.

But let’s say you know all that. You’re a manager, you were dropped into this gig with precious little guidance (most of it contradictory), and all you want (for heaven’s sake!)  is some straightforward advice on how to grow and become a better one.

Perfect. Let’s get started.

With great power comes the responsibility to understand power dynamics

‘I totally support women/LGBTQIA+/POC in tech, but come on, it’s not as bad as some of them keep going on about. It can’t be, or I would have seen it myself by now. I’ve been in tech for ten years, and we’ve always treated every woman just the same as the guys. Besides, you know, there are two sides to every story.’

Does that sound like you? If so, please don’t become anyone’s manager until you have done a lot of listening and learning.

It’s one thing to be unaware of power dynamics and structural inequities as an individual contributor, and another when you wield formal hiring and firing powers on behalf of the org. You can really mess people up that way, and it doesn’t matter if the harm came from your ignorance instead of malice.

Build a trusted peer group of other managers and talk regularly

Get yourself some peers. You need a personal circle of other managers for mutual support, swapping stories, and talking through tricky interpersonal scenarios with strict confidentiality. You need to know these people well enough, and they need to know you and the personalities in your life well enough, that when something is bothering you it won’t take an hour of preamble just to set the stage for the story you need to tell.

I enjoyed my peer circle as an engineer. I couldn’t survive without them as a manager.

It can take some time to amass the right trusted circle, and it will shift with time, as some connections drift away and new ones emerge. Building, nurturing, and curating your circle is a continuous background process.

And it’s the number one thing you can do for your managerial career.

Give as much as you have been given

If you went to university, don’t let it fool you: our profession is fundamentally a web of apprenticeships.

This is why you should be extra generous with your time; everything you learned came from someone else.

Teach, explain, and tell stories about your own successes and failures as a manager. Helping somebody else with their problem has a way of giving you fresh eyes for your own problems, and your skills become more firmly ingrained after you teach them to others.

Invest in your public speaking skills and broadcasting skills

If public speaking terrifies you, lean into that fear and crush it like a bug. Verbal fluency in front of a group is a learnable skill, just like any other. And as a manager, you will draw on this skill 20 times a day, whether it’s running efficient meetings, holding people’s attention, representing your team well, etc.

I used to be catastrophically terrified of speaking in front of people. Back in 2015, I had to give my first internal infrastructure talk in front of maybe eight people, and for weeks in advance I woke up almost every night, sick and clammy from having nightmares about it.

Personally, I have found that a combination of practice, experience, and sometimes using anti-anxiety medication has made this an easier journey for me.

Get a therapist

Get a therapist. Get a work coach too, if work will pay for it. Yes, it’s good to work on your self-awareness, your relationship patterns, etc., but it’s also super useful to learn new metaphors and descriptive phrases from people who do this for a living. The richer your descriptive arsenal, the wider your range when connecting with people.

If you’re reluctant to do this, ask yourself honestly: couldn’t you use some better coping mechanisms and tools for emotional regulation? (Couldn’t everyone?)

Such a huge part of being a manager is making sure you don’t turn your problems into other people’s problems, or make your feelings other people’s problems. And that’s hard. It’s hard, and the ground is constantly shifting beneath your feet. Just when you think you’ve got it down, things change again.

Do a tour of duty as a manager at a big company

I list this one somewhat begrudgingly, but it’s a big one. If you really want to accelerate your career as a manager and learn lots of core skills fast: consider going to work at a Google/Facebook/Amazon, or any large and relatively faster-paced (or extremely fast-growing) tech company.

I learned so, so, so much about management (and organizational politics, and bureaucracies, and formal and informal power structures) at Facebook.

None of it was what I wanted to learn, mind you, but in retrospect it turned out to be pretty darn invaluable.

These big tech companies have their flaws, but they aren’t stupid, and they have invested a lot more into sophisticated business operations than any startup. Startups get away with so, so, so much sloppy behavior from a management and human resources perspective.

Managing a team at a big company for a year is a killer crash course in the best practices and established defaults of engineering management. If you can’t or won’t go work for a big company, try Will Larson’s book An Elegant Puzzle, which outlines a lot of the same best practices and the reasons behind them.

Overcome your fear of confrontation

If you hate confrontation, I’m sorry honey, but you have to push through that barrier ASAP. It is the one thing that will hold you back, poison your relationships, result in loss of trust, and ultimately hurt your direct reports faster than anything else. Read Radical Candor, practice with your therapist, practice with your group of peers, do whatever it takes. You have got to learn to lean into that itchy feeling and open your mouth instead of shrinking away.

Experiment with your 1:1s

Learn to have better, more interesting, more fulfilling, more varied 1:1s with your reports. This is a tough one for introverts, but it’s super worth investing some time into. I will leave ‘how’ as an exercise for the reader – there are lots of interesting blog posts out there, or you could start picking the brains of other managers who you respect.

Most of us have never had a very good manager, let alone a great manager. If you dreaded your 1:1s or were bored by them, etc., it doesn’t have to be this way. 1:1s are the great bidirectional lens into each other’s roles and lives, and with a little curiosity and experimentation, they can be much more than 30 minutes of awkward conversation or the same 3 questions every week.

Run better meetings

Running great meetings – efficient, productive, interesting, with exactly the right people present – is an understated, underestimated art. We usually only notice when a meeting is being run poorly. Train yourself to notice meetings where everyone is leaning in, eyes sparkling, devices forgotten, or even just the meetings that you never dread going to: who runs those? Pay attention to those jedi masters, and copy the hell out of them.

Volunteer to do things that are new and challenging. (Nobody else knows how to do them, either)

Put your hand up and volunteer for things. Is there a new job ladder that needs to be written or revised? Does the on call schedule need refactoring? Has anyone checked to make sure salaries are fair and not biased by race or gender?

Who better than you? 😁

And finally, having saved the hardest and most open-ended for last:

Look for ways to improve and optimize your organization as a sociotechnical system

Anyone who calls themselves a technical leader, whether engineer or manager, shares responsibility for identifying, maintaining, and optimizing the inner feedback loops that help you build and ship software. They are at the heart of how your organization delivers value and becomes a place people want to work.

Figure out how high-performing your team is (start with the DORA metrics), then begin to evaluate how much time is being wasted and consumed by wheel-spinning due to inefficiencies in your core sociotechnical feedback loops.

Try to step back from fighting all the symptoms and pathologies; instead, try to address them at the source.

This may involve:

  • Staying up to date on technical news
  • Getting your hands dirty and tinkering from time to time
  • Talking to others in the org to identify shared pain
  • Learning what really motivates each person on your team
  • Crafting a pitch that excites them and pushes their boundaries
  • Breaking down massive technical debt into bite-sized initiatives
  • Knowing when to speak and when to shut up
  • Doling out strategic praise

In conclusion

Career development as a manager isn’t about racing up the org chart at a breakneck speed. What makes someone a great, fulfilled, happy engineering manager doesn’t necessarily translate into making them a great director or VP (or a happy one). Opportunities to advance to upper management are scarce and somewhat random. There’s a lot to learn about the craft of management and technical leadership, which is a much healthier thing to focus on.

Check in with yourself regularly about whether you still enjoy your work. Swinging the pendulum back to engineering every few years is a great way to ensure you remain employable and your skills remain fresh over the long run.

Building a long and rewarding career in tech is not about chasing titles or levels, it’s about learning to follow your curiosity (and learning to love the uncertain feeling of being out of your depth.)

One final warning: managers have a tendency to slip into a pattern of only doing things or growing in ways that are useful to the org, but useless to them personally (once they change jobs or leave the company). Don’t do this! Be more selfish. Insist on building skills that are transferable and inherently valuable, not just knowing where all the bodies are buried at your current employer.

Anyone who manages to be happy in management will have to seek out fresh dopamine hits to replace those they gave up in engineering. Happy hunting.

Career Development For Engineering Managers

How To Hire An Engineering Manager

Cross-posted from leaddev.com

Why it matters to look internally first.

So, you need an engineering manager. Maybe what you need is to replace a departing manager – someone who is leaving the company, being promoted, or going back to hands-on individual contributor (IC) work. Maybe your team has gotten too large and unwieldy and is ready to split in two, amoeba-like, or you’re on the verge of hiring a few more people and spinning off another team. Maybe you are a baby startup and this engineering manager will be your first formal structure, or maybe something else entirely.

Whatever the genesis, the point is the same: you need to find a manager. And your very first fork in the road will be, “Should I hire one from the outside, or tap one of my engineers for the role?”

Here is a brief overview of the considerations involved in asking this question and making that decision.

First of all, I should confess to my biases. I am a big believer in the philosophy that management is not a promotion, but a change of career. There is a great deal of evidence that hierarchy produces worse outcomes in creative industries, of which software is decidedly one. I think that management is not and should not be synonymous with leadership; in fact, there are areas of technical leadership and implementation details where managers have no business exerting ownership. There should be technical leadership levels that track management levels as their equals in compensation, authority, and respect, but with different domains – ‘non-overlapping magisteria’, as the great Stephen Jay Gould might have said.

Engineering managers are ultimately responsible for the health of the team, the career progression of the engineers under their care, and for making sure that engineering teams are focused on the right priorities. They are the essential link between business and engineering; they must be able to translate between the two worlds many times a day. But once the decision is made that something is a priority, how it is done is the proper domain of engineers and technical leads.

Now, back to our choice. What should we do – hire from within, or hire from without?

Give preference to internal candidates

I am a firm believer that you always, always owe your existing team first crack at any and every opportunity that arises from within. It may not always be possible; you may need someone with more experience or a different skillset than you have on your team, but you owe it to your team to seriously explore the possibility, every time, before you give in and begin to explore outside possibilities.

That is an ethical stance which I happen to hold. But there are also pragmatic reasons for preferring internal candidates over external candidates where possible. Effective line management requires a manager to have a reasonably good understanding of the tech stack and the slate of current problems and tradeoffs; who better than one of the engineers working on it? And good management, in general, is all about relationships, relationships, relationships; someone coming in the door will always take longer to build those relationships and be effective than someone who already knows the lay of the land and the people who dwell in it. Internal candidates taking a lateral transfer to a management role are the most likely to succeed, in my experience.

So, what are some questions you might ask yourself when looking around for management candidates?

Who is the best engineer?

This is traditionally the first (and sometimes only) question leadership asks before ‘promoting’ (sic) said engineer to management. Unfortunately, this is the wrong question to ask yourself. There is no correlation that I’ve observed between someone’s engineering skills and management skills – no positive correlation, at any rate.

I do believe engineering managers need to have a strong technical background in order to be effective, but they don’t need to be the best engineer in the room. Many fervently mediocre senior engineers have developed into strong, admirable engineering managers – some of whom then swing the pendulum back to IC and leverage those hybrid skillsets to become supremely valuable senior technologists.

This is one big reason why it’s important to decouple leadership from management. Your best engineers are likely hungry for advancement, autonomy, and impact (just like everyone else), and you do not want them to conclude that management is the only vehicle for their ambitions – the only way they’ll ever get a seat in the room where it happens. You want them to solve your hardest technical problems if that is where their heart is. A better question to start with might be:

Who is senior enough to be a manager?

Don’t consider anyone for management until they have achieved a solid senior level and status – at least 5–7 years of hands-on engineering. Their technical judgment should be well-respected by their peers; they should know enough to know how much they don’t know, and where to trust their own opinion or not.

You won’t be doing them any favors by asking them to switch careers earlier in their tenure, and you might well be inflicting permanent damage on their career. At least half of those who try management end up not liking it, or go back to engineering at some point. If they make the leap to management too early, it will be extremely difficult for them to return to engineering a couple of years later. You might be sentencing them to a one-way trip whether they like it or not.

It will also be difficult for them to effectively manage junior engineers, who need mentorship and career development, or senior engineers, whose experience and judgment will far outstrip their own. Too many managers end up frustrating their teams by lurching between command-and-control leadership, and blind trust and abdication of leadership – not because they want to, but because they lack the technical subtlety to know when to speak or shut up.

Are any of your engineers interested?

Is anyone on your team interested in trying out the manager gig? Don’t guess; ask. Not just once, but regularly. And don’t wait until a position opens up to ask, either. Managers should have a chat about career plans with everyone on their team at least once or twice a year, and that includes asking whether they would like to try being a manager at some point.

Management is just a bundle of skills and practices, after all. You don’t have to have a formal manager title in order to be tasked with or practice things like running an interview loop, mentoring an intern, running team meetings, and writing up plans. If someone wants to become a manager at some point in their career, you should help them find opportunities to practice some of these skills and habits.

This is the quickest and easiest way for someone to find out if the work actually doesn’t interest them. And it’s a great low-risk way for you to watch them in action and observe their skills, without having to entrust the fate of an entire team to them yet.

When an opening does arise, you should already have a mental map of who on the team is likely to be interested, and how skilled they are.

Do I need to interview them, or what?

If you’re seriously considering an internal candidate for management, start by having a conversation with them – a different sort of conversation than you may have had up to this point. Start with some simple, open-ended questions, like:

“Why do you think there are so many more men than women in software roles?”

And let them talk for a while. You aren’t trying to trap them, or looking for ideological purity, but if you are going to put someone into a position of formal organizational power over others, it’s your responsibility to make sure they won’t inflict harm.

You have to have a higher bar for managers when it comes to awareness of structural racism and interpersonal power dynamics. Being a ‘really good person’ is not enough to do the job; ‘good people’ inflict immense harm every day without meaning to. You need managers who are humble, self-aware, and eager to learn – not ones who will have a meltdown and become the problem any time they mess up.

What if there are two or more who want it?

This is another great argument for treating management as a change of career, not a promotion, and for legitimizing and normalizing the idea that it’s okay to go back and forth between management and engineering, every few years.

If it’s not a promotion, then you don’t have any status to give up. If it’s an ordinary thing, there will be other opportunities. Not every opportunity exists at every company at every time, but the more you can do to lower the perceived scarcity and intensity, the more you will find people becoming managers or engineers for the right reasons rather than the wrong ones.

The team’s opinion matters more than yours

You should never inflict a manager on a team who doesn’t want to receive them in that role, no matter how qualified they seem on paper. If someone wants to be manager, your next move should be to quietly chat with each of the team members to see how they receive the idea, giving special weight to the opinions (and body language) of any underrepresented minorities on the team. Do they want to report to this person? If not, don’t push it – even a little.

Slight caveat: if the proposed candidate is a gender or racial minority, and if you suspect the team’s discomfort is related to their status, it’s worth digging in a little more to understand the situation and even possibly overrule some hesitation. As ever, trust your intuition but interrogate it, and use your best judgment.

Reasons to go ahead and open an external req

I said that internal candidates are my preference by default, but there remains a long, long list of reasons why you might decide to hire externally after all. A few of them include:

  • Your team is mostly junior-to-intermediate, and your senior folks are lukewarm on the idea of management (never push someone into it!)
  • All your existing managers are first-timers, and you really need some more experienced managers in the mix to help you formalize policies and processes as you grow
  • The team is split, and there is no consensus behind any internal candidate.

Listen to your own intuition, too: if you don’t have a great feeling about any of your options, take that seriously.

If you decide to hire a manager from the outside, the first thing you need to do is sit down and take a long hard look in the mirror. Hiring isn’t about finding the ‘best person’, it’s about crafting the strongest team. Where is your team weak? Who do you worry about? What are your strategic goals for the upcoming year or two, and what kind of leadership experience will help you get there? Do you need someone who can be very hands-on with the technology, or someone skilled at systems building, or driving transformation? Will recruiting be make-or-break for you? How diverse is your team, and what are your diversity goals?

All of these archetypes call for different types of leaders. But that’s a topic for another time.

How To Hire An Engineering Manager

Questionable Advice: “What Should I Say In My Exit Interview?”

I recently received this gem of a note::

Hi Charity, I really enjoy your writing and a lot of it has directly contributed to me finally deciding to leave a company with a toxic management culture. I’ll also be leaving many great IC friends that will have lost a strong voice.

My exit interview will be next week. Any advice on how honest I should be?

I’ve googled quite a bit but there are only generic “don’t burn bridges” comments. Would love to see something a little more authoritative 🙂

–Anonymous Reader

Ew, fuck that. That’s exactly the kind of quivering, self-serving, ass covering advice you’d get from HR. It’s exactly the kind of advice that good people use to perpetuate harm.

I wouldn’t worry about “too much honesty” or personal repercussions or whatnot. I would worry about just one thing: being effective. This is your last chance to do the people you care about a solid, and you don’t want to waste it.

So … ranting about every awful person, boring project and offensive party theme of your tenure: not effective. Ranting about people who were personally irritating but had very limited power: not effective. Talking only in vague, high level abstractions (“toxic culture”), or about things only engineers understand and are bugged by: not effective.

What is effective? Hm, let’s think on this..

  • Start off with your high level assertion (toxic culture) and methodically assemble a list of stories, incidents, and consequences that support your thesis. Structure-wise, this is a lot like writing a good essay
  • Tie your critiques to the higher ups who enabled or encouraged the bad behavior, not just the flunkies who carried it out.
  • Wherever possible, draw a straight line to material consequences — people quitting, customers leaving, your company’s reputation suffering
  • Keep it crisp. No more than three pages total. Pick your top 1-3 points and drive them home. No detours.
  • This one sucks, but … if someone was perceived as an underperformer or a problem employee, avoid using them as evidence in support of your argument. It won’t help you or them, it will be used as an excuse to discredit you.
  • Keep it mostly professional. I am not saying don’t show any anger or strong emotion; it can be a powerful tool; just be careful with it. Get a proofread from someone with upper management experience, ideally with no connection to your work. (Me, if necessary.)
  • Put it in WRITING!✨ Deliver your feedback in person, but hand over a written copy as well. Written words are harder to ignore or distort.
  • For extra oomph, give a copy to any execs, managers, or high level ICs you trust. Don’t just email it to them, though. Have a face to face conversation where you state your case, and hand them a written copy at the end.

The sad fact is that most exit feedback is dutifully entered in by a low ranking employee who makes a third your salary and has no reason whatsoever to rock the boat, after which it gets tossed in a folder or the trash and is never seen again.

If you want to use your voice on your way out the door, the challenge you face isn’t one of retribution, it’s inertia and apathy. HR doesn’t care about your feedback … but they care if they think their boss saw it and cares about it

And I think you should use your voice! You clearly have some clout, and what’s the point of having power if you won’t extend yourself now and then on behalf of those who don’t?

Good luck!!

charity

Questionable Advice: “What Should I Say In My Exit Interview?”