On Call Shouldn’t Suck: A Guide For Managers

There are few engineering topics that provoke as much heated commentary as oncall. Everybody has a strong opinion. So let me say straight up that there are few if any absolutes when it comes to doing this well; context is everything. What’s appropriate for a startup may not suit a larger team. Rules are made to be broken.

That said, I do have some feelings on the matter. Especially when it comes to the compact between engineering and management. Which is simply this:

It is engineering’s responsibility to be on call and own their code. It is management’s responsibility to make sure that on call does not suck. This is a handshake, it goes both ways, and if you do not hold up your end they should quit and leave you.

As for engineers who write code for 24×7 highly available services, it is a core part of their job is to support those services in production. (There are plenty of software jobs that do not involve building highly available services, for those who are offended by this.) Tossing it off to ops after tests pass is nothing but a thinly veiled form of engineering classism, and you can’t build high-performing systems by breaking up your feedback loops this way.

Someone needs to be responsible for your services in the off-hours. This cannot be an afterthought; it should play a prominent role in your hiring, team structure, and compensation decisions from the very start. These are decisions that define who you are and what you value as a team.

Some advice on how to organize your on call efforts, in no particular order.

  • It is easier to keep yourself from falling into an operational pit of doom than it is to claw your way out of one. Make good operational hygiene a priority from the start. Value good, clean, high-level abstractions that allow you to delegate large swaths of your infrastructure and operational burden to third parties who can do it better than you — serverless, AWS, *aaS, etc. Don’t fall into the trap of disrespecting operations engineering labor, it’s the only thing that can save you.
     
  • Invest in good release and deploy tooling. Make this part of your engineering roadmap, not something you find in the couch cushions. Get code into production within minutes after merging, and watch how many of your nightmares melt away or never happen.
     
  • Invest in good instrumentation and observability. Impress upon your engineers that their job is not done when tests pass; it is not done until they have watched users using their code in production. Promote an ownership mentality over the full software life cycle. This is how dev.to did it.
     
  • Construct your feedback loops thoughtfully. Try to alert the person who made the broken change directly. Never send an alert to someone who isn’t fully equipped and empowered to fix it.
     
  • When an engineer is on call, they are not responsible for normal project work — period. That time is sacred and devoted to fixing things, building tooling, and creating guard-rails to protect people from themselves. If nothing is on fire, the engineer can take the opportunity to fix whatever has been annoying them. Allow for plenty of agency and following one’s curiosity, wherever it may lead, and it will be a special treat.
     
  • Closely track how often your team gets alerted. Take ANY out-of-hours-alert seriously, and prioritize the work to fix it. Night time pages are heart attacks, not diabetes.
     
  • Consider joining the on call rotation yourself! If nothing else, generously pinch hit and be an eager and enthusiastic backup on the regular.
     
  • Reliability work and technical debt are not secondary to product work. Budget them into your roadmap, right alongside your features and fixes. Don’t plan so tightly that you have no flex for the unexpected. Don’t be afraid to push back on product and don’t neglect to sell it to your own bosses. People’s lives are in your hands; this is what you get paid to do.
     
  • Consider making after-hours on call fully-elective. Why not? What is keeping you from it? Fix those things. This is how Intercom did it.
     
  • Depending on your stage and available resources, consider compensating for it. This doesn’t have to be cash, it could be a Friday off the week after every on call rotation. The more established and funded a company you are, the more likely you should do this in order to surface the right incentives up the org chart.
     
  • Once you’ve dug yourself out of firefighting mode, invest in SLOs (Service Level Objectives). SLOs and observability are the mature way to get out of reactive mode and plan your engineering work based on tradeoffs and user impact.
     

I believe it is thoroughly possible to construct an on call rotation that is 100% opt-in, a badge of pride and accomplishment, something that brings meaning and mastery to people’s engineering roles and ties them emotionally to their users. I believe that being on call is something that you can genuinely look forward to.

But every single company is a unique complex sociotechnical snowflake. Flipping the script on whether on call is a burden or a blessing will require a unique solution, crafted to meet your specific needs and drawing on your specific history. It will require tinkering. It will take maintenance.

Above all: ✨RAISE YOUR STANDARDS✨ for what you expect from yourselves. Your greatest enemy is how easily you accept the status quo, and then make up excuses for why it is necessarily this way. You can do better. I know you can.

There is lots and lots of prior art out there when it comes to making on call work for you, and you should research it deeply. Watch some talks, read some pieces, talk to some people. But then you’ll have to strike out on your own and try something. Cargo-culting someone else’s solution is always the wrong answer.

Any asshole can write some code; owning and tending complex systems for the long run is the hard part. How you choose to shoulder this burden will be a deep reflection of your values and who you are as a team.

And if your on call experience is mandatory and severely life-impacting, and if you don’t take this dead seriously and fix it ASAP? I hope your team will leave you, and go find a place that truly values their time and sleep.

 

On Call Shouldn’t Suck: A Guide For Managers

Things to know about engineering levels

This twitter thread seemed to strike a chord with people, rather astonishingly so. I am transcribing parts of it for the sake of longevity and findability.

I keep talking to engineers who are frustrated that they aren’t leveling up faster, aren’t reaching senior levels as quickly as other people they know, feel stuck and don’t know how to get to the next level, etc. And I’ve begun to notice a common blind spot around leveling —

✨ not every opportunity exists ✨
✨ at every company ✨
✨ at every time.✨

Sure, if you’re a junior engineer, you should be able to level up to intermediate pretty much anywhere. But it gets progressively trickier after that. Even the path from intermediate to senior can depend on a number of situational variables:

Is there oxygen?

  • How many other senior engineers do you work with? how many other intermediate engineers around your level? All of these people will be pulling from the same bin of work, looking for promo-worthy, solidly-senior projects.
  • Does your ladder explicitly call for mentorship or leading small teams of lower-leveled engineers? Are there enough of those folks to go around?
  • Have you sufficiently wrapped up your last project well enough to move on? Was it actually completed in a way that demonstrated clear mastery and readiness for bigger and harder work, or did you leave a mess behind you? That may limit people’s appetite to take a risk on you with mission-critical projects.
  • What are the biggest needs of the business right now? Any process that generates projects ought to begin with this question before proceeding on to carve out a chunk that fits your promo desires, not the other way around. 🙃
  • Do you happen to work in a niche or specialty area of engineering, particularly one crammed with super-senior, world-famous highly leveled people? This can be fantastic when it comes to your ability to soak up knowledge from the world’s best, but it may simultaneously delay your ability to level up.

In short, is there oxygen at the next level? Does the company need more of the type of engineer you want to be, vs more of the type of engineer you are now? If they need more people pounding out code and fewer architects, they’re unlikely to want to promote you to a role that involves mostly architecture..

Literally no company can possibly make use of a top-heavy eng org stuffed with senior+ engineers, if all of them are expected to demonstrate company-wide impact or global impact every review period. There’s only so much high-level work to go around for every fifty engineers writing code and features and executing on those systems.

There is only so much oxygen at each level.

Inflation.

Of course, this is all assuming that your company takes leveling seriously. Most … really … don’t.

It’s tough. It’s tough to hold your ground when a valued engineer is complaining and dropping hints they may leave if they don’t get that promotion soon. It’s much easier to give in, make an exception, argue for rounding up.

This may sound good, but it is not ultimately in your best interests as that engineer. Seriously. </3

There is sooo much title inflation in this industry already. People are given the title “senior engineer” in just 3-5 years, need I say more??

If you let a little inflation into your system by making exceptions, it causes more trouble than it’s worth. Always. The only leverage you have when people try to get you to make exceptions is if you can honestly say, “no exceptions.” Give in just once, and your moral authority evaporates.

Leveling up.

I would urge you not to make most, if any, career decisions based on levels or titles that are offered you. But I do understand how frustrating and infuriating it can be to be in a situation that is clearly unfair (usually because a manager got pressured into making an exception… tsk tsk), or if you don’t understand how to move your career forward.

So, here are a few strategic tips for leveling up.

  1. Generalists level up faster than specialists.
     
  2. When evaluating roles, choose ones where your specialty is part of their mission, or at least key to its execution. It has a far lower likelihood of getting outsourced, deprioritized, lacking investment in, or just forgotten about if what you do is core to what they do.
     
  3. Always ask to see the job ladder when interviewing. If they hedge or fumble, don’t take that job.
     
  4. Talk to your manager about the job ladder. Talk to your skip level about levels too! Managers love this shit. They can talk on and on and on about levels, long past your exhaustion point. It can be annoying, but it’s actually a sign of a good manager who cares and thinks about the edge cases in processes, and their impacts on people and teams.
     
  5. That said, don’t take the ladder as a checklist to memorize or thing to be pored over and obsessed over. It’s an incomplete attempt at both shaping and reflecting relative impact. Focus on impact.
     
  6. Is it easier to level up as a manager than as an engineer? Sorta-kinda, I guess so? There are at least two real phenomena at play here.
    1. There are simply more roles to go around in the management track. You need like, what, 1-2 E7/E8 (or principal, or architect?) level engineers per 100-500 engineers, but several managers/directors/etc
    2. Manager effectiveness is grounded in their relationships. It takes managers longer to have impact after they start a new role, but their potential impact grows and grows as their tenure gets longer. So yes, there’s a bit more of an escalator effect if you stay on the manager track at a company for several years. There is no similar escalator on the eng side; you have to be truly exceptional or truly lucky.
    3. But it really depends on the organization.

     

  7. It is much easier to level up quickly at fast-growing companies. When there is far more work than workers, and everyone is getting dropped in the deep end to sink or swim, you level up fast. Don’t underestimate what a stressful and awful experience this can be, though.
     
  8. Many engineers get stuck on the bubble getting to senior because they are impatient and want a map. They just want someone to *tell them what to do*. Which is the very opposite of what a senior engineer does. 🙃  Develop your judgment around what needs to be done, and do it.
     
  9. Your relationship with your manager matters. So does your ability to communicate about the work you are doing, its difficulty, its unexpected challenges and triumphs, etc. This is called “managing up”, and it is an actual skill which I am *terrible* at. So are most of you. 😉
     
  10. TLDR, if leveling matters to you (and it should matter to everyone, to some extent!), then look curiously and critically around for opportunities, and seek to maximize them. Want to become an E6/E7? Probably don’t join a startup that doesn’t have any very high-level work to do, or already has more than enough people functioning at those levels and many more nipping their heels looking for the same opportunity.
     
    This sort of thing is very obvious to us with the manager track (if you want to go from M->Dir, don’t join a startup that already HAS directors and managers who want to level up), but seems less obvious with engineering.

Most reasonable, non-desperate companies with options won’t hire you directly into the next level up which you haven’t done before, on either the manager or the engineer track. (Yellow flag if they do.)

But it is perfectly reasonable to express your career objectives in the interview, and make sure you’re on the same wavelength and seeing the same opportunities. Do you want to become a manager or a tech lead in a few months? Say so.

If it doesn’t exist now, do they think this opportunity may soon open up? Can they see a path forward for you there, if all goes well? Would they be interested in helping you get there? How many people may already be eyeing that same path? Is there enough opportunity for more than one? On what timeframe? Who will decide who gets the role, and how?

Engineers tend to find these conversations uncomfortable, and so they tend to avoid them because they don’t want to make the hiring manager uncomfortable by being pushy.

Relax. Managers don’t find this uncomfortable at all, it’s their bread and butter. (And even fi they do find it uncomfortable, tough beans.. it’s their job.) Ask away. ☺️

Misc notes on leveling.

P.S. Engineers seem to have a very sparse mental model of how leveling works, so here are a few more notes on how levels work at Honeycomb, which is adapted from conventions at Facebook/Google.

  • Each level after senior engineer (E5 for us) gets approx an order of magnitude harder to achieve, and an order of magnitude fewer engineers hold that title.
  • E5 is considered a “terminal level”, which sounds scary, but just means “you do not have to advance beyond this level.” If you never get promoted again, you won’t get fired either.
  • Whereas if you do not advance from E3-> E4 within 2 years, and E4->E5 within 3 years, you are automatically put on a performance improvement plan (at Facebook, I mean, not Honeycomb).
  • We (Honeycomb) hire into E5 as our highest level to start at, both because a) our interview process is not designed to let us parse differences between senior vs super-senior or super-duper senior, and b) we figure nobody is really able to come in the door with >E5 impact for the first 6 months anyway. So we can level them up quickly after they join and we get a feel for their work.

<3 charity.

Things to know about engineering levels

If Management Isn’t A Promotion, Then Engineering Isn’t A Demotion

I wrote a piece this week about what motivates people to become managers (tldr mostly org dysfunction), and Julian Dunn replied with some typically insightful tweets:

(I originally titled this article “Julian Dunn and the Case of the Bad Manager”, lol)

God, YES. This is something that has been on my queue of “topics to write about” for so long, and I haven’t because it’s just too big (and sometimes I tell myself, optimistically, it’s just too obvious?).

Julian’s point is that the reason so many bad managers persist is because it’s perceived as a promotion. Which means going back to engineering after managing is, ipso facto, a demotion. Which is really fucking hard to swallow. For anyone.

I touched on this briefly in an earlier post, the Pendulum or the Ladder, when I wrote,

“If management isn’t a promotion, then returning to hands-on work isn’t a demotion, either.  Right?”

There are a few separate points here which are worth unfurling separately.

  1. Management is widely seen as a promotion
  2. Management really does grant you some formal powers over your peers, which contributes to perceived hierarchy
  3. Humans are hierarchical mammals, exquisitely sensitive to any loss of status — we hates it
  4. But this is a cultural choice, not destiny. And we can change it.

Management is seen as a promotion

The notion that management is a promotion is deeply ingrained into our culture. It’s in language, pop culture, business books, any and all sources of career advice. If you became a manager and told your mom about it, she probably congratulated you and told you how proud she was. If you go out on a job interview, you’re expected to reach for the same rung or a higher one — or eyebrows will raise.

That’s a lot of cultural baggage to lean against. But I believe this is an idea whose time has come.

Any technical company should work hard to center and celebrate the work being done to build the product and make customers happy. Management is overhead, to be brutally frank about it, and we should not design organizations that would lead any rational, ambitious person to aspire to be overhead, should we?

The surest path to acclaim and glory (and promotions and raises) should be found through contributing. Not managing. Not being overhead.

… Because it mostly is a promotion, honestly

It is absolutely true that when you become a manager, you acquire new powers. As a tool of the org, you are granted certain powers to act on behalf of the organization, in exchange for being held accountable for certain outcomes.

These explicit powers often include hiring and firing decisions, access to privileged information, and making and meeting budgets.

But most of your powers aren’t formal at all. Most of your power comes from people listening more closely to what you say, giving your opinions more weight, and (consciously or subconsciously) just trying to please you, because they know you hold some influence over their career outcomes. It comes from the fact that so much information flows through managers. And finally, it comes from relationships — the strength of your personal relationships and mutual trust with other people throughout the org.

So how is this not a promotion? Well, it is a promotion at most companies, to be perfectly honest. But it does not have to be a promotion, if you acknowledge that these privileges and powers are accepted only by sacrificing other privileges and powers, and if you structurally allocate power to other roles. For example, you should acquire managerial powers only at the expense of technical decision-making powers.

I believe that the healthiest companies are ones where managerial powers are limited, enumerated, and minimal, with robust powers explicitly reserved for technical ICs. (much like the Constitution provides for Congress and the States, respectively.)

But it shouldn’t be. “Management” is a support role

Here are some of the reasons why we should invert the hierarchy and embrace management as a service role, a support position.

  1. Tech is a creative industry. Hierarchical leadership is a relic, a holdover from the days of manual labor. Hierarchy kills creativity, which leads to worse business outcomes.
  2. Bad managers are a huge problem in tech. Just like Julian says, the wrong people are doing the job, for the wrong reasons, because they can’t to take the hit to the ego (and paycheck) of the demotion. This leads to unhappy teams and ultimately loss of talent.
  3. I firmly believe that the engineer-manager pendulum is the way to build great technical leaders. The great line managers are never more than a few years removed from hands on work themselves, the great tech leads have always done a stint or two as a people manager. The promotion myth therefore both starves us of powerful technical leadership.and leaves us saddled with unhappy managers who have dwindling relevant skills, year after year.
  4. The ladder is a trap. There are an order of magnitude fewer jobs for each rung you ascend. Meanwhile, the higher you climb the farther removed you are from the work most find meaningful (building things, making customers happy). The perception that you are a failure if you do anything but climb higher therefore traps a great many people in a cycle of intense anxiety and unhappiness.
  5. Management is only one of many forms leadership can take. Yes, you have formal powers delegated to you on behalf of the org, but formal authority is the weakest form of power, and you should resort to using it rarely. Good leaders lead by influence and persuasion, weak leaders with “because I said so.”

Most engineers become managers to cope with org fuckery

Many people (like me!) become managers because they want access to the powers it gives them. As I argued in my last article, this is usually because they are frustrated with some organizational fuckery and it seems the only plausible way to fix or work around said fuckery is by becoming a manager.

Earlier this year I was having a 1×1 with one of our engineers, Martin Holman, who has been a manager before and had expressed interest in doing it again. So, I asked him, was he still interested?[1]

He thought for a moment, and replied, “You know, I thought I wanted to be a manager again, I really did. But I think what I actually wanted was a seat at the table — to know what was going on, to have a say in what work I do. But I don’t feel out of the loop here. So it turns out I don’t feel any need to become a manager.”

Not only did that warm my heart, it answered a question I didn’t know I had. I think they would be a good manager, and should they change their mind again in the future, I will completely support them changing their mind again (minds change! it’s what they do!) — but I hope it is never because they feel that technical contributors are left out of the loop, or don’t have a say in what they do.

That’s what I’d call organizational fuckery.

A roadmap for changing your company culture

If “management is not a promotion” is a cultural value you would like to embrace at your company, here are some concrete actions you should take.

  1. Make sure the pay bands for engineers and managers are equal, or even pay engineers more than managers of the same rank. (Slack does this, or used to.)
  2. Have IC (individual contributor) levels for engineers that track management levels, all the way up to VP.
  3. Look for ways to give high-level ICs information and opportunities for company impact that are on par with their people-manager counterparts.
  4. Technical contributors should own and be accountable for technical strategy and decision-making, not managers.
  5. Demystify management. Break it down into its constituent skills (giving feedback, running meetings, planning and budgeting, mentoring, running programs) and encourage everyone to develop those leadership skills.
  6. Offer any management roles that may open up to internal transfers before considering external candidates.
  7. Offer training and support for first-time managers who are undergoing that first career change. Offer engineers the same leadership coaching opportunities as managers.
  8. Explicitly encourage managers to swing back to IC roles after two or three years. Support them through a generous grace period while refreshing their technical skills.
  9. Watch your language. Loaded terms are everywhere, whether hierarchical (referring to people as being “above” others), or authoritarian (talking about bosses, managers). While it’s impossible to strip it from our vocabulary, it’s worth being thoughtful in how you represent reality, and using neutral phrases like “I support two teams” whenever possible.
  10. Be explicit; repeat yourself. Say over and over that management is not a promotion, it is a change of career. Say it internally and externally, in your interview processes and recruiting messages. Educate your recruiting staff too (and be stern about it).

This isn’t a thing you can do once and be done with it; it’s an ongoing effort you must commit to. Managers tend to accrue power over time, like a gravitational force. In order to counterbalance this drift, managers need to consciously push power out to others. They must use their role as “information router” to inform and empower people to own decisions, instead of hoarding it for themselves.

“Management is not a promotion” is my favorite bat signal

“Management is not a promotion, it’s a change of career.” I say this over and over again, even though it’s more aspirational than accurate.

Yet I say it anyway, because it’s a bat signal. It’s how the people I want to work with can find their way to me. And it repels the people I don’t want to work with just as efficiently.

When we recently posted our first-ever job req for an engineering manager, I included this under the list of optional skills:

  • You have worked as an engineering director or higher before, and decided to return to line management. Why? Because we value people who don’t blindly climb hierarchies just because they’re there. We value people who know themselves and what they find fulfilling in work and in life, and who can handle the hit to the ego that it takes to move “down” in pursuit of that fulfillment. Also, it would be interesting to talk about how you have solved org problems at other companies.

I cannot tell you how many amazing candidates zeroed in on that paragraph and came running. People who had been VPs before, been CTO, been director. People who were not only interested in becoming a line manager again, but were hungry to go back, to be closer to the people doing the work.

Something I heard them say again and again was, “People look at me like I’m crazy for wanting this,” “I have never had anyone see this as a strength.”

These were candidates who were acutely attuned to power dynamics, had exceptional self-knowledge, and who had seen and done so much to make organizations successful at multiple levels. What a set of superpowers!

Humans HAAAAAATE losing status.

We hate it. We hate it so bad. Even when we tell ourselves it’s what we wanted, even when we know it’s best for us, even when all the stars align. Something inside of us kicks and screams and feels excruciatingly attuned to the ripple effects of any status loss for a long time.

Like all such powerful irrational feelings, it’s evolution’s fault. Once upon a time it helped us survive and procreate. Now it’s just a nuisance, something to be worked around and minimized.

Where someone sits on the org chart should not determine that person’s ability to drive change, nor should their preference for tech problems or people problems. We need to see the work that engineers, managers, directors, VPs, and CxOs do as equally valuable and equally capable of prestige. We need to flip the org chart upside down, and treat “management” roles like the support systems they should be.

The work done by a database engineer is different from the work done by a VP marketing, or a director of database engineering. It is not inherently better or worse, easier or harder, more or less deserving of praise and admiration. It is simply different.[2]

And we will have the best chance finding the work that brings the most meaning and joy to our lives if we can drain the hierarchical residue out of our perception of these roles, by flattening pay structures, equalizing power dynamics, and making sure everyone has the tools they need to do their job with as little hierarchical bullshit as possible.

charity.

[1] Martin said I could tell this story and use his name. I actually try to avoid talking about people, conversations, or anecdotes from Honeycomb as a more or less blanket rule, because I don’t want people to be perpetually on edge wondering if I am talking about them. (So if you’re wondering if I’m talking about you: I’m not. Unless I asked first.)

[2] Raise your hand if you’ve worked at a company where a DB engineer had a far greater impact on the bottom line some quarters than any of the VPs did. ✋

If Management Isn’t A Promotion, Then Engineering Isn’t A Demotion

Questionable Advice: War Rooms? Really?!?

My company has recently begun pushing for us to build and staff out what I can only describe as “command centers”. They’re picturing graphs, dashboards…people sitting around watching their monitors all day just to find out which apps or teams are having issues. With your experience in monitoring and observability, and your opinions on teams supporting their own applications…do you think this sounds like a bad idea? What are things to watch out for, or some ways this might all go sideways?

— Anonymous

Jesus motherfucking Christ on a stick. Is it 1995 where you work? That’s the only way I can try and read this plan like it makes sense.

It’s a giant waste of money and no, it won’t work. This path leads into a death spiral where alarms are going off constantly (yet somehow never actually catching the real problems), people getting burned out, and anyone competent will either a) leave or b) refuse to be on call. Sideways enough for you yet?

Snark aside, there are two foundational flaws with this plan.

1) watching graphs is pointless. You can automate that shit, remember?  ✨skillsComputers!✨ Furthermore, this whole monitoring-based approach will only ever help you find the known unknowns, the problems you already know to look for. But most of your actual problems will be unknown unknowns, the ones you don’t know about yet.

2) those people watching the graphs… When something goes wrong, what exactly can they do about it? The answer, unfortunately, is “not much”. The only people who can swiftly diagnose and fix complex systems issues are the people who build and maintain those systems, and those people are busy building and maintaining, not watching graphs.

That extra human layer is worse than useless; it is actively harmful. By insulating developers from the consequences of their actions, you are concealing from them the information they need to understand the consequences of their actions. You are interfering with the most basic of feedback loops and causing it to malfunction.

The best time to find a bug is as soon as possible after writing it, while it’s all fresh in your head. If you let it fester for days, weeks, or months, it will be exponentially more challenging to find and solve. And the best people to find those bugs are the people who wrote them

Helpful? Hope so. Good luck. And if they implement this anyway — leave. You deserve to work for a company that won’t waste your fucking time.

with love, charity.

selfie - 4

Questionable Advice: War Rooms? Really?!?

The Official, Authorized List Of Legitimate Reasons For Deciding to Become a Manager

“Why did you decide to become a manager?”

It’s a question that gets asked a lot, in job interviews, 1x1s, and plain old casual conversation. I ask this question a lot, and I am often frustrated (or bored) by the answers I hear back.

Most of them can be bucketed in one of three ways:

  1. The pious. “I just really, really love helping other people achieve their goals.”
  2. The pleasers. the ones who answer, then pause uncertainly: “Is that what you’re looking for?”
  3. The sheepish. “I probably shouldn’t say this, but..” (followed by something very close to real honesty)

People are rarely inclined to divulge the range and depth of their reasons for going into management. And why should they? We are constantly being lectured about what the RIGHT reasons for going into management are, with aspersions cast upon anyone who dares enter the profession for any reasons that are not completely selfless.

“I LOVE mentoring.” “I wanted to protect my team.” “I’m motivated by people problems.” “I just really love helping people grow.”

Okay.

I’m not saying that everybody who says these words is lying, but I would be surprised if it was the entire story. People make career moves for a complex mix of altruism and self-interest.

It’s socially acceptable to cop to the selfless reasons. But what about the rest? Like “I wanted more money”? “I wanted career progression and couldn’t get any as an IC”? What about “I couldn’t get a seat at the table as an engineer”, “I was tired of being left out of important decisions”, or “My reporting chain was opaque and kept fucking up, and I figured I couldn’t do any worse than those bozos”?

Now we’re talking.

Most people become managers to compensate for org fuckery.

In my experience, most engineers become managers primarily due to organizational dysfunction. When you become a manager you acquire certain institutional powers, and you can use those powers to change the thing that makes you miserable.

It’s a hack. A gnarly one. And like most hacks, it kinda works.

For example, say it pisses you off to be left out of decisions. So you become a manager, and then you can either a) use your power and access to push for including engineers in the decision-making process, or at very least b) you personally will no longer left out.

In a healthy org, I would argue that most of these reasons should not exist. You should not have to become a manager to have career progression, pay equity, access to information, to be included in the decision-making process, even to set company strategy (to an extent congruent with your level, impact, role, tenure, etc)..

Everybody can’t weigh in on everything, obviously, but technical leaders are the best people to make technical decisions, not managers. In healthy orgs, managers work to push those powers outwards to the people closest to the work rather than hoarding it for themselves.

Legitimate reasons for being interested in management.

If you claw away all the org fuckery that forces so many people who care deeply about their work and coworkers into management, there is only one honest reason left for why anyone should try management.

✨Because you feel like it.✨

Because you’re curious. Because there’s an opportunity, maybe, or it seems interesting. Because why not? It’s as good a reason as any. Why do you learn a new framework, a new language, why do you write about your work, why do you pick up any new skill or new role? Why do any of it?

We are not rational beings. First comes emotional urge (“I want that”), then comes rationalization (“because, uh, I love people?”). That’s just how our brains work. You don’t really have to defend or justify it any further.

In reality …

I have observed that many people (especially early-career) are semi-obsessed with getting in to management.

There are many reasons for this. In most places, it is still regarded as a promotion, not a support role / change of career. With high achievers, all you have to do is plunk a ladder next to them to make them want to climb it. Many people feel a lack of agency and lack of autonomy in their role, and they think becoming a manager will solve all their problems.

The swiftest cure for this delusion is  … actually becoming a manager.

Management is a role where you are granted certain institutional powers, at the expense of other powers, freedoms and benefits. Many people who try management figure out pretty quickly that it’s not for them. Formal powers are, in many ways, the weakest powers of them all.

Which is why I think anybody who is interested in management should get a shot at it. Let’s demystify the role, strip it of its mystique and glamour, and make it what it should be: a role of service to others not dominance over others; staffed by people who genuinely take joy in that people side of sociotechnical problem solving.

 

charity

bed - 13 (1)

 

The Official, Authorized List Of Legitimate Reasons For Deciding to Become a Manager

Questionable Advice: “What’s the critical path?”

Dan Golant asked a great question today: “Any advice/reading on how to establish a team’s critical path?”

I repeated back: “establish a critical path?” and he clarified:

Yea, like, you talk about buttoning up your “critical path”, making sure it’s well-monitored etc. I think that the right first step to really improving Observability is establishing what business processes *must* happen, what our “critical paths” are. I’m trying to figure out whether there are particularly good questions to ask that can help us document what these paths are for my team/group in Eng.

“Critical path” is one of those phrases that I think I probably use a lot. Possibly because the very first real job I ever had was when I took a break from college and worked at criticalpath.net (“we handle the world’s email”) — and by “work” I mean, “lived in SF for a year when I was 18 and went to a lot of raves and did a lot of drugs with people way cooler than me”. Then I went back to college, the dotcom boom crashed, and the CP CFO and CEO actually went to jail for cooking the books, becoming the only tech execs I am aware of who actually went to jail.

Where was I.

Right, critical path. What I said to Dan is this: “What makes you money?”

Like, if you could only deploy three end-to-end checks that would perform entire operations on your site and ensure they work at all times, what would they be? what would they do? “Submit a payment” is a super common one; another is new user signups.servicelevel

The idea here is to draw up a list of the things that are absolutely worth waking someone up to fix immediately, night or day, rain or shine. That list should be as compact and well-defined as possible. This allows you to be explicit about the fact that anything else can wait til morning, or some other less-demanding service level agreement.

And typically the right place to start on this list is by asking yourselves: “what makes us money?” as a proxy for the real questions, which are: “what actions allow us to survive as a business? What do our customers care the absolute most about? What makes us us?” That’s your critical path.

Someone will usually seize this opportunity to argue that absolutely any deterioration in service is worth paging someone immediately to fix it, day or night. They are wrong, but it’s good to flush these assumptions out and have this argument kindly out in the open.

(Also, this is really a question about service level objectives. So if you’re asking yourself about the critical path, you should probably consider buying Alex Hidalgo’s book on SLOs, and you may want to look into the Honeycomb SLO product, the only one in the industry that actually implements SLOs as the Google SRE book defines them (thanks Liz!) and lets you jump straight from “what are our customers experiencing?” to “WHY are they experiencing it”, without bouncing awkwardly from aggregate metrics to logs and back and just … hoping … the spikes line up according to your visual approximations.)

charity.
Questionable Advice: “What’s the critical path?”

How to make boba at home…without ruining any pans, making yourself ill, or ending up with a soggy, blobby mess

Last year I was diagnosed with ADHD, which was a great surprise to me (if no one else). Since then I have been trying to pay attention to things I do that might be, let’s say, outside the norm. One of those things is, apparently, food.

I tend to fixate on one food at a time. When I wake up in the morning, it’s the first and only thing I crave. When I’m hungry, I’m dying for it, and I don’t really experience cravings or desire for other foods, although I will eat them to be polite. The phase tends to last for…six months to two years? and then it shifts to something else.

The target of my appetite has been, at various times in the past: honeycrisp apples with peanut butter (I was DEVASTATED when honeycrisp season ended; other apples weren’t the same), dry cheerios with freeze-dried strawberries, chopped broccoli with sharp cheddar, a cashew chicken dish at a now-defunct Thai restaurant, etc.

One year it was manhattans (makers mark, sweet vermouth and bitters) and I seriously worried I was becoming an alcoholic. 🙈

But since September 19th, 2019, the only thing I have been interested in eating is … boba. Those little brown tapioca balls. I can rattle off to you the top boba places in every city I’ve visited since then (LA has some seriously adventurous ones). And when the world strapped in for quarantine, I was on the verge of panic. What to do??

I finally figured out how to make my own boba. This was NONTRIVIAL. It took the sacrifice of countless pans and far too many nights doubled over with nausea and stomach cramps (read my buying tips, I cannot this stress enough), and months of trial and error. But here is how to get the plump, chewy, slightly sweet boba of your dreams.

(Just the boba. Drinks are up to you. I recommend The Boba Book.)

Buying boba.

Do not buy any boba from China. Do not buy any boba labeled “quick cook”, or boba with instructions that are on the order of 5 minutes. Do not buy any flavored boba. I got violently ill from about half a dozen different brands I ordered randomly off Amazon, all made in China. Some had an odd aftertaste.

Supposedly, the Boba Guys are planning to let us buy the stuff they make domestically in California “soon”. Until then, stick to the stuff that is made of tapioca flour only, and manufactured in Taiwan or The U.S.

Also, the little balls are very fragile and turn to powder in the mail unless they are packed very tightly. This boba, from The Tea Zone is what I buy and recommend buying. Pick up some large diameter straws if you don’t have a stash at home.

Equipment.

You need a big-ass pot of boiling water. The biggest pot you’ve got. I use a big soup pot that holds like 16 or 20 quarts.

IMG_0923
Big Ass Pot

If you only have a few quarts of water, you will ruin pans. The tapioca dust turns to gummy that sticks to the sides and bottom and gets baked on like a motherfucker. You want a ratio of SHIT TONS of water to a handful or two of boba.

Cooking.

Fill it up with water to within an inch or two of the top — Bring it to a fast boil, then put your boba in — a cup or two or three, whatever you think you need. Let it boil for 20-25 minutes… only reduce the heat if you have to to keep it from boiling over.

Uncooked boba will have these little white spots in the middle. Once you see only a few of those in a sea of black pearls, turn off the heat. Let it sit in the hot water for another 20-25 minutes.

IMG_0939
Spot the uncooked boba

Then take the pot to the sink, pour off the excess water, fill it back up with cold water, swoosh it around to rinse; pour, fill, rinse a couple times til the balls are rinsed and lukewarm. You don’t have to drain them dry-dry; leave a small bit of water in the pan.

Flavoring and eating.

Add some sweetener — I like brown sugar, but honey is good too, or molasses and white sugar — and let the balls soak for another 30 minutes so they absorb the flavor. Now they are ready to eat. They will only keep for about a day, and don’t refrigerate them or they get gross.

**If you want the syrupy consistency of the gourmet boba shops, leave a little extra water in there, add the sugar, then simmer on low and STIR CONSTANTLY for 5-10 minutes or until it gets syrupy. I cannot stress this enough: rinse the boba first, and do not stop stirring, if you enjoy your pans and want to use them again

The easiest possible recipe (besides eating from the pot with a spoon) is, fill a glass 1/3 of the way with boba, add milk, add brown sugar simple syrup to taste. Add a couple ice cubes if you like your boba on the firm side. Also, try adding a little bit of rum and Frangelico for your bedtime boba.

Cheers!

IMG_9586
Boba, milk, frangelico
How to make boba at home…without ruining any pans, making yourself ill, or ending up with a soggy, blobby mess

Questionable Advice: Can Engineering Productivity Be Measured?

I follow you on Twitter and read your blog.  I particularly enjoy this post: https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/ I’m reaching out looking for some guidance.

I work as an engineering manager for a company whose non-technology leadership insists there has to be a way to measure the individual productivity of a software engineer. I have the opposite belief. I don’t believe you can measure the productivity of “professional” careers, or thought workers (ex: how do measure productivity of a doctor, lawyer, or chemist?).

For software engineering in particular, I feel that metrics can be gamed, don’t tell the whole story, or in some cases, are completely arbitrary. Do you measure individual developer productivity? If so, what do you measure, and why do you feel it’s valuable? If you don’t and share similar feelings as mine, how would you recommend I justify that position to non-technology leadership?

Thanks for your time.

Anonymous Engineering Manager

Dear Anon,

Once upon a time I had a job as a sysadmin, 100% remote, where all work was tracked using RT tasks. I soon realized that the owner didn’t have a lot of independent technical judgment, and his main barometer for the caliber of our contributions was the number of tasks we closed each day.

I became a ticket-closing machine. I’d snap up the quick and easy tasks within seconds. I’d pattern match and close in bulk when I found a solution for a group of tasks. I dove deep into the list of stale tickets looking for ones I could close as “did not respond” or “waiting for response”, especially once I realized there was no penalty for closing the same ticket over and over.

My boss worshiped me. I was bored as fuck. Sigh.

I guess what I’m trying to say is, I am fully in your camp. I don’t think you can measure the “productivity” of a creative professional by assigning metrics to their behaviors or process markers, and I think that attempting to derive or inflict such metrics can inflict a lot of damage.

In fact, I would say that to the extent you can reduce a job to a set of metrics, that job can be automated away. Metrics are for easy problems — discrete, self-contained, well-understood problems. The more challenging and novel a problem, the less reliable these metrics will be.

Your execs should fucking well know this: how would THEY like to be evaluated based on, like, how many emails they send in a day? Do they believe that would be good for the business? Or would they object that they are tasked with the holistic success of the org, and that their roles are too complex to reduce to a set of metrics without context?

This actually makes my blood boil. It is condescending as fuck for leadership to treat engineers like task-crunching interchangeable cogs. It reveals a deep misunderstanding of how sociotechnical systems are developed and sustained (plus authoritarian tendencies, and usually a big dollop of personal insecurity).

But what is the alternative?

In my experience, the “right” answer, i.e. the best way to run consistently high-performing teams, involves some combination of the following:

  • Outcome-based management that practices focusing on impact, plus
  • Team level health metrics, combined with
  • Engineering ladder and regular lightweight reviews, and
  • Managers who are well calibrated across the org, and encouraged to interrogate their own biases openly & with curiosity.

The right way to look at performance is at the team level. Individual engineers don’t own or maintain code; teams do. The team is the irreducible unit of ownership. So you need to incentivize people to think about work and spending their time cooperatively, optimizing for what is best for the team.

Some of the hardest and most impactful engineering work will be all but invisible on any set of individual metrics. You want people to trust that their manager will have their backs and value their contributions appropriately at review time, if they simply act in the team’s best interest. You do not want them to waste time gaming the metrics or courting personal political favor.

This is one of the reasons that managers need to be technical — so they can cultivate their own independent judgment, instead of basing reviews on hearsay. Because some resources (i.e. your budget for individual bonuses) are unfortunately zero-sum, and you are always going to rely on the good judgment of your engineering leaders when it comes to evaluating the relative impact of individual contributions.

This also is why it’s important for leaders to model the act of openly exploring whether they might be biased in some way:

“I would say that Joe’s contribution this quarter had greater impact than Jane’s. But is that really true? Jane did a LOT of mentoring and other “glue” work, which tends to be under-acknowledged as leadership work, so I just want to make sure I am evaluating this fairly … Does anyone else have a perspective on this? What might I be missing?” — a manager keeping themselves honest in calibrations

I do think every team should be tracking the 4 DORA metrics — time elapsed between merge and deploy, frequency of deploy, time to recover from outages, duration of outages — as well as how often someone is paged outside of business hours. These track pretty closely to engineering productivity and efficiency.

But leadership should do its best to be outcome oriented. The harder the problem, the more senior the contributor, the less business anyone has dictating the details of how or why. Make your agreements, then focus on impact.

This is harder on managers, for sure — it’s easier to count the hours someone spends at their desk or how many lines of code they commit than to develop a nuanced understanding of the quality and timbre of an engineer’s contributions to the product, team and the company over time. It is easier to micromanage the details than to negotiate a mutual understanding of what actually matters, commit to doing your part … and then step away, trusting them to fill in the gaps.

But we should expect this; it’s worth it. It is in those gaps where we feel trusted to act that we find joy and autonomy in our labor, where we do our best work as skilled artisans.

p.s. someone just shared this link with me, OF COURSE martin fowler has already said this 

Questionable Advice: Can Engineering Productivity Be Measured?

Trolley Problems as a Service

Consider:

  • Is it ethical to discriminate in whom you will sell to as a business?  What would you do if you found out that the work you do every day was being used to target and kill migrants at the border? 
  • Is it ethical or defensible to pay two people doing the same job different salaries if they live in different locations and have a different cost of living?  What if paying everyone the same rate means you are outcompeted by those who peg salaries to local rates, because they can vastly out-hire you?
  • You’re at the crowded hotel bar after a company-sponsored event, and one of your most valued customers begins loudly venting opinions about minorities in tech that you find alarming and abhorrent.  What responsibility do you have, if any?  How should you react?
  • If we were close to running out of money in the hypothetical future, should we do layoffs or offer pay cuts?

It’s not getting any simpler to live in this world, is it?  💔

Ethical problems are hard.  Even the ones that seem straightforward on the face of them get stickier the closer you look at them.  There are more stakeholders, more caveats, more cautionary tales, more unintended consequences than you can generally see at face value. It’s like fractal hardness, and anyone who thinks it’s easy is fooling themselves.

We’ve been running an experiment at Honeycomb for the past 6 months, where we talk through hypothetical ethical questions like these once a month. Sometimes they are ripped from the headlines, sometimes they are whatever I can invent the night before. I try to send them around in advance. The entire company is invited.**

Honeycomb is not a democracy, nor do I think that would be an effective way to run a company, any more than I think we should design our SDKs by committee or give everyone an equal vote on design mocks.

But I do think that we have a responsibility to act in the best interests of our stakeholders, to the best of our abilities, and to represent our employees. And that means we need to know where the team stands.

That’s one reason. Another is that people make the worst possible decisions when they’re taken off guard, when they are in an unfamiliar situation (and often panicking). Talking through a bunch of nightmare scenarios is a way for us to exercise these decision-making muscles while the stakes are low. We all get to experience what it’s like to hear a problem, have a kneejerk reaction .. then peeling back the onion to reveal layer after layer of dismaying complexities that muddy our snap certainties.

Honeycomb is a pretty transparent company; we believe that companies are created every day by the people who show up to labor together, so those people have a right to know most things. But it’s not always possible or ethically desirable to share all the gritty details that factor into a decision. My hope is that these practice runs help amplify employees’ voices, help them understand the way we approach big decisions, and help everyone make better decisions — and trust each other’s decisions — when things move fast and times get hard.

(Plus, these ethical puzzles are astonishingly fun to work through together. I highly recommend you borrow this idea and try it out at your own company.)

cheers, and please let me know if you do try it ☺️

charity

** We used to limit attendance to the first 6 people to show up, to try and keep the discussion more authentic and less performative. We recently relaxed this rule since it doesn’t seem to matter, peacocking hasn’t really been an issue.

Trolley Problems as a Service

Good Days, Bad Days, Impossible Days

Last night I was talking with Mark Ferlatte about the advice we have given our respective companies in this pandemic era.  He shared with me this link, on how to salvage a disastrous day.  It’s a good link: you should read it.

My favorite part: “Your feelings will follow your actions.  Just do it.”

The hardest part for me is, “Book-end your day.  Don’t push it into the midnight hours.”  Ugh.  I really, really struggle with this because my brain takes a long long time to settle in and get started on a task to the point where I feel like I’m on a roll with it, and once I’m on a roll I do not want to stop until I’m done.  Because god knows how long it will be — days? weeks?? — until I can catch this wave again, feel inspired again.  But it’s true, if I stay up all night working I’m just setting myself up for a fuzzy, blundery tomorrow.

The advice we gave Honeycombers was differently shaped, though similar in spirit.  I’ve had a few people ask me to share it, so here it is.

We formally request …

First, we would like to point out that what you are all being asked to do right now is impossible.  Parenting, homeschooling, working, caregiving, correcting misinformed neighbors, being an engaged citizen … it is fifteen people’s worth of work.  It is literally impossible.

But hey, it has always been impossible.  We have never been able to do everything we want to do — there isn’t enough time.  There was never enough time!  We succeed as a company not by doing everything on our list, but by saying no to the right things; by NOT-doing enough most things so we can focus on the few things we have identified that matter most.  That was true before COVID, it’s just truer now.

So: let’s all focus hard on our top priority.  Shed as much of the other stuff as you have to.  Shed more.  Ask your manager for help figuring out what to shed, until you are down to an amount you can probably manage.

And speaking of focus:

You aren’t operating at full capacity.  We all get that right now: none of us are.  And nobody expects you to.  So please spend zero energy on performing like you’re doing work, or acting extra-responsive, or keeping up a front like things are normal and you’re doing fine.  That performance costs you precious energy,  while doing nothing to get us closer to our goals.

What we need from you is not performance or busy-busy-ness but your engaged creative self  — your active, curious mind engaging with our top problem.  I would rather have 30 minutes of your creative energy applied to our biggest problem today than five hours of your distracted split-brain, juggling, trying to keep up with chat and seem like you’re as available per usual today.

So when you’re figuring out your schedule, please optimize for that — focused time on our biggest problem — and then communicate your availability to your team.  If you’re a parent and you can only really work three days a week, calendar that.  (If you’re not a parent, remember that you too are allowed to feel overwhelmed and underwater.  Just because some have it even harder, doesn’t invalidate what you’re going through.)

In Summary,

Take care of yourself
Take care of your loved ones
Say no to as much as you possibly can
Focus on impact
No performative normalcy
Remember: this is temporary 🖤

We are incredibly fortunate — to be here, to have these resources, to have each other.  It’s okay to have bad days; this is why we have teams, to carry each other through the hardest spots.  Do your best.  Everything is going to be okay, more or less.

 

Good Days, Bad Days, Impossible Days