Why every software engineering interview should include ops questions

I’ve fallen way behind on my blog posts — my goal was to write one per month, and I haven’t published anything since MAY. Egads. So here I am dipping into the drafts archives! This one was written in April of 2016, when I was noodling over my CraftConf 2016 talk on “DevOps for Developers (see slides).”

So I got to the part in my talk where I’m talking about how to interview and hire software engineers who aren’t going to burn the fucking house down, and realized I could spend a solid hour on that question alone. That’s why I decided to turn it into a blog post instead.

Stop telling ops people to code better, start telling SWEs to ops better

Our industry has gotten very good at pressing operations engineers to get better at writing code, writing tests, and software engineering in general these past few years. Which is great! But we have not been nearly so good at pushing software engineers to level up their systems skills. Which is unfortunate, because it is just as important.

Most systems suffer from the syndrome of running too much software. Tossing more software into the heap is as likely to cause more problems as often as it solves them.

We see this play out at companies stacked with good software engineers who have built horrifying spaghetti messes of their infrastructure, and then commence paging themselves to death.

The only way to unwind this is to reset expectations, and make it clear that

  1. you are still responsible for your code after it’s been deployed to production, and 
  2. operational excellence is everyone’s job.

Operations is the constellation of tools, practices, policies, habits, and docs around shipping value to users, and every single one of us needs to participate in order to do this swiftly and safely.

Every software engineering interviewing loop should have an ops component.

Nobody interviews candidates for SRE or ops nowadays without asking some coding questions. You don’t have to be the greatest programmer in the world, but you can’t be functionally illiterate. The reverse is less common: asking software engineers basic, stupid questions about the lifecycle of their code, instrumentation best practices, etc. 

It’s common practice at lots of companies now to have a software engineer in the loop for hiring SREs to evaluate their coding abilities. It should be just as common to have an ops engineer in the loop for a SWE hire, especially for any SWE who is being considered for a key senior position. Those are the people you most rely on to be mentors and role models for junior hires. All engineers should embrace the ethos of owning their code in production, and nobody should be promoted or hired into a senior role if they don’t.

And yes, that means all engineers!  Even your iOS/Android engineers and website developers should be interested in what happens to their code after they hit deploy.  They should care about things like instrumentation, and what kind of data they may need later to debug their problems, and how their features may impact other infrastructure components.

You need to balance out your software engineers with engineers who don’t react to every problem by writing more code. You need engineers who write code begrudgingly, as a last resort. You’ll find these priceless gems in ops and SRE.

ops questions for software engineers

The best questions are broad and start off easy, with plenty of reasonable answers and pathways to explore. Even beginners can give a reasonable answer, while experts can go on talking for hours.

For example: give them the specs for a new feature, and ask them to talk through the infrastructure choices and dependencies to support that feature. Do they ask about things like which languages, databases, and frameworks are already supported by the team? Do they understand what kind of monitoring and observability tools to use, do they ask about local instrumentation best practices?

Or design a full deployment pipeline together. Probe what they know about generating artifacts, versioning, rollbacks, branching vs master, canarying, rolling restarts, green/blue deploys, etc. How might they design a deploy tool? Talk through the tradeoffs.

Some other good starting points:

  • “Tell me about the last time you caused a production outage. What happened, how did you find out, how was it resolved, and what did you learn?”
  • “What are some of your favorite tools for visibility, instrumentation, and debugging?
  • “Latency seems to have doubled over the last 6 hours. Where do you start looking, how do you start debugging?”
  • And this chestnut: “What happens when you type ‘google.com’ into a web browser?” You would be fucking *astonished* how many senior software engineers don’t know a thing about DNS, HTTP, SSL/TLS, cookies, TCP/IP, routing, load balancers, web servers, proxies, and on and on.

Another question I really like is: “what’s your favorite API (or database, or language) and why?” followed up by “… and what are the worst things about it?” (True love doesn’t mean blind worship.)

Remember, you’re exploring someone’s experience and depth here, not giving them a pass-fail quiz. It’s okay if they don’t know it all. You’re also evaluating them on communication skills, which is severely underrated by most people but is actually as a key technical skill.

Signals to look for

You’re not looking for perfection. You are teasing out signals for things like, how will this person perform on a team where software engineers are expected to own their code? How much do they know about the world outside the code they write themselves? Are they curious, eager, and willing to learn, or fearful, incurious and begrudging?

Do they expect networks to be reliable? Do they expect databases to respond, retries to succeed? Are they offended by the idea of being on call? Are they overly clever or do they look to simplify? (God, I hate clever software engineers 🙃.)

It’s valuable to get a feel for an engineer’s operational chops, but let’s be clear, you’re doing this for one big reason: to set expectations. By making ops questions part of the interview, you’re establishing from the start that you run an org where operations is valued, where ownership is non-optional. This is not an ivory tower where software engineers can merrily git push and go home for the day and let other people handle the fallout

It can be toxic when you have an engineer who thinks all ops work is toil and operations engineering is lesser-than. It tends to result in operations work being done very poorly. This is your best chance to let those people self-select out.

You know what, I’m actually feeling uncharacteristically optimistic right now. I’m remembering how controversial some of this stuff was when I first wrote it, five years ago in 2016. Nowadays it just sounds obvious. Like table stakes.

Hell yeah. 🤘

Why every software engineering interview should include ops questions

Questionable Advice: Premature Senior Followup Q’s

Some interesting followup questions arose from my recent post on the trap of the premature senior:

I found your blog (from Hacker News, I think) and your “Questionable Advice: The Trap of the Premature Senior” spoke to me, but I was wondering, do you have any followup advice on handling compensation in this situation? Should one be open to making less with this type of move, or is that not actually an issue?

You should ABSOLUTELY be open to making less. Consider it an investment in your long term career. Think of the extra money your company is paying you now as a hostage premium on top of your real market value. Staying isn’t what’s good for you, and that makes it hazard pay.

Your career is the single most valuable asset you have — it’s a multimillion dollar appreciating asset, and you should curate and guide it with an eye toward longevity. The first decade of your career is way, way too early to start optimizing for salary over experience.

This question was very relevant to me right now, as I recently spoke to a recruiter about a position that’s more in line with my career goals but pays less, and her immediate reaction was “don’t go backward on compensation.” But I can see the trade-off value in losing some comp to gain “better” experience.

Yeah, I think that’s terrible advice. 🙂 In general, if the compensation is fair and respectful, I think that is the absolute *worst* reason to make a job decision.

Salary is not a one-way escalator that you hop on after school, and gracefully exit at the peak upon retiring. It’s possible your recruiter’s advice was based on the assumption that your employers are likely to ask for your current salary and base their offers off of it. That used to be common practice, but is less and less so because of the fairness issues involved. Comp should be based on the value of your labor to the company, not your past comp, and in many states it is illegal to ask about your salary.

You may choose to take lower salaries at various times in your career — to work at a nonprofit or gov job, to learn new skills, in exchange for more flexibility or vacation time or titles or stock options, or as a result of moving to a different city or country.

I am not a financial advisor, but I am a big believer in retaining optionality. If the opportunity of your dreams came along with a starting salary of 150k, but every penny of your 170k salary is already committed, that’s a big loss of opportunity for you. This is a strong argument for trying to live well within your means, save religiously, and always have fuck you money in the bank. If you’re young and you get a salary bump, consider automatically diverting the raise into savings. Avoiding lifestyle inflation is the most painless way to save.

We have the unfathomable luxury of being incredibly well compensated for what we do. What is that luxury worth if we don’t use it to liberate ourselves, to facilitate happiness and fulfillment?

Questionable Advice: Premature Senior Followup Q’s

“Why are my tests so slow?” A list of likely suspects, anti-patterns, and unresolved personal trauma.

Over the past couple of weeks I’ve been tweeting a LOT about lead time to deploy: the interval encompassing the time from when the code gets written and when it’s been deployed to production. Also described as “how long it takes you to run CI/CD.”

How important is this?

Fucking central.

Here is a quickie thread from this week, or just go read “Accelerate” like everybody already should have. 🙃

It’s nigh impossible to have a high-performing team with a long lead time, and becomes drastically easier with a dramatically shorter lead time.

🌷 Shorter is always better.
🌻 One mergeset per deploy.
🌹 Deploy should be automatic.

And it should clock in under 15 minutes, all the way from “merging!” to “deployed!”.

Now some people will nod and agree here, and others freak the fuck out. “FIFTEEN MINUTES?” they squall, and begin accusing me of making things up or working for only very small companies. Nope, and nope. There are no magic tricks here, just high standards and good engineering, and the commitment to maintaining your goals quarter by quarter.

If you get CI/CD right, a lot of other critical functions, behaviors, and intuitions are aligned to be comfortably successful and correct with minimal effort. If you get it wrong, you will spend countless cycles chasing pathologies. It’s like choosing to eat your vegetables every day vs choosing a diet of cake and soda for fifty years, then playing whackamole with all the symptoms manifesting on your poor, mouldering body.

Is this ideal achievable for every team, on every stack, product, customer and regulatory environment in the world? No, I’m not being stupid or willfully blind. But I suggest pouring your time and creative energy into figuring out how closely you can approximate the ideal given what you have, instead of compiling all the reasons why you can’t achieve it.

Most of the people who tell me they can’t do this are quite wrong, turns out. And even if you can’t down to 15 minutes, ANY reduction in lead time will pay out massive, compounding, benefits to your team and adjacent teams forever and ever.

So — what was it you said you were working on right now, exactly? that was so important? 🤔

“Cutting my build time by 90%!” — you

Huzzah. 🤠

So let’s get you started! Here, courtesy of my twitterfriends, is a long compiled list of Likely Suspects and CI/CD Offenders, a long list of anti-patterns, and some unresolved personal pain & suffering to hunt down and question when your build gets slow..

✨15 minutes or bust, baby!✨

 
Where it all started: what keeps you from getting under 15 minute CI/CD runs?

Generally good advice.

  • Instrument your build pipeline with spans and traces so you can see where all your time is going. ALWAYS. Instrument.
  • Order tests by time to execute and likelihood of failure.
  • Don’t run all tests, only tests affected by your change
  • Similarly, reduce build scope; if you only change front-end code, only build/test/deploy the front end, and for heaven’s sake don’t fuss with all the static asset generation
  • Don’t hop regions or zones any more than you absolutely must.
  • Prune and expire tests regularly, don’t wait for it to get Really Bad
  • Combine functionality of tests where possible — tests need regular massages and refactors too
  • Pipeline, pipeline, pipeline tests … with care and intention
  • You do not need multiple non-production environment in your CI/CD process. Push your artifacts to S3 and pull them down from production. Fight me on this
  • Pull is preferable to push. (see below)
  • Set a time elapsed target for your team, and give it some maintenance any time it slips by 25%

The usual suspects

  • tests that take several seconds to init
  • setup/teardown of databases (HINT try ramdisks)
  • importing test data, seeding databases, sometimes multiple times
  • rsyncing sequentially
  • rsyncing in parallel, all pulling from a single underprovisioned source
  • long git pulls (eg cloning whole repo each time)
  • CI rot (eg large historical build logs)
  • poor teardown (eg prior stuck builds still running, chewing CPU, or artifacts bloating over time
  • integration tests that spin up entire services (eg elasticsearch)
  • npm install taking 2-3 minutes
  • bundle install taking 5 minutes
  • resource starvation of CI/CD system
  • not using containerized build pipeline
  • …(etc)
 
Continuous deployment to industrial robots in prod?? Props, man.

Not properly separating the streams of “Our Software” (changes constantly) vs “infrastructure” (changes rarely)

  • running cloudformation to setup new load balancers, dbs, etc for an entire acceptance environment
  • docker pulls, image builds, docker pushes container spin up for tests

“Does this really go here?”

  • packaging large build artifacts into different format for distribution
  • slow static source code analysis tools
  • trying to clone production data back to staging, or reset dbs between runs
  • launching temp infra of sibling services for end-to-end tests, running canaries
  • selenium and other UX tests, transpiling and bundling assets

“Have a seat and think about your life choices.”

  • excessive number of dependencies
  • extreme legacy dependencies (things from the 90s)
  • tests with “sleep” in them
  • entirely too large frontends that should be broken up into modules

“We regret to remind you that most AWS calls operate at the pace of ‘Infrastructure’, not ‘Software'”

  • AWS CodeBuild has several minutes of provisioning time before you’re even executing your own code — even a few distinct jobs in a pipeline and you might suffer 15 min of waiting for CodeBuild to do actual work
  • building a new AMI
  • using EBS
  • spinning up EC2 nodes .. sequentially 😱
  • cool it with the AWS calls basically



A few responses were oozing with some unresolved trauma, lol.

Natural Born Opponents: “Just cache it” and “From the top!”

  • builds install correct version of toolchain from scratch each time
  • rebuilding entire project from source every build
  • failure to cache dependencies across runs (eg npm cache not set properly)

“Parallelization: the cause of, and solution to, all CI problems”

  • shared test state, which prevents parallel testing due to flakiness and non-deterministic test results
  • not parallelizing tests
 
 
I have so many questions….

Thanks to @wrd83, @sorenvind, @olitomli, @barney_parker, @dastbe, @myajpitz, @gfodor, @mrz, @rwilcox, @tomaslin, @pwyliu, @runewake2, @pdehlkefor, and many more for their contributions!

P.S. what did I say about instrumenting your build pipeline? For more on honeycomb + instrumentation, see this thread. Our free tier is incredibly generous, btw ☺️

Stay tuned for more long form blog posts on this topic. Coming soon. 🌈

charity

P.S. this blog post is the best thing i’ve ever read about reducing your build time. EVER.

“Why are my tests so slow?” A list of likely suspects, anti-patterns, and unresolved personal trauma.

Questionable Advice: The Trap of The Premature Senior

I’ve been at my current job for three years, and I am suddenly, accidentally, the most senior engineer on the team. I spend my days handling things like bootcamps, mentoring, architecture, and helping other engineers carve off meaningful work. This has taken a huge toll on the kind of work I want to do as an IC. I still enjoy writing and shipping features, and I am not a manager, but now I feel like I spend my days conducting meetings, interviewing, and unblocking others constantly instead of writing code myself.

What should I do? How can I deal with this situation in an effective manner? How can I keep from getting burned out on zoom? How can I reclaim more of my time to write code for myself, without sacrificing my influence? Should I get a new job? I have thought about going out and getting a new job, but I really like having a say at a high level. Here I get looped into all of the most important decisions and meetings. If I get a new job, how can I avoid starting over at the bottom of the heap and just taking assignments from other people? P.S., this is my first job.

 

Get a new job.

Yes, you will reset your seniority and have to earn it all over again. Yes, it will be uncomfortable and your ego will be cranky over it. Yes, you will be at the bottom of the heap and take assignments from other people for a while. Yes, you should do it anyway.

What you are experiencing now is the alluring comfort of premature seniority. You’re the smartest kid in the room, you know every corner of the system inside and out, you win every argument and anticipate every objection and you are part of every decision and you feel so deeply, pleasingly needed by the people around you.

It’s a trap.

Get the fuck out of there.

There is a world of distance between being expert in this system and being an actual expert in your chosen craft. The second is seniority; the first is merely .. familiarity

Deep down I think you know this, and feel a gnawing insecurity over your position; why else would you have emailed me? You were right. Treasure that uneasy feeling in your gut, that discomfort in the face of supreme comfortable-ness. It will lead you to a long and prosperous career as an engineer if you learn to trust it.

Think of every job like an escalator — a 50-foot high escalator that takes about two years to ride to the top. But once you’ve summited, you stall out. You can either stay and wander on that floor, or you can step to the left and pick another escalator and ride it up another 50 feet. And another.

In my mind, someone becomes a real senior engineer after they’ve done this about three times. 2-3 teams, stacks, languages, and roles, over a 5-8 year period, and then they’re solidly baked. There are insights you can derive from having seen problems solved in a few different ways that you can’t with only a single point of reference.

You don’t become a senior engineer at the 50-foot ascent, no matter how thoroughly you know the landscape. You become a senior engineer somewhere well over 100 feet, with a couple of lane changes under your belt.

The act of learning a new language and/or stack is itself an important skill. Experiencing how different orgs ship code in vastly different ways is how you internalize that there’s no one blessed path, only different sets of tradeoffs, and how you learn to reason about those tradeoffs.

And it is good for us to start over with beginner eyes. It’s humbling, it’s clarifying, it’s a cleanse for the soul. If you get too attached to feeling senior, to feeling necessary, you will undervalue the virtues of fresh eyes and questioning, of influence without authority. It is good for you to practice uncertainty and influencing others without the cheat codes of deep familiarity.

Nobody wants to work with seniors who clutch their authority with a white knuckled grip. We want to work with those who wear it lightly, who remember what it was like in our shoes.

Ultimately, this is a strong argument for building our teams behind a Rawlsian veil of ignorance concerning our own place in the pecking order. Starting fresh yourself will help you build teams where it is not miserable to be a beginner, where beginners’ contributions are recognized, where even beginners do not simply “take orders”, as you said. Because literally nobody wants that, including the beginners you are working with on  your teams today.

After you have gotten a new job or two, and proven to yourself that you can level up again and master new stacks and technologies, that fretful inner voice questioning whether you deserve the respect you receive or not will calm down. You will have proven to yourself that your success wasn’t just a one-off, that you can be dropped into any situation, learn the local ropes and succeed. You will be a senior engineer.

Get the fuck out of there. Go. <3

 

 

 

Questionable Advice: The Trap of The Premature Senior

Things to know about engineering levels

This twitter thread seemed to strike a chord with people, rather astonishingly so. I am transcribing parts of it for the sake of longevity and findability.

I keep talking to engineers who are frustrated that they aren’t leveling up faster, aren’t reaching senior levels as quickly as other people they know, feel stuck and don’t know how to get to the next level, etc. And I’ve begun to notice a common blind spot around leveling —

✨ not every opportunity exists ✨
✨ at every company ✨
✨ at every time.✨

Sure, if you’re a junior engineer, you should be able to level up to intermediate pretty much anywhere. But it gets progressively trickier after that. Even the path from intermediate to senior can depend on a number of situational variables:

Is there oxygen?

  • How many other senior engineers do you work with? how many other intermediate engineers around your level? All of these people will be pulling from the same bin of work, looking for promo-worthy, solidly-senior projects.
  • Does your ladder explicitly call for mentorship or leading small teams of lower-leveled engineers? Are there enough of those folks to go around?
  • Have you sufficiently wrapped up your last project well enough to move on? Was it actually completed in a way that demonstrated clear mastery and readiness for bigger and harder work, ordid you leave a mess behind you? That may limit people’s appetite to take a risk on you with mission-critical projects.
  • What are the biggest needs of the business right now? Any process that generates projects ought to begin with this question before proceeding on to carve out a chunk that fits your promo desires, not the other way around. 🙃
  • Do you happen to work in a niche or specialty area of engineering, particularly one crammed with super-senior, world-famous highly leveled people? This can be fantastic when it comes to your ability to soak up knowledge from the world’s best, but it may simultaneously delay your ability to level up.

In short, is there oxygen at the next level? Does the company need more of the type of engineer you want to be, vs more of the type of engineer you are now? If they need more people pounding out code and fewer architects, they’re unlikely to want to promote you to a role that involves mostly architecture..

Literally no company can possibly make use of a top-heavy eng org stuffed with senior+ engineers, if all of them are expected to demonstrate company-wide impact or global impact every review period. There’s only so much high-level work to go around for every fifty engineers writing code and features and executing on those systems.

There is only so much oxygen at each level.

Inflation.

Of course, this is all assuming that your company takes leveling seriously. Most … really … don’t.

It’s tough. It’s tough to hold your ground when a valued engineer is complaining and dropping hints they may leave if they don’t get that promotion soon. It’s much easier to give in, make an exception, argue for rounding up.

This may sound good, but it is not ultimately in your best interests as that engineer. Seriously. </3

There is sooo much title inflation in this industry already. People are given the title “senior engineer” in just 3-5 years, need I say more??

If you let a little inflation into your system by making exceptions, it causes more trouble than it’s worth. Always. The only leverage you have when people try to get you to make exceptions is if you can honestly say, “no exceptions.” Give in just once, and your moral authority evaporates.

Leveling up.

I would urge you not to make most, if any, career decisions based on levels or titles that are offered you. But I do understand how frustrating and infuriating it can be to be in a situation that is clearly unfair (usually because a manager got pressured into making an exception… tsk tsk), or if you don’t understand how to move your career forward.

So, here are a few strategic tips for leveling up.

  1. Generalists level up faster than specialists.
  2. When evaluating roles, choose ones where your specialty is part of their mission, or at least key to its execution. It has a far lower likelihood of getting outsourced, deprioritized, lacking investment in, or just forgotten about if what you do is core to what they do.
  3. Always ask to see the job ladder when interviewing. If they hedge or fumble, don’t take that job.
  4. Talk to your manager about the job ladder. Talk to your skip level about levels too! Managers love this shit. They can talk on and on and on about levels, long past your exhaustion point. It can be annoying, but it’s actually a sign of a good manager who cares and thinks about the edge cases in processes, and their impacts on people and teams.
  5. That said, don’t take the ladder as a checklist to memorize or thing to be pored over and obsessed over. It’s an incomplete attempt at both shaping and reflecting relative impact. Focus on impact.
  6. Is it easier to level up as a manager than as an engineer? Sorta-kinda, I guess so? There are at least two real phenomena at play here.
    1. There are simply more roles to go around in the management track. You need like, what, 1-2 E7/E8 (or principal, or architect?) level engineers per 100-500 engineers, but several managers/directors/etc
    2. Manager effectiveness is grounded in their relationships. It takes managers longer to have impact after they start a new role, but their potential impact grows and grows as their tenure gets longer. So yes, there’s a bit more of an escalator effect if you stay on the manager track at a company for several years. There is no similar escalator on the eng side; you have to be truly exceptional or truly lucky.
    3. But it really depends on the organization.
  7. It is much easier to level up quickly at fast-growing companies. When there is far more work than workers, and everyone is getting dropped in the deep end to sink or swim, you level up fast. Don’t underestimate what a stressful and awful experience this can be, though.
  8. Many engineers get stuck on the bubble getting to senior because they are impatient and want a map. They just want someone to *tell them what to do*. Which is the very opposite of what a senior engineer does. 🙃  Develop your judgment around what needs to be done, and do it.
  9. Your relationship with your manager matters. So does your ability to communicate about the work you are doing, its difficulty, its unexpected challenges and triumphs, etc. This is called “managing up”, and it is an actual skill which I am *terrible* at. So are most of you. 😉
  10. TLDR, if leveling matters to you (and it should matter to everyone, to some extent!), then look curiously and critically around for opportunities, and seek to maximize them. Want to become an E6/E7? Probably don’t join a startup that doesn’t have any very high-level work to do, or already has more than enough people functioning at those levels and many more nipping their heels looking for the same opportunity.This sort of thing is very obvious to us with the manager track (if you want to go from M->Dir, don’t join a startup that already HAS directors and managers who want to level up), but seems less obvious with engineering.

Most reasonable, non-desperate companies with options won’t hire you directly into the next level up which you haven’t done before, on either the manager or the engineer track. (Yellow flag if they do.)

But it is perfectly reasonable to express your career objectives in the interview, and make sure you’re on the same wavelength and seeing the same opportunities. Do you want to become a manager or a tech lead in a few months? Say so.

If it doesn’t exist now, do they think this opportunity may soon open up? Can they see a path forward for you there, if all goes well? Would they be interested in helping you get there? How many people may already be eyeing that same path? Is there enough opportunity for more than one? On what timeframe? Who will decide who gets the role, and how?

Engineers tend to find these conversations uncomfortable, and so they tend to avoid them because they don’t want to make the hiring manager uncomfortable by being pushy.

Relax. Managers don’t find this uncomfortable at all, it’s their bread and butter. (And even fi they do find it uncomfortable, tough beans.. it’s their job.) Ask away. ☺️

Misc notes on leveling.

P.S. Engineers seem to have a very sparse mental model of how leveling works, so here are a few more notes on how levels work at Honeycomb, which is adapted from conventions at Facebook/Google.

  • Each level after senior engineer (E5 for us) gets approx an order of magnitude harder to achieve, and an order of magnitude fewer engineers hold that title.
  • E5 is considered a “terminal level”, which sounds scary, but just means “you do not have to advance beyond this level.” If you never get promoted again, you won’t get fired either.
  • Whereas if you do not advance from E3-> E4 within 2 years, and E4->E5 within 3 years, you are automatically put on a performance improvement plan (at Facebook, I mean, not Honeycomb).
  • We (Honeycomb) hire into E5 as our highest level to start at, both because a) our interview process is not designed to let us parse differences between senior vs super-senior or super-duper senior, and b) we figure nobody is really able to come in the door with >E5 impact for the first 6 months anyway. So we can level them up quickly after they join and we get a feel for their work.

<3 charity.

Things to know about engineering levels

If Management Isn’t A Promotion, Then Engineering Isn’t A Demotion

I wrote a piece this week about what motivates people to become managers (tldr mostly org dysfunction), and Julian Dunn replied with some typically insightful tweets:

(I originally titled this article “Julian Dunn and the Case of the Bad Manager”, lol)

God, YES. This is something that has been on my queue of “topics to write about” for so long, and I haven’t because it’s just too big (and sometimes I tell myself, optimistically, it’s just too obvious?).

Julian’s point is that the reason so many bad managers persist is because it’s perceived as a promotion. Which means going back to engineering after managing is, ipso facto, a demotion. Which is really fucking hard to swallow. For anyone.

I touched on this briefly in an earlier post, the Pendulum or the Ladder, when I wrote,

“If management isn’t a promotion, then returning to hands-on work isn’t a demotion, either.  Right?”

There are a few separate points here which are worth unfurling separately.

  1. Management is widely seen as a promotion
  2. Management really does grant you some formal powers over your peers, which contributes to perceived hierarchy
  3. Humans are hierarchical mammals, exquisitely sensitive to any loss of status — we hates it
  4. But this is a cultural choice, not destiny. And we can change it.

Management is seen as a promotion

The notion that management is a promotion is deeply ingrained into our culture. It’s in language, pop culture, business books, any and all sources of career advice. If you became a manager and told your mom about it, she probably congratulated you and told you how proud she was. If you go out on a job interview, you’re expected to reach for the same rung or a higher one — or eyebrows will raise.

That’s a lot of cultural baggage to lean against. But I believe this is an idea whose time has come.

Any technical company should work hard to center and celebrate the work being done to build the product and make customers happy. Management is overhead, to be brutally frank about it, and we should not design organizations that would lead any rational, ambitious person to aspire to be overhead, should we?

The surest path to acclaim and glory (and promotions and raises) should be found through contributing. Not managing. Not being overhead.

… Because it mostly is a promotion, honestly

It is absolutely true that when you become a manager, you acquire new powers. As a tool of the org, you are granted certain powers to act on behalf of the organization, in exchange for being held accountable for certain outcomes.

These explicit powers often include hiring and firing decisions, access to privileged information, and making and meeting budgets.

But most of your powers aren’t formal at all. Most of your power comes from people listening more closely to what you say, giving your opinions more weight, and (consciously or subconsciously) just trying to please you, because they know you hold some influence over their career outcomes. It comes from the fact that so much information flows through managers. And finally, it comes from relationships — the strength of your personal relationships and mutual trust with other people throughout the org.

So how is this not a promotion? Well, it is a promotion at most companies, to be perfectly honest. But it does not have to be a promotion, if you acknowledge that these privileges and powers are accepted only by sacrificing other privileges and powers, and if you structurally allocate power to other roles. For example, you should acquire managerial powers only at the expense of technical decision-making powers.

I believe that the healthiest companies are ones where managerial powers are limited, enumerated, and minimal, with robust powers explicitly reserved for technical ICs. (much like the Constitution provides for Congress and the States, respectively.)

But it shouldn’t be. “Management” is a support role

Here are some of the reasons why we should invert the hierarchy and embrace management as a service role, a support position.

  1. Tech is a creative industry. Hierarchical leadership is a relic, a holdover from the days of manual labor. Hierarchy kills creativity, which leads to worse business outcomes.
  2. Bad managers are a huge problem in tech. Just like Julian says, the wrong people are doing the job, for the wrong reasons, because they can’t to take the hit to the ego (and paycheck) of the demotion. This leads to unhappy teams and ultimately loss of talent.
  3. I firmly believe that the engineer-manager pendulum is the way to build great technical leaders. The great line managers are never more than a few years removed from hands on work themselves, the great tech leads have always done a stint or two as a people manager. The promotion myth therefore both starves us of powerful technical leadership.and leaves us saddled with unhappy managers who have dwindling relevant skills, year after year.
  4. The ladder is a trap. There are an order of magnitude fewer jobs for each rung you ascend. Meanwhile, the higher you climb the farther removed you are from the work most find meaningful (building things, making customers happy). The perception that you are a failure if you do anything but climb higher therefore traps a great many people in a cycle of intense anxiety and unhappiness.
  5. Management is only one of many forms leadership can take. Yes, you have formal powers delegated to you on behalf of the org, but formal authority is the weakest form of power, and you should resort to using it rarely. Good leaders lead by influence and persuasion, weak leaders with “because I said so.”

Most engineers become managers to cope with org fuckery

Many people (like me!) become managers because they want access to the powers it gives them. As I argued in my last article, this is usually because they are frustrated with some organizational fuckery and it seems the only plausible way to fix or work around said fuckery is by becoming a manager.

Earlier this year I was having a 1×1 with one of our engineers, Martin Holman, who has been a manager before and had expressed interest in doing it again. So, I asked him, was he still interested?[1]

He thought for a moment, and replied, “You know, I thought I wanted to be a manager again, I really did. But I think what I actually wanted was a seat at the table — to know what was going on, to have a say in what work I do. But I don’t feel out of the loop here. So it turns out I don’t feel any need to become a manager.”

Not only did that warm my heart, it answered a question I didn’t know I had. I think they would be a good manager, and should they change their mind again in the future, I will completely support them changing their mind again (minds change! it’s what they do!) — but I hope it is never because they feel that technical contributors are left out of the loop, or don’t have a say in what they do.

That’s what I’d call organizational fuckery.

A roadmap for changing your company culture

If “management is not a promotion” is a cultural value you would like to embrace at your company, here are some concrete actions you should take.

  1. Make sure the pay bands for engineers and managers are equal, or even pay engineers more than managers of the same rank. (Slack does this, or used to.)
  2. Have IC (individual contributor) levels for engineers that track management levels, all the way up to VP.
  3. Look for ways to give high-level ICs information and opportunities for company impact that are on par with their people-manager counterparts.
  4. Technical contributors should own and be accountable for technical strategy and decision-making, not managers.
  5. Demystify management. Break it down into its constituent skills (giving feedback, running meetings, planning and budgeting, mentoring, running programs) and encourage everyone to develop those leadership skills.
  6. Offer any management roles that may open up to internal transfers before considering external candidates.
  7. Offer training and support for first-time managers who are undergoing that first career change. Offer engineers the same leadership coaching opportunities as managers.
  8. Explicitly encourage managers to swing back to IC roles after two or three years. Support them through a generous grace period while refreshing their technical skills.
  9. Watch your language. Loaded terms are everywhere, whether hierarchical (referring to people as being “above” others), or authoritarian (talking about bosses, managers). While it’s impossible to strip it from our vocabulary, it’s worth being thoughtful in how you represent reality, and using neutral phrases like “I support two teams” whenever possible.
  10. Be explicit; repeat yourself. Say over and over that management is not a promotion, it is a change of career. Say it internally and externally, in your interview processes and recruiting messages. Educate your recruiting staff too (and be stern about it).

This isn’t a thing you can do once and be done with it; it’s an ongoing effort you must commit to. Managers tend to accrue power over time, like a gravitational force. In order to counterbalance this drift, managers need to consciously push power out to others. They must use their role as “information router” to inform and empower people to own decisions, instead of hoarding it for themselves.

“Management is not a promotion” is my favorite bat signal

“Management is not a promotion, it’s a change of career.” I say this over and over again, even though it’s more aspirational than accurate.

Yet I say it anyway, because it’s a bat signal. It’s how the people I want to work with can find their way to me. And it repels the people I don’t want to work with just as efficiently.

When we recently posted our first-ever job req for an engineering manager, I included this under the list of optional skills:

  • You have worked as an engineering director or higher before, and decided to return to line management. Why? Because we value people who don’t blindly climb hierarchies just because they’re there. We value people who know themselves and what they find fulfilling in work and in life, and who can handle the hit to the ego that it takes to move “down” in pursuit of that fulfillment. Also, it would be interesting to talk about how you have solved org problems at other companies.

I cannot tell you how many amazing candidates zeroed in on that paragraph and came running. People who had been VPs before, been CTO, been director. People who were not only interested in becoming a line manager again, but were hungry to go back, to be closer to the people doing the work.

Something I heard them say again and again was, “People look at me like I’m crazy for wanting this,” “I have never had anyone see this as a strength.”

These were candidates who were acutely attuned to power dynamics, had exceptional self-knowledge, and who had seen and done so much to make organizations successful at multiple levels. What a set of superpowers!

Humans HAAAAAATE losing status.

We hate it. We hate it so bad. Even when we tell ourselves it’s what we wanted, even when we know it’s best for us, even when all the stars align. Something inside of us kicks and screams and feels excruciatingly attuned to the ripple effects of any status loss for a long time.

Like all such powerful irrational feelings, it’s evolution’s fault. Once upon a time it helped us survive and procreate. Now it’s just a nuisance, something to be worked around and minimized.

Where someone sits on the org chart should not determine that person’s ability to drive change, nor should their preference for tech problems or people problems. We need to see the work that engineers, managers, directors, VPs, and CxOs do as equally valuable and equally capable of prestige. We need to flip the org chart upside down, and treat “management” roles like the support systems they should be.

The work done by a database engineer is different from the work done by a VP marketing, or a director of database engineering. It is not inherently better or worse, easier or harder, more or less deserving of praise and admiration. It is simply different.[2]

And we will have the best chance finding the work that brings the most meaning and joy to our lives if we can drain the hierarchical residue out of our perception of these roles, by flattening pay structures, equalizing power dynamics, and making sure everyone has the tools they need to do their job with as little hierarchical bullshit as possible.

charity.

[1] Martin said I could tell this story and use his name. I actually try to avoid talking about people, conversations, or anecdotes from Honeycomb as a more or less blanket rule, because I don’t want people to be perpetually on edge wondering if I am talking about them. (So if you’re wondering if I’m talking about you: I’m not. Unless I asked first.)

[2] Raise your hand if you’ve worked at a company where a DB engineer had a far greater impact on the bottom line some quarters than any of the VPs did. ✋

If Management Isn’t A Promotion, Then Engineering Isn’t A Demotion

Questionable Advice: Can Engineering Productivity Be Measured?

I follow you on Twitter and read your blog.  I particularly enjoy this post: https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/ I’m reaching out looking for some guidance.

I work as an engineering manager for a company whose non-technology leadership insists there has to be a way to measure the individual productivity of a software engineer. I have the opposite belief. I don’t believe you can measure the productivity of “professional” careers, or thought workers (ex: how do measure productivity of a doctor, lawyer, or chemist?).

For software engineering in particular, I feel that metrics can be gamed, don’t tell the whole story, or in some cases, are completely arbitrary. Do you measure individual developer productivity? If so, what do you measure, and why do you feel it’s valuable? If you don’t and share similar feelings as mine, how would you recommend I justify that position to non-technology leadership?

Thanks for your time.

Anonymous Engineering Manager

Dear Anon,

Once upon a time I had a job as a sysadmin, 100% remote, where all work was tracked using RT tasks. I soon realized that the owner didn’t have a lot of independent technical judgment, and his main barometer for the caliber of our contributions was the number of tasks we closed each day.

I became a ticket-closing machine. I’d snap up the quick and easy tasks within seconds. I’d pattern match and close in bulk when I found a solution for a group of tasks. I dove deep into the list of stale tickets looking for ones I could close as “did not respond” or “waiting for response”, especially once I realized there was no penalty for closing the same ticket over and over.

My boss worshiped me. I was bored as fuck. Sigh.

I guess what I’m trying to say is, I am fully in your camp. I don’t think you can measure the “productivity” of a creative professional by assigning metrics to their behaviors or process markers, and I think that attempting to derive or inflict such metrics can inflict a lot of damage.

In fact, I would say that to the extent you can reduce a job to a set of metrics, that job can be automated away. Metrics are for easy problems — discrete, self-contained, well-understood problems. The more challenging and novel a problem, the less reliable these metrics will be.

Your execs should fucking well know this: how would THEY like to be evaluated based on, like, how many emails they send in a day? Do they believe that would be good for the business? Or would they object that they are tasked with the holistic success of the org, and that their roles are too complex to reduce to a set of metrics without context?

This actually makes my blood boil. It is condescending as fuck for leadership to treat engineers like task-crunching interchangeable cogs. It reveals a deep misunderstanding of how sociotechnical systems are developed and sustained (plus authoritarian tendencies, and usually a big dollop of personal insecurity).

But what is the alternative?

In my experience, the “right” answer, i.e. the best way to run consistently high-performing teams, involves some combination of the following:

  • Outcome-based management that practices focusing on impact, plus
  • Team level health metrics, combined with
  • Engineering ladder and regular lightweight reviews, and
  • Managers who are well calibrated across the org, and encouraged to interrogate their own biases openly & with curiosity.

The right way to look at performance is at the team level. Individual engineers don’t own or maintain code; teams do. The team is the irreducible unit of ownership. So you need to incentivize people to think about work and spending their time cooperatively, optimizing for what is best for the team.

Some of the hardest and most impactful engineering work will be all but invisible on any set of individual metrics. You want people to trust that their manager will have their backs and value their contributions appropriately at review time, if they simply act in the team’s best interest. You do not want them to waste time gaming the metrics or courting personal political favor.

This is one of the reasons that managers need to be technical — so they can cultivate their own independent judgment, instead of basing reviews on hearsay. Because some resources (i.e. your budget for individual bonuses) are unfortunately zero-sum, and you are always going to rely on the good judgment of your engineering leaders when it comes to evaluating the relative impact of individual contributions.

“I would say that Joe’s contribution this quarter had greater impact than Jane’s. But is that really true? Jane did a LOT of mentoring and other “glue” work, which tends to be under-acknowledged as leadership work, so I just want to make sure I am evaluating this fairly … Does anyone else have a perspective on this? What might I be missing?” — a manager keeping themselves honest in calibrations

I do think every team should be tracking the 4 DORA metrics — time elapsed between merge and deploy, frequency of deploy, time to recover from outages, duration of outages — as well as how often someone is paged outside of business hours. These track pretty closely to engineering productivity and efficiency.

But leadership should do its best to be outcome oriented. The harder the problem, the more senior the contributor, the less business anyone has dictating the details of how or why. Make your agreements, then focus on impact.

This is harder on managers, for sure — it’s easier to count the hours someone spends at their desk or how many lines of code they commit than to develop a nuanced understanding of the quality and timbre of an engineer’s contributions to the product, team and the company over time. It is easier to micromanage the details than to negotiate a mutual understanding of what actually matters, commit to doing your part … and then step away, trusting them to fill in the gaps.

But we should expect this; it’s worth it. It is in those gaps where we feel trusted to act that we find joy and autonomy in our labor, where we do our best work as skilled artisans.

Questionable Advice: Can Engineering Productivity Be Measured?

Questionable Advice #2: How Do I Get My Team Into Observability?

Welcome to the second installment of my advice column! Last time we talked about the emotional impact of going back to engineering after a stint in management. If you have a question you’d like to ask, please email me or DM it to me on twitter.

Hi Charity! I hope it’s ok to just ask you this… 

I’m trying to get our company more aware of observability and I’m finding it difficult to convince people to look more into it. We currently don’t have the kind of systems that would require it much – but we will in future and I want us to be ahead of the game. 

If you have any tips about how to explain this to developers (who are aware that quality is important but don’t always advocate for it / do it as much as I’d prefer), or have concrete examples of “here’s a situation that we needed observability to solve – and here’s how we solved it”, I’d be super grateful. 

If this is too much to ask, let me know too 🙂 

I’ve been talking to Abby Bangser a lot recently – and I’m “classifying” observability as “exploring in production” in my mental map – if you have philosophical thoughts on that, I’d also love to hear them 🙂

alex_schl

 

Dear Alex,

Everyone’s systems are broken. Not just yours!

Yay, what a GREAT note!  I feel like I get asked some subset or variation of these questions several times a week, and I am delighted for the opportunity to both write up a response for you and post it for others to read.  I bet there are orders of magnitude more people out there with the same questions who *don’t* ask, so I really appreciate those who do. <3

I want to talk about the nuts and bolts of pitching to engineering teams and shepherding technical decisions like this, and I promise I will offer you some links to examples and other materials. But first I want to examine some of the assumptions in your note, because they elegantly illuminate a couple of common myths and misconceptions.

Myth #1: you don’t need observability til you have problems of scale

First of all, there’s this misconception that observability is something you only need when you have really super duper hard problems, or that it’s only justified when you have microservices and large distributed systems or crazy scaling problems.  No, no no nononono. 

There may come a point where you are ABSOLUTELY FUCKED if you don’t have observability, but it is ALWAYS better to develop with it.  It is never not better to be able to see what the fuck you are doing!  The image in my head is of a hiker with one of those little headlamps on that lets them see where they’re putting their feet down.  Most teams are out there shipping opaque, poorly understood code blindly — shipping it out to systems which are themselves crap snowballs of opaque, poorly understood code. This is costly, dangerous, and extremely wasteful of engineering time.


Ever seen an engineering team of 200, and struggled to understand how the product could possibly need more than one or two teams of engineers? They’re all fighting with the crap snowball.

Developing software with observability is better at ANY scale.  It’s better for monoliths, it’s better for tiny one-person teams, it’s better for pre-production services, it’s better for literally everyone always.  The sooner and earlier you adopt it, the more compounding value you will reap over time, and the more of your engineers’ time will be devoted to forward progress and creating value.

Myth #2: observability is harder and more technically advanced than monitoring

Actually, it’s the opposite — it’s much easier.  If you sat a new grad down and asked them to instrument their code and debug a small problem, it would be fairly straightforward with observability. Observability speaks the native language of variables, functions and API endpoints, the mental model maps cleanly to the request path, and you can straightforwardly ask any question you can come up with. (A key tenet of observability is that it gives an engineer the ability to ask any question, without having had to anticipate it in advance.)

With metrics and logging libraries, on the other hand, it’s far more complicated.you have to make a bunch of awkward decisions about where to emit various types of statistics, and it is terrifyingly easy to make poor choices (with terminal performance implications for your code and/or the remote data source).  When asking questions, you are locked in to asking only the questions that you chose to ask a long time ago. You spend a lot of time translating the relationships between code and lowlevel systems resources, and since you can’t break down by users/apps you are blocked from asking the most straightforward and useful questions entirely!  

Doing it the old way Is. Fucking. Hard.  Doing it the newer way is actually much easier, save for the fact that it is, well, newer — and thus harder to google examples for copy-pasta. But if you’re saturated in decades of old school ops tooling, you may have some unlearning to do before observability seems obvious to you.

Myth #3: observability is a purely technical solution

To be clear, you can just add an observability tool to your stack and go on about your business — same old things, same old way, but now with high cardinality!

You can, but you shouldn’t.  

These are sociotechnical systems and they are best improved with sociotechnical solutions.  Tools are an absolutely necessary and inextricable part of it.  But so are on call rotations and the fundamental virtuous feedback loop of you build it, you run it.  So are code reviews, monitoring checks, alerts, escalations, and a blameless culture.  So are managers who allocate enough time away from the product roadmap to truly fix deep technical rifts and explosions, even when it’s inconvenient, so the engineers aren’t in constant monkeypatch mode.

I believe that observability is a prerequisite for any major effort to have saner systems, simply because it’s so powerful being able to see the impact of what you’ve done.  In the hands of a creative, dedicated team, simply wearing a headlamp can be transformational.

Observability is your five senses for production.

You’re right on the money when you ask if it’s about exploring production, but you could also use words that are even more basic, like “understanding” or “inspecting”.  Observability is to software systems as a debugger is to software code.  It shines a light on the black box.  It allows you to move much faster, with more confidence, and catch bugs much sooner in the lifecycle — before users have even noticed.  It rewards you for writing code that is easy to illuminate and understand in production.

So why isn’t everyone already doing it?  Well, making the leap isn’t frictionless.  There’s a minimal amount of instrumentation to learn (easier than people expect, but it’s nonzero) and then you need to learn to see your code through the lens of your own instrumentation.  You might need to refactor your use of older tools, such as metrics libraries, monitoring checks and log lines.  You’ll need to learn another query interface and how it behaves on your systems.  You might find yourself amending your code review and deploy processes a bit.  

Nothing too terrible, but it’s all new.  We hate changing our tool kits until absolutely fucking necessary.  Back at Parse/Facebook, I actually clung to my sed/awk/shell wizardry until I was professionally shamed into learning new ways when others began debugging shit faster than I could.  (I was used to being the debugger of last resort, so this really pissed me off.)  So I super get it!  So let’s talk about how to get your team aligned and hungry for change.

Okay okay okay already, how do I get my team on board?

If we were on the phone right now, I would be peppering you with a bunch of questions about your organization.  Who owns production?  Who is on call?  Who runs the software that devs write?  What is your deploy process, and how often does it get updated, and by who?  Does it have an owner?  What are the personalities of your senior folks, who made the decisions to invest in the current tools (and what are they), what motivates them, who are your most persuasive internal voices?  Etc.  Every team is different.  <3

There’s a virtuous feedback loop you need to hook up and kickstart and tweak here, where the people with the original intent in their heads (software engineers) are also informed and motivated, i.e. empowered to make the changes and personally impacted when things are broken. I recommend starting by putting your software engineers on call for production (if you haven’t).  This has a way of convincing even the toughest cases that they have a strong personal interest in quality and understandability. 

Pay attention to your feedback loop and the alignment of incentives, and make sure your teams are given enough time to actually fix the broken things, and motivation usually isn’t a problem.  (If it is, then perhaps another feedback loop is lacking: your engineers feeling sufficiently aligned with your users and their pain.  But that’s another post.)

Technical ownership over technical outcomes

I appreciate that you want your team to own the technical decisions.  I believe very strongly that this is the right way to go.  But it doesn’t mean you can’t have influence or impact, and particularly in times like this. 

It is literally your job to have your head up, scanning the horizon for opportunities and relevant threats.  It’s their job to be heads down, focusing on creating and delivering excellent work.  So it is absolutely appropriate for you to flag something like observability as both an opportunity and a potential threat, if ignored.

If I were in your situation and wanted my team to check out some technical concept, I might send around a great talk or two and ask folks to watch it, and then maybe schedule a lunchtime discussion.  Or I might invite a tech luminary in to talk with the team, give a presentation and answer their questions.  Or schedule a hack week to apply the concept to a current top problem, or something else of that nature.

But if I really wanted them to take it fucking seriously, I would put my thumb on the scale.  I would find myself a champion, load them up with context, and give them ample time and space to skill up, prototype, and eventually present to the team a set of recommendations.  (And I would stay in close contact with them throughout that period, to make sure they didn’t veer too far off course or lose sight of my goals.)

  1. Get a champion.

    Ideally you want to turn the person who is most invested in the old way of doing things — the person who owns the ELK cluster, say, or who was responsible for selecting the previous monitoring toolkit, or the goto person for ops questions — from your greatest obstacle into your proxy warrior.  This only works if you know that person is open-minded and secure enough to give it a fair shot & publicly change course, has sufficiently good technical judgment to evaluate and project into the future, and has the necessary clout with their peers.  If they don’t, or if they’re too afraid to buck consensus: pick someone else.

  2. Give them context.  

    Take them for a long walk.  Pour your heart and soul out to them.  Tell them what you’ve learned, what you’ve heard, what you hope it can do for you, what you fear will happen if you don’t.  It’s okay to get personal and to admit your uncertainties.  The more context they have, the better the chance they will come out with an outcome you are happy with.  Get them worried about the same things that worry you, get them excited about the same possibilities that excite you.  Give them a sense of the stakes. 

    And don’t forget to tell them why you are picking them — because they are listened to by their peers, because they are already expert in the problem area, because you trust their technical judgment and their ability to evaluate new things — all the reasons for picking them will translate well into the best kind of flattery — the true kind.  

  3. Give them a deadline.

    A week or two should be plenty.  Most likely, the decision is not going to be unilaterally theirs (this also gives you a bit of wiggle room should they come back going “ah no ELK is great forever and ever”), but their recommendations should carry serious weight with the team and technical leadership.  Make it clear what sort of outcome you would be very pleased with (e.g. a trial period for a new service) and what reasons you would find compelling for declining to pursue the project (i.e. your tech is unsupported, cost prohibitive, etc).  Ideally they should use this time to get real production data into the services they are testing out, so they can actually experience and weigh the benefits, not just read the marketing copy.

As a rule of thumb, I always assume that managers can’t convince engineers to do things: only other engineers can.  But what you can do instead is set up an engineer to be your champion.  And then just sit quietly in the corner, nodding, with an interested look on your face.

The nuclear option


if you <3 prod,
prod will <3 you back

You have one final option.  If there is no appropriate champion to be found, or insufficient time, or if you have sufficient trust with the team that you judge it the right thing to do: you can simply order them to do something your way.  This can feel squicky. It’s not a good habit to get into.  It usually results in things being done a bit slower, more reluctantly, more half-assedly. And you sacrifice some of your power every time you lean on your authority to get your team to do something.

But it’s just as bad for a leader to take it off the table entirely.

Sometimes you will see things they can’t.  If you cannot wield your power when circumstances call for it, then you don’t fucking have real power — you have unilaterally disarmed yourself, to the detriment of your org.  You can get away with this maybe twice a year, tops. 

But here’s the thing: if you order something to be done, and it turns out in the end that you were right?  You earn back all the power you expended on it plus interest.  If you were right, unquestionably right in the eyes of the team, they will respect you more for having laid down the law and made sure they did the right thing.

xo

charity

Some useful resources:

 

Questionable Advice #2: How Do I Get My Team Into Observability?

Questionable Advice: “After Being A Manager, Can I Be Happy As A Cog?”

One of my stretch goals for 2019 was to start writing an advice column.  I get a lot of questions about everything under the sun: observability, databases, career advice, management problems, what the best stack is for a startup, how to hire and interview, etc.  And while I enjoy this, having a high opinion of my own opinions and all, it doesn’t scale as well as writing essays.  I do have a (rather all-consuming) day job.

So I’d like to share some of the (edited and lightly anonymized) questions I get asked and some of the answers I have given.  With permission, of course.  And so, with great appreciation to my anonymous correspondent for letting me publish this, here is one.

Hi Charity,

I’ve been in tech for 25 years.  I don’t have a degree, but I worked my way up from menial jobs to engineering, and since then I have worked on some of the biggest sites in the world.  I have been offered a management role many times, but every time I refused.  Until about two years ago, when I said “fuck it, I’m almost 40; why not try.”

I took the job with boundless enthusiasm and motivation, because the team was honestly a mess.  We were building everything on-prem, and ops was constantly bullying developers over their supposed incompetence.  I had gone to conferences, listened to podcasts, and read enough blog posts that my head was full of “DevOps/CloudNative/ServiceOriented//You-build-it-you-run-it/ServantLeaders” idealism.  I knew I couldn’t make it any worse, and thought maybe, just maybe I could even make it better.softwarenegineeroncall_2 (1)

Soon after I took the job, though, there were company-wide layoffs.   It was not done well, and morale was low and sour.  People started leaving  for happier pastures.  But I stayed.  It was an interesting challenge, and I threw my heart and soul into it.

For two years I have stayed and grinded it out: recruiting (oh that is so hard), hiring, and then starting a migration to a cloud provider, and with the help of more and more people on the new team, slowly shifted the mindset of the whole engineering group to embrace devops best practices.  Now service teams own their code in production and are on-call for them, migrate themselves to the cloud with my team supporting them and building tools for them.  It is almost unrecognizable compared to where we were when I began managing.

A beautiful story isn’t it?  I hope you’re still reading.  🙂

Now I have to say that with my schedule full of 1:1s, budgeting, hiring, firing, publishing papers of mission statements and OKRs, shaping the teams, wielding influence, I realized that I enjoyed none of the above.  I read your 17 reasons not to be a manager, and I check so many boxes.  It is a pain in the ass to constantly listen to people’s egos, talk to them and keep everybody aligned (which obviously never happens).  And of course I am being crushed between top-down on-the-spot business decisions and bottom-up frustration of poorly executed engineering work under deadlines.  I am also destroyed by the mistrust and power games I am witnessing (or involved in, sometimes). while I long for collaboration and trust.  And of course when things go well my team gets all the praise, and when things go wrong I take all the blame.  I honestly don’t know how one can survive without the energy provided by praise and a sense of achievement.

All of the above makes me miss being an IC (Individual Contributor), where I could work for 8 hours straight without talking to anyone, build stuff, say what I wanted when I wanted, switch jobs if I wasn’t happy, and basically be a little shit like the ones you mention in your article.

Now you may say it’s obvious: I should find a new IC job in a healthier company.  You even wrote about it.  Going back to IC after two years of management is actually a good move.

But when I think about doing it, I get stuck.  I don’t know if I would be able to do it again, or if I could still enjoy it.  I’ve seen too many things, I’ve tasted what it’s like to be (sometimes) in control, and I did have a big impact on the company’s direction over time.  I like that.  If I went back to being an IC, I would feel small and meaningless, like just another cog in the machine.  And of course, being 40-ish, I will compete with all those 20-something smartasses who were born with kubernetes.

Thank you for reading.  Could you give me your thoughts on this?  In any case, it was good to get it off my chest.

Cheers,

Cog?

Dear Cog?,

Holy shitballs!  What an amazing story!  That is an incredible achievement in just two years, let alone as a rookie manager.  You deserve huge props for having the vision, the courage, and the tenacity to drive such a massive change through.

Of COURSE you’re feeling bored and restless.  You didn’t set out on a glorious quest for a life of updating mission statements and OKRs, balancing budgets, tending to people’s egos and fluffing their feelings, tweaking job descriptions, endless 1x1s and meetings meetings meetings, and the rest of the corporate middle manager’s portfolio.  You wanted something much bigger.  You wanted to change the world.  And you did!

But now you’ve done it.  What’s next?testinprod_3

First of all, YOUR COMPANY SUCKS.  You don’t once mention your leadership — where are they in all this?  If you had a good manager, they would be encouraging you and eagerly lining up a new and bigger role to keep you challenged and engaged at work.  They are not, so they don’t deserve you.  Fuck em.  Please leave.

Another thing I am hearing from you is, you harbor no secret desire to climb the managerial ranks at this time.  You don’t love the daily rhythms of management (believe it or not, some do); you crave novelty and mastery and advancement.  It sounds like you are willing to endure being a manager, so long as that is useful or required in order to tackle bigger and harder problems.  Nothing wrong with that!  But when the music stops, it’s time to move on.  Nobody should be saddled with a manager whose heart isn’t in the work.

You’re at the two year mark.  This is a pivotal moment, because it’s the beginning of the end of the time when you can easily slip back into technical work.  It will get harder and harder over the next 2-3 years, and at some point you will no longer have the option.

Picking up another technical role is the most strategic option, the one that maximizes your future opportunities as a technical leader.  But you do not seem excited by this option; instead you feel many complex and uncomfortable things.  It feels like going backwards.  It feels like losing ground.  It feels like ceding status and power.

“Management isn’t a promotion, it’s a career change.”

But if management is not a promotion, then going back to an engineering role should not feel like a demotion!  What the fuck?!

Imeplusprodt’s one thing to say that.  Whether it’s true or not is another question entirely, a question of policy and org dynamics.  The fact is that in most places, most of the power does go to the managers, and management IS a promotion.  Power flows naturally away from engineers and towards managers unless the org actively and vigorously pushes back on this tendency by explicitly allocating certain powers and responsibilities to other roles.

I’m betting your org doesn’t do this.  So yeah, going back to being an IC WILL be a step down in terms of your power and influence and ability to set the agenda.  That’s going to feel crappy, no question. We humans hate that.

Three points.

      1. You cannot go back to doing exactly what you did before, for the very simple reason that you are not the same person.  You are going to be attuned to power dynamics and ways of influencing that you never were before — and remember, leadership is primarily exercised through influence, not explicit authority.Senior ICs who have been managers are supremely powerful beings, who tend to wield outsize influence.  Smart managers will lean on them extensively for everything from shadow management and mentorship to advice, strategy, etc.  (Dumb managers don’t.  So find a smart manager who isn’t threatened by your experience.)
      2. You’re a short-timer here, remember?  Your company sucks.  You’re just renewing your technical skills and pulling a paycheck while finding a company that will treat you better, that is more aligned with your values.
      3. Lastly (and most importantly), I have a question.  Why did you need to become a manager in order to drive sweeping technical change over the past two years?  WHY couldn’t you have done it as a senior IC?  Shouldn’t technical people be responsible for technical decisions, and people managers responsible for people decisions?
        Could this be your next challenge, or part of it?  Could you go back to being an engineer, equipped with your shiny new powers of influence and mystical aura of recent management experience, and use it to organize the other senior ICs to assert their rightful ownership over technical decisions?  Could you use your newfound clout with leadership and upper management to convince them that this will help them recruit and retain better talent, and is a better way to run a technical org — for everyone?

     

I believe this is a better way, but I have only ever seen these changes happen when agitated for and demanded by the senior ICs.  If the senior ICs don’t assert their leadership, managers are unlikely to give it to them.  If managers try, but senior ICs don’t inhabit their power, eventually the managers just shrug and go back to making all the decisions.  That is why ultimately this is a change that must be driven and owned — at a minimum co-owned — by the senior individual contributors.Shared Joys, Kittens

I hope you can push back against that fear of being small and meaningless as an individual contributor.  The fact that it very often is this way, especially in strongly hierarchical organizations, does not mean that it has to be this way; and in healthy organizations it is not this way.  Command-and-control systems are not conducive to creative flourishing.  We have to fight the baggage of the authoritarian structures we inherited in order to make better ones.

Organizations are created afresh each and every day — not created for us, but by us.  Help create the organization you want to work at, where senior people are respected equally and have domains of ownership whether they manage people or technology.  If your current gig won’t value that labor, find one that will..

They exist.  And they want to hire you.

Lots of companies are DYING to hire this kind of senior IC, someone who is still hands on yet feels responsibility for the team as a whole, who knows the business side, who knows how to mentor and craft a culture and can herd cats when nec

There are companies that know how to use ICs at the strategic level, even executive level.  There are bosses who will see you not as a threat, but as a *huge asset* they can entrust with monumental work.

As a senior contributor who moves fluidly between roles, you are especially well-equipped to help shape a sociotechnical organization.  Could you make it your mission to model the kind of relationship you want to see between management and ICs, whichever side you happen to be on?  We need more people figuring out how to build organizations where management is not a promotion, just a change of career, and where going back and forth carries no baggage about promotions and demotions.  Help us.

And when you figure it out, please don’t keep it to yourself.  Expand your influence and share your findings by writing your experiences in blog posts, in articles, in talks.  Tell stories.  Show people people how much better it is this way.  Be so magnificently effective and mysteriously influential as a senior IC that all the baby engineers you work with want to grow up to be just like you.

Hope this helps.


charity

P.S. — Oh and stop fretting about “competing” with the 20-somethings kuberneteheads, you dork. You have been learning shit your whole career and you’ll learn this shit too.  The tech is the easy part.  The tech will always be the easy part.  🙂

Questionable Advice: “After Being A Manager, Can I Be Happy As A Cog?”

A Manager’s Bill of Responsibilities (and Rights)

Over a year and a half ago, I wrote up a post about the rights and responsibilities due any engineer at Honeycomb.  At the time we were in the middle of a growth spurt, had just hired several new engineers, and I was in the process of turning over day-to-day engmeme2engineering management over to Emily.  Writing things down helped me codify what I actually cared about, and helped keep us true to our principles as we grew.

Tacked on to the end of the post was a list of manager responsibilities, almost as an afterthought. Many people protested, “don’t managers get any rights??” (and naturally I snapped “NO!  hahahahahha”)

I always intended to circle back and write a followup post with the rights and responsibilities for managers.  But it wasn’t til recently, as we are gearing up for another hiring spurt and have expanded our managerial ranks, that it really felt like its time had come.

The time has come, the time is now, as marvin k. mooney once said.  Added the bill of rights, and updated and expanded the list of responsibilities.  Thanks Emily Nakashima for co-writing it with me.

 

Manager’s Bill of Rights

  1. You shall receive honest, courageous, timely feedback about yourself and your team, from your reports, your peers, and your leaders.  (No one is exempt from feeding the hungry hungry feedback hippo!  NOO ONNEEEE!)  🦛🦛🦛🦛🦛🦛🦛
  2. Management will be treated with the same respect and importance as individual work.  reviewmeme
  3. You have the final say over hiring, firing, and leveling decisions for your team.  It is expected that you solicit feedback from your team and peers and drive consensus where possible.  But in the end, the say is yours.
  4. Management can be draining, difficult work, even at places that do it well.  You will get tactical, strategic, and emotional support from other managers.
  5. You cannot take care of others unless you first practice self-care.  You damn well better take vacations.  (Real ones.)
  6. You have the right to personal development, career progression, and professional support.  We will retain a leadership coach for you.
  7. You do not have to be a manager if you do not want to.  No one will ever pressure you.

Manager’s Responsibilities

  • Recruit and hire and train your team. Foster a sense of solidarity and “teaminess” as well as real emotional safety.
  • Cultivate an inclusive culture and redistribute opportunity.  Fuck a pedigree.  Resist monoculture.
  • Care for the people on your team. Support them in their career trajectory, personal goals, work/life balance, and inter- and intra-team dynamics.
  • Keep an eye out for people on other teams who aren’t getting the support they need, and work with your leadership and manager peers to fix the situation. catplays
  • Give feedback early and often. Receive feedback gracefully. Always say the hard things, but say them with love.
  • Move us relentlessly forward, staying alert for rabbit-holing and work that doesn’t contribute to our goals. Ensure redundancy/coverage of critical areas.
  • Own the planning process for your team, be accountable for the goals you set. Allocate resources by communicating priorities and requesting support. Add focus or urgency where needed.
  • Own your time and attention. Be accessible. Actively manage your calendar. Try not to make your emotions everyone else’s problems (but do lean on your own manager and your peers for support).
  • Make your own personal growth and self-care a priority. Model the values and traits we want employees to pattern themselves after.
  • Stay vulnerable.

(Easier said than done, huh?)

<3 charity

Screen Shot 2019-10-30 at 8.04.07 AM

A Manager’s Bill of Responsibilities (and Rights)