Most people aren’t doing true CI/CD. Most teams wait far too long to get their code into prod after writing it. Most painful of all are the teams who have done all the hard parts — wired up continuous integration, achieved test coverage, etc — but still deploy by hand, thus depriving themselves of the payoff for their hard work.
Any time an engineer merges a diff back to main, this ought to trigger a full run of your CI/CD pipeline, culminating with an automatic deploy to production. This should happen once per mergeset, never batching multiple engineers’ diffs in a run, and it should be over and done with in 15 minutes or less with no human intervention.
Okay, but what if you don’t? How costly can it be, really?
Let’s do some back of the envelope math. First you’ll need to answer a couple questions about your org and deploy pipeline.
How many engineers do you have? ____________
How long typically elapses between when someone writes code and that code is live in production? _____________
Let (n) be the number of engineers it takes to efficiently build and run your product, assuming each set of changes will autodeploy individually in <15 min.
If changes typically ship on the order of hours, you need 2(n).
If changes ship on the order of days, you need 2(2(n)).
If changes ship on the order of weeks, you need 2(2(2(n)))
If changes ship on the order of months, you need 2(2(2(2(n))))
Your 6 person team with a consistent autodeploy loop would take 24 people to do the same amount of work, if it took days to deploy their changes. Your 10 person team that ships in weeks would need 80 people.
At cost to the company of approx 200k per engineer, that’s $3.6 million in the first example and $14 million in the second example. That’s how much your neglect of internal tools and kneejerk fear of autodeploy might be costing you.
It’s not just about engineers. The more delay you add into the process of building and shipping code, the more pathologies multiply, and you find yourselves needing to spend more and more time addressing those pathologies instead of making forward progress for the business. Longer diffs. Manual deploy processes. Bunching up multiple engineers’ diffs in a single deploy, then spending the rest of the day trying to figure out which one was at fault for the error.
Soon you need an SRE team to handle your reliability issues, build engineering specialists to build internal tools for all these engineers, managers to manage the teams, product folks to own the roadmap and project managers to coordinate all this blocking and waiting on each other…
You could have just fixed your build process. You could have just committed to continuous delivery. You would be moving more swiftly and confidently as a small, killer team than you ever could at your lumbering size.
✨✨15 minutes or bust✨✨
In 2021, how will *you* achieve the dream of CI/CD, and liberate your engineers from the shackles of pointless toil?
P.S. if you want to know my methodology for coming up with this equation, it’s called “pulled out of my ass because it sounded about right, then checked with about a dozen other technical folks to see if it aligned with their experience.”
This is a great question. I’ve talked a lot about my philosophy for interviews, which boils down to equalizing the power dynamics as much as possible to reflect the reality that the candidate should be interviewing the company just as much as the reverse. You are all highly compensated subject matter experts who can find jobs relatively easily; there’s no reason all the judgment and critique should flow in a single direction.
But HOW? What questions can you ask, to get a feel for whether you will join this team and belatedly discover that you’re expected to be a jira ticket monkey, or that the manager is passive aggressive and won’t advocate for you or take a stand on anything?
Glad you asked.
First of all, backchannel like mad if you can. If you can’t, do ask the same question of multiple people and compare their answers. Pay particular attention to the different answers given by junior vs senior members of the team, and give extra weight to the experiences of any underrepresented minorities
Questions to consider asking the manager:
“How did you become a manager? Do you enjoy it?” Trust me, you never want to work for someone who was pushed into management against their will and still seems openly bitter about it.
“What do you like about your job?” Red flags include, “I was tired of being out of the loop and left out of decision-making processes.” That could be you next.
“Is management a promotion here, or a lateral move? Do people ever go back from managing to engineering? Is that considered a viable career path?”
“What kind of training do you offer new managers, and what are they evaluated on? What are YOU evaluated on?”
“Do you have a job ladder for individual contributors (ICs)? Can I see it? Do the IC levels track management levels all the way to the top, or top out at some point?”
“What does your review and promotion process look like?”
“Are your levels public or private? What is the distribution of engineers across levels, roughly?” Here you are looking for how high the IC track goes, and how many engineers exist at the upper bound. Distribution should be scant at the upper levels, roughly an order of magnitude less for each level after “senior engineer”. Do the written level descriptions seem reasonable and appealing to you? Do you see yourself in them, and feel like there is a path for growth?
“Do you have any junior or intermediate engineers, and how many? If not, why the fuck not?” Ask.
“How often do you have 1x1s with each of your direct reports? How often are the skip levels?”
And then there is the ur-question, which every one of you should ask in every single interview you ever have:
“What is the process by which someone ends up working on a particular project?” In other words, how does work get decided and allocated? Bad signs: they get flustered, don’t have a clear answer, you have no say, there is no product/design process, it’s all done via jira assignments.
Pay just as much attention to their body language and signals here as the answer they give you. Are they telling the truth, or describing their ideal world? Ask what happens when there are problems with the normal process, or how often it gets circumvented, or how you know if your work aligns with company strategy.
Questions to ask an engineer:
“What do you talk about in your 1x1s with your manager?”
“How often does your manager have career conversations with you, asking how you want to develop over the next few years?” Ideally at least a couple times a year, but really any answer is fine other than a blank stare.
“Does your manager care about you? Has working here moved your career forward? How so?”
“Have you ever been surprised by feedback you received in a review from your manager?” You should never be surprised.
Most of all, can you get the manager to tell you “no” on a thing you clearly want during the interview? (Maybe a level, or WFH schedule, or travel, etc.) Or are they squirmy, evasive, or hedging, or make you promises that they’ll look into it later, etc? Anyone who will look you straight in the eye and say “no, and here’s why”, is someone you can probably trust to be straight with you down the line, in good times and bad.
Depending on your level and career goals, it’s a good idea to ask questions to ascertain if the right gap exists for you to fill to reach your goals. Don’t join a team where five other people have the same exact skill sets as you do and are already eyeing the same role you want. (I wrote more on levels here.)
To figure out if they're a decent manager, all the above. And ask about specific situations. Any half-decent manager can BS on hypothetical stuff. – Who was the last person you promoted? Why/how? – How do you handle when X complains to you about Y? Can you give an example?
“Ask about specific situations, any half decent manager can BS on hypothetical stuff.”
“Who was the last person you promoted? Why/how?”
“How do you handle when X complains to you about Y? Can you give an example?”
“What was the best team you managed and why?”
“What is the biggest challenge the team currently has and why? What are you doing about it? Biggest strength?”
“How many people have left the team since you’ve been here?”
“Who was the last person to leave your team? Why did they leave?” (These leaving questions are tough to ask but will give you a lot of signal. Especially seeing how the manager frames it — are they playing victim, or owning up to it?)
Anyone who won’t be honest about their real personal weaknesses and struggles, probably isn’t someone you want to report to.
Thanks for the tag.
For me I want to hear from ICs: – when their manager as gone to bat for them – When their manager gave them hard to hear feedback (if its never, its a flag, all positive feedback = little growth) – when their manager coached them through a tricky situation
when their manager gave them hard to hear feedback (if it’s never, it’s a flag — all positive feedback = little growth)
when their manager coached them through a tricky situation
and from Managers:
when they went to bat for someone on their team
a team member’s growth they feel really proud of
when they got negative feedback about someone on their team and how it was handled
Obviously, you won’t be able to ask all of these questions in an hourlong slot. So ask a few, and lean in whenever they seem avoidant or uncomfortable — that’s where the juicy stuff comes from. And personally, I wouldn’t consider working somewhere that sees management as a promotion rather than a peer/support role, but that may be a high bar to clear in some parts of the world.
Good luck. Joining a team with a good manager can be one of the best ways to accelerate your career and open the door to opportunities, in part because it ensures healthy team dynamics. Workplace friction, bullying, harassment, passive aggressiveness, etc can be truly terrible, and even in the milder cases it’s still a huge distraction from your work and a big quality of life issue.
A manager’s One Job is to make sure that shit doesn’t happen. It’s really is worth trying to find a good one you can trust.
I found your blog (from Hacker News, I think) and your “Questionable Advice: The Trap of the Premature Senior” spoke to me, but I was wondering, do you have any followup advice on handling compensation in this situation? Should one be open to making less with this type of move, or is that not actually an issue?
You should ABSOLUTELY be open to making less. Consider it an investment in your long term career. Think of the extra money your company is paying you now as a hostage premium on top of your real market value. Staying isn’t what’s good for you, and that makes it hazard pay.
Your career is the single most valuable asset you have — it’s a multimillion dollar appreciating asset, and you should curate and guide it with an eye toward longevity. The first decade of your career is way, way too early to start optimizing for salary over experience.
This question was very relevant to me right now, as I recently spoke to a recruiter about a position that’s more in line with my career goals but pays less, and her immediate reaction was “don’t go backward on compensation.” But I can see the trade-off value in losing some comp to gain “better” experience.
Yeah, I think that’s terrible advice. 🙂 In general, if the compensation is fair and respectful, I think that is the absolute *worst* reason to make a job decision.
Salary is not a one-way escalator that you hop on after school, and gracefully exit at the peak upon retiring. It’s possible your recruiter’s advice was based on the assumption that your employers are likely to ask for your current salary and base their offers off of it. That used to be common practice, but is less and less so because of the fairness issues involved. Comp should be based on the value of your labor to the company, not your past comp, and in many states it is illegal to ask about your salary.
You may choose to take lower salaries at various times in your career — to work at a nonprofit or gov job, to learn new skills, in exchange for more flexibility or vacation time or titles or stock options, or as a result of moving to a different city or country.
I am not a financial advisor, but I am a big believer in retaining optionality. If the opportunity of your dreams came along with a starting salary of 150k, but every penny of your 170k salary is already committed, that’s a big loss of opportunity for you. This is a strong argument for trying to live well within your means, save religiously, and always have fuck you money in the bank. If you’re young and you get a salary bump, consider automatically diverting the raise into savings. Avoiding lifestyle inflation is the most painless way to save.
We have the unfathomable luxury of being incredibly well compensated for what we do. What is that luxury worth if we don’t use it to liberate ourselves, to facilitate happiness and fulfillment?
“How do I feel worthwhile as a manager when my people are doing all the implementing?”
— An Engineering Manager
Hey, real quick: how long have you been managing? If it’s less than two years, honey, the answer is “you don’t.” Your feelings about your performance don’t mean much in a new role. If you think you’re crushing it, you probably aren’t. But hey, if you think you’re screwing it up royally, you probably aren’t that either. ☺️
It took years for you to develop reliable instincts as an engineer, right? Then you switched careers and went right back to beginnerhood. That rarely feels good. So just don’t worry about it. Try not to obsess over how well you’re doing or not doing. Just engage your beginner brain, set phasers to “curiosity!” and actively pursue every learning opportunity for a year or two. Your judgment will improve. Give it time.
But experienced managers still struggle with this too. So if that’s you: let’s talk.
job satisfaction feels different for managers
First, let’s be clear: job satisfaction as a manager, should you find it, will feel very different than it did as an engineer. As an engineer, you get that very tactile sense of merging code, solving puzzles and incrementally pushing the business forward. It has a rhythm and a powerful drip, drip, drip of dopamine, and as a manager you will never ever feel that. Sorry! Some people eventually make peace with this, but many never do. No shame in that.
This is partly a function of time and proximity. Manager successes and failures play out over a much longer period of time than the successes and failures of writing and debugging code, and you can only indirectly trace your impact. It can be hard to draw a straight line from cause to effect. Some of your greatest successes may resonate and compound for years to come, yet the person might not remember, may never even have known how you contributed to their triumph. (Hell, you might not either.)
It is also related to your changing relationship with public credit and attribution. It is extremely poor form for managers to go around taking credit for things, so hopefully your org has a sturdy culture of celebrating the people doing the work and not their manager. But if you are used to receiving that stream of praise and recognition, it can be disorienting and deeply demoralizing when it dries up.
Most managers are unreliable narrators
There seems to be precisely one acceptable answer to the question of what motivates managers: loftily waxing on and on about how they get ALL their joy and fulfillment from empowering others and watching other people succeed without ever personally building anything tangible or receiving ANY of the credit. I call bullshit. (This bugs the ever-loving crap out of me.)
It reminds me of the self-abnegating monologues women are supposed to give about how amazing motherhood is, as they’re covered in vomit and haven’t slept in a week.
There is nothing wrong with wanting credit for your work, and affirmation and validation, and there is nothing inherently noble about not wanting those things. Whatever motivates you, motivates you. What matters is that you are self-aware about your needs, generous with credit, and conscious of who you lean on to get those needs met. Anyway, lots of people who become managers find themselves suddenly adrift and lacking reliable indicators about their job performance.
Part of becoming an effective manager over time is learning to recognize your own contributions and derive your own inner sense of worth. Nobody wants a needy manager. So here’s where I’d start: by locating your impact in the Really Big Stuff, the small personal moments, and any sort of crisis.
1 💜 The Really Big Stuff.
Are your users happy, and your business growing? Are you setting ambitious strategic goals and hitting them? Are your DORA metrics excellent? Are people happy to join your team and report to you? Are they awesome? Great, then you deserve some portion of the credit for that.
Most big shit is unfortunately only truly visible over much longer timelines, **but!** the longer you are a manager the more sensitive your feelers will become, the more they will pick up on subtle hints that betray deeper concerns. The sideways glance that suggests lack of trust, the offhand comment with an edge that sticks with you — those are fleeting clues which you may then delicately and expertly probe and use to disarm bad situations before they deteriorate or detonate.
(And sure, it can be really lovely to watch someone succeed when you know you had a small part to play. Congrats, you earned your salary. I just find it a little creepy and culty to act like this is what every manager must live for. There is nothing whatsoever wrong with you if living vicariously thru your reports’ successes doesn’t do it for you.)
2 💙 Random little moments.
On the flip side are the small and precious moments: did you just make someone feel supported in taking their mental health days? Did you pick up on someone’s anxiety and take a moment to check in them? Did they leave with a smile? Did you amplify someone’s voice, or help them work through a problem, or argue someone into receiving a well-deserved raise? Wielding your manager powers for good can be so easy and so gratifying.
(Seriously — give yourself a little pat on the back. This is the closest you’re going to get to a compliment most weeks. And nobody else is going to do it for you. 🤣)
3 💚 In a crisis.
Every manager will eventually encounter a crisis, and those are the moments that reveal the most about how well you have done your job. Do you have the credibility to speak for your team? Does your manager reach out for your support? Do your peers take you seriously or confide in your? Will people vouch for you? You’ll find out!
Not to put it too harshly, but in a clutch situation, are you a source of calm or are you often on the list of “situations to be managed”? Do you consistently tamp down drama and lower the stakes and the volume, or do you react in ways that amplify and escalate emotionally-charged situations? Do your feelings become other people’s problems? This reflects your ability to regulate your own feelings and emotional impulses under stress, and most of us quite overestimate our own power to self-regulate.
Go to therapy. Practice that mindfulness shit. Find what works for you, but pay attention to the energy you are contributing to any situation.
“Does it even matter if I come to work or not?”
A friend of mine was recently lamenting that it didn’t feel like it mattered if he came to work or missed all his 1x1s or not. What even was the point of showing up, as a manager?
In a way, he’s right. It shouldn’t matter if you’re out for a day, or a week. No single 1×1 should make or break something major, or you were already on terribly thin ice.
It is impossible to predict what the next crisis will be. All you can do in the meantime is keep your sociotechnical systems humming along and steadily work to improve them. Build good relationships and deepen the web of trust around you. Optimize your systems so that your people can spend as much of their time as possible on satisfying, high impact work. Make sure nobody is running ragged or being taken advantage of. Ensure redundancy and resiliency across all social and technical domains. Never stop learning. Keep your troops shiny.
Managerial value accrues over time. You can’t show up in the middle of a crisis and start fixing trust issues, any more than you can be a good coach who only shows up on game days. Train yourself to rejoice when things go smoothly in your absence (and really mean it).
In the end, your worth as a manager is seen in the trail you leave behind. The teams that got buy-in to achieve continuous delivery. The coworkers who fondly remember working together and recruit each other, or follow you from job to job for years. The way they saw you advocate for them. Set the bar high enough that their future managers will be compared to you. 🙃
If you are a good manager, you will rack up moments over the years that mean the world to you, heartwarming and vulnerable moments when people share the impact you’ve had on them. Treasure those like rare, unpredictable treats that they are, but don’t confuse them with fuel. It will never be enough to run on.
(And hey — if you’re just starting out, and all this sounds impossibly long-term? — never underestimate the value of just being fucking kind and generous and a pleasure to interact with. The job isn’t a popularity contest — the day will come when your effectiveness means the right people hating you. But that day is not today. And it is hard to be a good manager unless people genuinely enjoy talking to you and affirmatively want to help you meet your goals.)
Over the past couple of weeks I’ve been tweeting a LOT about lead time to deploy: the interval encompassing the time from when the code gets written and when it’s been deployed to production. Also described as “how long it takes you to run CI/CD.”
How important is this?
Here is a quickie thread from this week, or just go read “Accelerate” like everybody already should have. 🙃
It’s nigh impossible to have a high-performing team with a long lead time, and becomes drastically easier with a dramatically shorter lead time.
🌷 Shorter is always better. 🌻 One mergeset per deploy. 🌹 Deploy should be automatic.
And it should clock in under 15 minutes, all the way from “merging!” to “deployed!”.
Now some people will nod and agree here, and others freak the fuck out. “FIFTEEN MINUTES?” they squall, and begin accusing me of making things up or working for only very small companies. Nope, and nope. There are no magic tricks here, just high standards and good engineering, and the commitment to maintaining your goals quarter by quarter.
If you get CI/CD right, a lot of other critical functions, behaviors, and intuitions are aligned to be comfortably successful and correct with minimal effort. If you get it wrong, you will spend countless cycles chasing pathologies. It’s like choosing to eat your vegetables every day vs choosing a diet of cake and soda for fifty years, then playing whackamole with all the symptoms manifesting on your poor, mouldering body.
Is this ideal achievable for every team, on every stack, product, customer and regulatory environment in the world? No, I’m not being stupid or willfully blind. But I suggest pouring your time and creative energy into figuring out how closely you can approximate the ideal given what you have, instead of compiling all the reasons why you can’t achieve it.
Most of the people who tell me they can’t do this are quite wrong, turns out. And even if you can’t down to 15 minutes, ANY reduction in lead time will pay out massive, compounding, benefits to your team and adjacent teams forever and ever.
So — what was it you said you were working on right now, exactly? that was so important? 🤔
“Cutting my build time by 90%!” — you
So let’s get you started! Here, courtesy of my twitterfriends, is a long compiled list of Likely Suspects and CI/CD Offenders, a long list of anti-patterns, and some unresolved personal pain & suffering to hunt down and question when your build gets slow..
Order tests by time to execute and likelihood of failure.
Don’t run all tests, only tests affected by your change
Similarly, reduce build scope; if you only change front-end code, only build/test/deploy the front end, and for heaven’s sake don’t fuss with all the static asset generation
Don’t hop regions or zones any more than you absolutely must.
Prune and expire tests regularly, don’t wait for it to get Really Bad
Combine functionality of tests where possible — tests need regular massages and refactors too
Pipeline, pipeline, pipeline tests … with care and intention
You do not need multiple non-production environment in your CI/CD process. Push your artifacts to S3 and pull them down from production. Fight me on this
Pull is preferable to push. (see below)
Set a time elapsed target for your team, and give it some maintenance any time it slips by 25%
The usual suspects
tests that take several seconds to init
setup/teardown of databases (HINT try ramdisks)
importing test data, seeding databases, sometimes multiple times
rsyncing in parallel, all pulling from a single underprovisioned source
long git pulls (eg cloning whole repo each time)
CI rot (eg large historical build logs)
poor teardown (eg prior stuck builds still running, chewing CPU, or artifacts bloating over time
integration tests that spin up entire services (eg elasticsearch)
npm install taking 2-3 minutes
bundle install taking 5 minutes
resource starvation of CI/CD system
not using containerized build pipeline
Not properly separating the streams of “Our Software” (changes constantly) vs “infrastructure” (changes rarely)
running cloudformation to setup new load balancers, dbs, etc for an entire acceptance environment
docker pulls, image builds, docker pushes container spin up for tests
“Does this really go here?”
packaging large build artifacts into different format for distribution
slow static source code analysis tools
trying to clone production data back to staging, or reset dbs between runs
launching temp infra of sibling services for end-to-end tests, running canaries
selenium and other UX tests, transpiling and bundling assets
Selenium tests can be speedy, but the main problem I see is people using too many of the things. Ideally, they’re there to check something that nothing else can. *sigh*@aslak_hellesoy did a fun talk about architectures to improve test times @seleniumconfhttps://t.co/kSzwfNAmlC
entirely too large frontends that should be broken up into modules
“We regret to remind you that most AWS calls operate at the pace of ‘Infrastructure’, not ‘Software'”
AWS CodeBuild has several minutes of provisioning time before you’re even executing your own code — even a few distinct jobs in a pipeline and you might suffer 15 min of waiting for CodeBuild to do actual work
building a new AMI
spinning up EC2 nodes .. sequentially 😱
cool it with the AWS calls basically
- QA insists on writing all the tests themselves so it takes 2+hrs to run, can only run one at a time, and doesn’t cover most of the actual bug surface area
(yes, these are burning red flags for SEVERAL parts of the dev process, but there we were)
I’ve been at my current job for three years, and I am suddenly, accidentally, the most senior engineer on the team. I spend my days handling things like bootcamps, mentoring, architecture, and helping other engineers carve off meaningful work. This has taken a huge toll on the kind of work I want to do as an IC. I still enjoy writing and shipping features, and I am not a manager, but now I feel like I spend my days conducting meetings, interviewing, and unblocking others constantly instead of writing code myself.
What should I do? How can I deal with this situation in an effective manner? How can I keep from getting burned out on zoom? How can I reclaim more of my time to write code for myself, without sacrificing my influence? Should I get a new job? I have thought about going out and getting a new job, but I really like having a say at a high level. Here I get looped into all of the most important decisions and meetings. If I get a new job, how can I avoid starting over at the bottom of the heap and just taking assignments from other people? P.S., this is my first job.
Get a new job.
Yes, you will reset your seniority and have to earn it all over again. Yes, it will be uncomfortable and your ego will be cranky over it. Yes, you will be at the bottom of the heap and take assignments from other people for a while. Yes, you should do it anyway.
What you are experiencing now is the alluring comfort of premature seniority. You’re the smartest kid in the room, you know every corner of the system inside and out, you win every argument and anticipate every objection and you are part of every decision and you feel so deeply, pleasingly needed by the people around you.
It’s a trap.
Get the fuck out of there.
There is a world of distance between being expert in this system and being an actual expert in your chosen craft. The second is seniority; the first is merely .. familiarity
Deep down I think you know this, and feel a gnawing insecurity over your position; why else would you have emailed me? You were right. Treasure that uneasy feeling in your gut, that discomfort in the face of supreme comfortable-ness. It will lead you to a long and prosperous career as an engineer if you learn to trust it.
Think of every job like an escalator — a 50-foot high escalator that takes about two years to ride to the top. But once you’ve summited, you stall out. You can either stay and wander on that floor, or you can step to the left and pick another escalator and ride it up another 50 feet. And another.
In my mind, someone becomes a real senior engineer after they’ve done this about three times. 2-3 teams, stacks, languages, and roles, over a 5-8 year period, and then they’re solidly baked. There are insights you can derive from having seen problems solved in a few different ways that you can’t with only a single point of reference.
You don’t become a senior engineer at the 50-foot ascent, no matter how thoroughly you know the landscape. You become a senior engineer somewhere well over 100 feet, with a couple of lane changes under your belt.
The act of learning a new language and/or stack is itself an important skill. Experiencing how different orgs ship code in vastly different ways is how you internalize that there’s no one blessed path, only different sets of tradeoffs, and how you learn to reason about those tradeoffs.
And it is good for us to start over with beginner eyes. It’s humbling, it’s clarifying, it’s a cleanse for the soul. If you get too attached to feeling senior, to feeling necessary, you will undervalue the virtues of fresh eyes and questioning, of influence without authority. It is good for you to practice uncertainty and influencing others without the cheat codes of deep familiarity.
Nobody wants to work with seniors who clutch their authority with a white knuckled grip. We want to work with those who wear it lightly, who remember what it was like in our shoes.
Ultimately, this is a strong argument for building our teams behind a Rawlsian veil of ignorance concerning our own place in the pecking order. Starting fresh yourself will help you build teams where it is not miserable to be a beginner, where beginners’ contributions are recognized, where even beginners do not simply “take orders”, as you said. Because literally nobody wants that, including the beginners you are working with on your teams today.
After you have gotten a new job or two, and proven to yourself that you can level up again and master new stacks and technologies, that fretful inner voice questioning whether you deserve the respect you receive or not will calm down. You will have proven to yourself that your success wasn’t just a one-off, that you can be dropped into any situation, learn the local ropes and succeed. You will be a senior engineer.
There are few engineering topics that provoke as much heated commentary as oncall. Everybody has a strong opinion. So let me say straight up that there are few if any absolutes when it comes to doing this well; context is everything. What’s appropriate for a startup may not suit a larger team. Rules are made to be broken.
That said, I do have some feelings on the matter. Especially when it comes to the compact between engineering and management. Which is simply this:
It is engineering’s responsibility to be on call and own their code. It is management’s responsibility to make sure that on call does not suck. This is a handshake, it goes both ways, and if you do not hold up your end they should quit and leave you.
As for engineers who write code for 24×7 highly available services, it is a core part of their job is to support those services in production. (There are plenty of software jobs that do not involve building highly available services, for those who are offended by this.) Tossing it off to ops after tests pass is nothing but a thinly veiled form of engineering classism, and you can’t build high-performing systems by breaking up your feedback loops this way.
Someone needs to be responsible for your services in the off-hours. This cannot be an afterthought; it should play a prominent role in your hiring, team structure, and compensation decisions from the very start. These are decisions that define who you are and what you value as a team.
Some advice on how to organize your on call efforts, in no particular order.
It is easier to keep yourself from falling into an operational pit of doom than it is to claw your way out of one. Make good operational hygiene a priority from the start. Value good, clean, high-level abstractions that allow you to delegate large swaths of your infrastructure and operational burden to third parties who can do it better than you — serverless, AWS, *aaS, etc. Don’t fall into the trap of disrespecting operations engineering labor, it’s the only thing that can save you.
Invest in good release and deploy tooling. Make this part of your engineering roadmap, not something you find in the couch cushions. Get code into production within minutes after merging, and watch how many of your nightmares melt away or never happen.
Invest in good instrumentation and observability. Impress upon your engineers that their job is not done when tests pass; it is not done until they have watched users using their code in production. Promote an ownership mentality over the full software life cycle. This is how dev.to did it.
Construct your feedback loops thoughtfully. Try to alert the person who made the broken change directly. Never send an alert to someone who isn’t fully equipped and empowered to fix it.
When an engineer is on call, they are not responsible for normal project work — period. That time is sacred and devoted to fixing things, building tooling, and creating guard-rails to protect people from themselves. If nothing is on fire, the engineer can take the opportunity to fix whatever has been annoying them. Allow for plenty of agency and following one’s curiosity, wherever it may lead, and it will be a special treat.
Closely track how often your team gets alerted. Take ANY out-of-hours-alert seriously, and prioritize the work to fix it. Night time pages are heart attacks, not diabetes.
Consider joining the on call rotation yourself! If nothing else, generously pinch hit and be an eager and enthusiastic backup on the regular.
Reliability work and technical debt are not secondary to product work. Budget them into your roadmap, right alongside your features and fixes. Don’t plan so tightly that you have no flex for the unexpected. Don’t be afraid to push back on product and don’t neglect to sell it to your own bosses. People’s lives are in your hands; this is what you get paid to do.
Consider making after-hours on call fully-elective. Why not? What is keeping you from it? Fix those things. This is how Intercom did it.
Depending on your stage and available resources, consider compensating for it. This doesn’t have to be cash, it could be a Friday off the week after every on call rotation. The more established and funded a company you are, the more likely you should do this in order to surface the right incentives up the org chart.
Once you’ve dug yourself out of firefighting mode, invest in SLOs (Service Level Objectives). SLOs and observability are the mature way to get out of reactive mode and plan your engineering work based on tradeoffs and user impact.
I believe it is thoroughly possible to construct an on call rotation that is 100% opt-in, a badge of pride and accomplishment, something that brings meaning and mastery to people’s engineering roles and ties them emotionally to their users. I believe that being on call is something that you can genuinely look forward to.
But every single company is a unique complex sociotechnical snowflake. Flipping the script on whether on call is a burden or a blessing will require a unique solution, crafted to meet your specific needs and drawing on your specific history. It will require tinkering. It will take maintenance.
Above all: ✨RAISE YOUR STANDARDS✨ for what you expect from yourselves. Your greatest enemy is how easily you accept the status quo, and then make up excuses for why it is necessarily this way. You can do better. I know you can.
treat every alarm like a heart attack. _fix_ the motherfucker.
i do not care if this causes product development to screech to a halt. amortize it over a slightly longer period of time and it will more than pay for itself. https://t.co/JSck2u86ff
There is lots and lots of prior art out there when it comes to making on call work for you, and you should research it deeply. Watch some talks, read some pieces, talk to some people. But then you’ll have to strike out on your own and try something. Cargo-culting someone else’s solution is always the wrong answer.
Any asshole can write some code; owning and tending complex systems for the long run is the hard part. How you choose to shoulder this burden will be a deep reflection of your values and who you are as a team.
And if your on call experience is mandatory and severely life-impacting, and if you don’t take this dead seriously and fix it ASAP? I hope your team will leave you, and go find a place that truly values their time and sleep.
I keep talking to engineers who are frustrated that they aren’t leveling up faster, aren’t reaching senior levels as quickly as other people they know, feel stuck and don’t know how to get to the next level, etc. And I’ve begun to notice a common blind spot around leveling —
✨ not every opportunity exists ✨
✨ at every company ✨
✨ at every time.✨
Sure, if you’re a junior engineer, you should be able to level up to intermediate pretty much anywhere. But it gets progressively trickier after that. Even the path from intermediate to senior can depend on a number of situational variables:
Is there oxygen?
How many other senior engineers do you work with? how many other intermediate engineers around your level? All of these people will be pulling from the same bin of work, looking for promo-worthy, solidly-senior projects.
Does your ladder explicitly call for mentorship or leading small teams of lower-leveled engineers? Are there enough of those folks to go around?
Have you sufficiently wrapped up your last project well enough to move on? Was it actually completed in a way that demonstrated clear mastery and readiness for bigger and harder work, or did you leave a mess behind you? That may limit people’s appetite to take a risk on you with mission-critical projects.
What are the biggest needs of the business right now? Any process that generates projects ought to begin with this question before proceeding on to carve out a chunk that fits your promo desires, not the other way around. 🙃
Do you happen to work in a niche or specialty area of engineering, particularly one crammed with super-senior, world-famous highly leveled people? This can be fantastic when it comes to your ability to soak up knowledge from the world’s best, but it may simultaneously delay your ability to level up.
In short, is there oxygen at the next level? Does the company need more of the type of engineer you want to be, vs more of the type of engineer you are now? If they need more people pounding out code and fewer architects, they’re unlikely to want to promote you to a role that involves mostly architecture..
Literally no company can possibly make use of a top-heavy eng org stuffed with senior+ engineers, if all of them are expected to demonstrate company-wide impact or global impact every review period. There’s only so much high-level work to go around for every fifty engineers writing code and features and executing on those systems.
There is only so much oxygen at each level.
Of course, this is all assuming that your company takes leveling seriously. Most … really … don’t.
It’s tough. It’s tough to hold your ground when a valued engineer is complaining and dropping hints they may leave if they don’t get that promotion soon. It’s much easier to give in, make an exception, argue for rounding up.
This may sound good, but it is not ultimately in your best interests as that engineer. Seriously. </3
There is sooo much title inflation in this industry already. People are given the title “senior engineer” in just 3-5 years, need I say more??
If you let a little inflation into your system by making exceptions, it causes more trouble than it’s worth. Always. The only leverage you have when people try to get you to make exceptions is if you can honestly say, “no exceptions.” Give in just once, and your moral authority evaporates.
I would urge you not to make most, if any, career decisions based on levels or titles that are offered you. But I do understand how frustrating and infuriating it can be to be in a situation that is clearly unfair (usually because a manager got pressured into making an exception… tsk tsk), or if you don’t understand how to move your career forward.
So, here are a few strategic tips for leveling up.
Generalists level up faster than specialists.
When evaluating roles, choose ones where your specialty is part of their mission, or at least key to its execution. It has a far lower likelihood of getting outsourced, deprioritized, lacking investment in, or just forgotten about if what you do is core to what they do.
Always ask to see the job ladder when interviewing. If they hedge or fumble, don’t take that job.
Talk to your manager about the job ladder. Talk to your skip level about levels too! Managers love this shit. They can talk on and on and on about levels, long past your exhaustion point. It can be annoying, but it’s actually a sign of a good manager who cares and thinks about the edge cases in processes, and their impacts on people and teams.
That said, don’t take the ladder as a checklist to memorize or thing to be pored over and obsessed over. It’s an incomplete attempt at both shaping and reflecting relative impact. Focus on impact.
Is it easier to level up as a manager than as an engineer? Sorta-kinda, I guess so? There are at least two real phenomena at play here.
There are simply more roles to go around in the management track. You need like, what, 1-2 E7/E8 (or principal, or architect?) level engineers per 100-500 engineers, but several managers/directors/etc
Manager effectiveness is grounded in their relationships. It takes managers longer to have impact after they start a new role, but their potential impact grows and grows as their tenure gets longer. So yes, there’s a bit more of an escalator effect if you stay on the manager track at a company for several years. There is no similar escalator on the eng side; you have to be truly exceptional or truly lucky.
But it really depends on the organization.
It is much easier to level up quickly at fast-growing companies. When there is far more work than workers, and everyone is getting dropped in the deep end to sink or swim, you level up fast. Don’t underestimate what a stressful and awful experience this can be, though.
Many engineers get stuck on the bubble getting to senior because they are impatient and want a map. They just want someone to *tell them what to do*. Which is the very opposite of what a senior engineer does. 🙃 Develop your judgment around what needs to be done, and do it.
Your relationship with your manager matters. So does your ability to communicate about the work you are doing, its difficulty, its unexpected challenges and triumphs, etc. This is called “managing up”, and it is an actual skill which I am *terrible* at. So are most of you. 😉
TLDR, if leveling matters to you (and it should matter to everyone, to some extent!), then look curiously and critically around for opportunities, and seek to maximize them. Want to become an E6/E7? Probably don’t join a startup that doesn’t have any very high-level work to do, or already has more than enough people functioning at those levels and many more nipping their heels looking for the same opportunity.
This sort of thing is very obvious to us with the manager track (if you want to go from M->Dir, don’t join a startup that already HAS directors and managers who want to level up), but seems less obvious with engineering.
Most reasonable, non-desperate companies with options won’t hire you directly into the next level up which you haven’t done before, on either the manager or the engineer track. (Yellow flag if they do.)
But it is perfectly reasonable to express your career objectives in the interview, and make sure you’re on the same wavelength and seeing the same opportunities. Do you want to become a manager or a tech lead in a few months? Say so.
If it doesn’t exist now, do they think this opportunity may soon open up? Can they see a path forward for you there, if all goes well? Would they be interested in helping you get there? How many people may already be eyeing that same path? Is there enough opportunity for more than one? On what timeframe? Who will decide who gets the role, and how?
Engineers tend to find these conversations uncomfortable, and so they tend to avoid them because they don’t want to make the hiring manager uncomfortable by being pushy.
Relax. Managers don’t find this uncomfortable at all, it’s their bread and butter. (And even fi they do find it uncomfortable, tough beans.. it’s their job.) Ask away. ☺️
Misc notes on leveling.
P.S. Engineers seem to have a very sparse mental model of how leveling works, so here are a few more notes on how levels work at Honeycomb, which is adapted from conventions at Facebook/Google.
Each level after senior engineer (E5 for us) gets approx an order of magnitude harder to achieve, and an order of magnitude fewer engineers hold that title.
E5 is considered a “terminal level”, which sounds scary, but just means “you do not have to advance beyond this level.” If you never get promoted again, you won’t get fired either.
Whereas if you do not advance from E3-> E4 within 2 years, and E4->E5 within 3 years, you are automatically put on a performance improvement plan (at Facebook, I mean, not Honeycomb).
We (Honeycomb) hire into E5 as our highest level to start at, both because a) our interview process is not designed to let us parse differences between senior vs super-senior or super-duper senior, and b) we figure nobody is really able to come in the door with >E5 impact for the first 6 months anyway. So we can level them up quickly after they join and we get a feel for their work.
(I originally titled this article “Julian Dunn and the Case of the Bad Manager”, lol)
God, YES. This is something that has been on my queue of “topics to write about” for so long, and I haven’t because it’s just too big (and sometimes I tell myself, optimistically, it’s just too obvious?).
Julian’s point is that the reason so many bad managers persist is because it’s perceived as a promotion. Which means going back to engineering after managing is, ipso facto, a demotion. Which is really fucking hard to swallow. For anyone.
“If management isn’t a promotion, then returning to hands-on work isn’t a demotion, either. Right?”
There are a few separate points here which are worth unfurling separately.
Management is widely seen as a promotion
Management really does grant you some formal powers over your peers, which contributes to perceived hierarchy
Humans are hierarchical mammals, exquisitely sensitive to any loss of status — we hates it
But this is a cultural choice, not destiny. And we can change it.
Management is seen as a promotion
The notion that management is a promotion is deeply ingrained into our culture. It’s in language, pop culture, business books, any and all sources of career advice. If you became a manager and told your mom about it, she probably congratulated you and told you how proud she was. If you go out on a job interview, you’re expected to reach for the same rung or a higher one — or eyebrows will raise.
That’s a lot of cultural baggage to lean against. But I believe this is an idea whose time has come.
Any technical company should work hard to center and celebrate the work being done to build the product and make customers happy. Management is overhead, to be brutally frank about it, and we should not design organizations that would lead any rational, ambitious person to aspire to be overhead, should we?
The surest path to acclaim and glory (and promotions and raises) should be found through contributing. Not managing. Not being overhead.
… Because it mostly is a promotion, honestly
It is absolutely true that when you become a manager, you acquire new powers. As a tool of the org, you are granted certain powers to act on behalf of the organization, in exchange for being held accountable for certain outcomes.
These explicit powers often include hiring and firing decisions, access to privileged information, and making and meeting budgets.
But most of your powers aren’t formal at all. Most of your power comes from people listening more closely to what you say, giving your opinions more weight, and (consciously or subconsciously) just trying to please you, because they know you hold some influence over their career outcomes. It comes from the fact that so much information flows through managers. And finally, it comes from relationships — the strength of your personal relationships and mutual trust with other people throughout the org.
So how is this not a promotion? Well, it is a promotion at most companies, to be perfectly honest. But it does not have to be a promotion, if you acknowledge that these privileges and powers are accepted only by sacrificing other privileges and powers, and if you structurally allocate power to other roles. For example, you should acquire managerial powers only at the expense of technical decision-making powers.
I believe that the healthiest companies are ones where managerial powers are limited, enumerated, and minimal, with robust powers explicitly reserved for technical ICs. (much like the Constitution provides for Congress and the States, respectively.)
But it shouldn’t be. “Management” is a support role
Tech is a creative industry. Hierarchical leadership is a relic, a holdover from the days of manual labor. Hierarchy kills creativity, which leads to worse business outcomes.
Bad managers are a huge problem in tech. Just like Julian says, the wrong people are doing the job, for the wrong reasons, because they can’t to take the hit to the ego (and paycheck) of the demotion. This leads to unhappy teams and ultimately loss of talent.
I firmly believe that the engineer-manager pendulum is the way to build great technical leaders. The great line managers are never more than a few years removed from hands on work themselves, the great tech leads have always done a stint or two as a people manager. The promotion myth therefore both starves us of powerful technical leadership.and leaves us saddled with unhappy managers who have dwindling relevant skills, year after year.
The ladder is a trap. There are an order of magnitude fewer jobs for each rung you ascend. Meanwhile, the higher you climb the farther removed you are from the work most find meaningful (building things, making customers happy). The perception that you are a failure if you do anything but climb higher therefore traps a great many people in a cycle of intense anxiety and unhappiness.
Management is only one of many forms leadership can take. Yes, you have formal powers delegated to you on behalf of the org, but formal authority is the weakest form of power, and you should resort to using it rarely. Good leaders lead by influence and persuasion, weak leaders with “because I said so.”
Most engineers become managers to cope with org fuckery
Many people (like me!) become managers because they want access to the powers it gives them. As I argued in my last article, this is usually because they are frustrated with some organizational fuckery and it seems the only plausible way to fix or work around said fuckery is by becoming a manager.
Earlier this year I was having a 1×1 with one of our engineers, Martin Holman, who has been a manager before and had expressed interest in doing it again. So, I asked him, was he still interested?
He thought for a moment, and replied, “You know, I thought I wanted to be a manager again, I really did. But I think what I actually wanted was a seat at the table — to know what was going on, to have a say in what work I do. But I don’t feel out of the loop here. So it turns out I don’t feel any need to become a manager.”
Not only did that warm my heart, it answered a question I didn’t know I had. I think they would be a good manager, and should they change their mind again in the future, I will completely support them changing their mind again (minds change! it’s what they do!) — but I hope it is never because they feel that technical contributors are left out of the loop, or don’t have a say in what they do.
That’s what I’d call organizational fuckery.
A roadmap for changing your company culture
If “management is not a promotion” is a cultural value you would like to embrace at your company, here are some concrete actions you should take.
Make sure the pay bands for engineers and managers are equal, or even pay engineers more than managers of the same rank. (Slack does this, or used to.)
Have IC (individual contributor) levels for engineers that track management levels, all the way up to VP.
Look for ways to give high-level ICs information and opportunities for company impact that are on par with their people-manager counterparts.
Technical contributors should own and be accountable for technical strategy and decision-making, not managers.
Demystify management. Break it down into its constituent skills (giving feedback, running meetings, planning and budgeting, mentoring, running programs) and encourage everyone to develop those leadership skills.
Offer any management roles that may open up to internal transfers before considering external candidates.
Offer training and support for first-time managers who are undergoing that first career change. Offer engineers the same leadership coaching opportunities as managers.
Explicitly encourage managers to swing back to IC roles after two or three years. Support them through a generous grace period while refreshing their technical skills.
Watch your language. Loaded terms are everywhere, whether hierarchical (referring to people as being “above” others), or authoritarian (talking about bosses, managers). While it’s impossible to strip it from our vocabulary, it’s worth being thoughtful in how you represent reality, and using neutral phrases like “I support two teams” whenever possible.
Be explicit; repeat yourself. Say over and over that management is not a promotion, it is a change of career. Say it internally and externally, in your interview processes and recruiting messages. Educate your recruiting staff too (and be stern about it).
This isn’t a thing you can do once and be done with it; it’s an ongoing effort you must commit to. Managers tend to accrue power over time, like a gravitational force. In order to counterbalance this drift, managers need to consciously push power out to others. They must use their role as “information router” to inform and empower people to own decisions, instead of hoarding it for themselves.
Exactly, that’s my point… it won’t pass the “recruiter screen” because of antiquated perceptions around career progression being a monotonous increase up the manager -> director -> sr. director -> VP ladder
“Management is not a promotion” is my favorite bat signal
“Management is not a promotion, it’s a change of career.” I say this over and over again, even though it’s more aspirational than accurate.
Yet I say it anyway, because it’s a bat signal. It’s how the people I want to work with can find their way to me. And it repels the people I don’t want to work with just as efficiently.
When we recently posted our first-ever job req for an engineering manager, I included this under the list of optional skills:
You have worked as an engineering director or higher before, and decided to return to line management. Why? Because we value people who don’t blindly climb hierarchies just because they’re there. We value people who know themselves and what they find fulfilling in work and in life, and who can handle the hit to the ego that it takes to move “down” in pursuit of that fulfillment. Also, it would be interesting to talk about how you have solved org problems at other companies.
I cannot tell you how many amazing candidates zeroed in on that paragraph and came running. People who had been VPs before, been CTO, been director. People who were not only interested in becoming a line manager again, but were hungry to go back, to be closer to the people doing the work.
Something I heard them say again and again was, “People look at me like I’m crazy for wanting this,” “I have never had anyone see this as a strength.”
These were candidates who were acutely attuned to power dynamics, had exceptional self-knowledge, and who had seen and done so much to make organizations successful at multiple levels. What a set of superpowers!
Humans HAAAAAATE losing status.
We hate it. We hate it so bad. Even when we tell ourselves it’s what we wanted, even when we know it’s best for us, even when all the stars align. Something inside of us kicks and screams and feels excruciatingly attuned to the ripple effects of any status loss for a long time.
Like all such powerful irrational feelings, it’s evolution’s fault. Once upon a time it helped us survive and procreate. Now it’s just a nuisance, something to be worked around and minimized.
Where someone sits on the org chart should not determine that person’s ability to drive change, nor should their preference for tech problems or people problems. We need to see the work that engineers, managers, directors, VPs, and CxOs do as equally valuable and equally capable of prestige. We need to flip the org chart upside down, and treat “management” roles like the support systems they should be.
The work done by a database engineer is different from the work done by a VP marketing, or a director of database engineering. It is not inherently better or worse, easier or harder, more or less deserving of praise and admiration. It is simply different.
And we will have the best chance finding the work that brings the most meaning and joy to our lives if we can drain the hierarchical residue out of our perception of these roles, by flattening pay structures, equalizing power dynamics, and making sure everyone has the tools they need to do their job with as little hierarchical bullshit as possible.
 Martin said I could tell this story and use his name. I actually try to avoid talking about people, conversations, or anecdotes from Honeycomb as a more or less blanket rule, because I don’t want people to be perpetually on edge wondering if I am talking about them. (So if you’re wondering if I’m talking about you: I’m not. Unless I asked first.)
 Raise your hand if you’ve worked at a company where a DB engineer had a far greater impact on the bottom line some quarters than any of the VPs did. ✋
My company has recently begun pushing for us to build and staff out what I can only describe as “command centers”. They’re picturing graphs, dashboards…people sitting around watching their monitors all day just to find out which apps or teams are having issues. With your experience in monitoring and observability, and your opinions on teams supporting their own applications…do you think this sounds like a bad idea? What are things to watch out for, or some ways this might all go sideways?
Jesus motherfucking Christ on a stick. Is it 1995 where you work? That’s the only way I can try and read this plan like it makes sense.
It’s a giant waste of money and no, it won’t work. This path leads into a death spiral where alarms are going off constantly (yet somehow never actually catching the real problems), people getting burned out, and anyone competent will either a) leave or b) refuse to be on call. Sideways enough for you yet?
Snark aside, there are two foundational flaws with this plan.
1) watching graphs is pointless. You can automate that shit, remember? ✨Computers!✨ Furthermore, this whole monitoring-based approach will only ever help you find the known unknowns, the problems you already know to look for. But most of your actual problems will be unknown unknowns, the ones you don’t know about yet.
2) those people watching the graphs… When something goes wrong, what exactly can they do about it? The answer, unfortunately, is “not much”. The only people who can swiftly diagnose and fix complex systems issues are the people who build and maintain those systems, and those people are busy building and maintaining, not watching graphs.
That extra human layer is worse than useless; it is actively harmful. By insulating developers from the consequences of their actions, you are concealing from them the information they need to understand the consequences of their actions. You are interfering with the most basic of feedback loops and causing it to malfunction.
The best time to find a bug is as soon as possible after writing it, while it’s all fresh in your head. If you let it fester for days, weeks, or months, it will be exponentially more challenging to find and solve. And the best people to find those bugs are the people who wrote them
Helpful? Hope so. Good luck. And if they implement this anyway — leave. You deserve to work for a company that won’t waste your fucking time.