Most people aren’t doing true CI/CD. Most teams wait far too long to get their code into prod after writing it. Most painful of all are the teams who have done all the hard parts — wired up continuous integration, achieved test coverage, etc — but still deploy by hand, thus depriving themselves of the payoff for their hard work.
Any time an engineer merges a diff back to main, this ought to trigger a full run of your CI/CD pipeline, culminating with an automatic deploy to production. This should happen once per mergeset, never batching multiple engineers’ diffs in a run, and it should be over and done with in 15 minutes or less with no human intervention.
Okay, but what if you don’t? How costly can it be, really?
Let’s do some back of the envelope math. First you’ll need to answer a couple questions about your org and deploy pipeline.
How many engineers do you have? ____________
How long typically elapses between when someone writes code and that code is live in production? _____________
Let (n) be the number of engineers it takes to efficiently build and run your product, assuming each set of changes will autodeploy individually in <15 min.
If changes typically ship on the order of hours, you need 2(n).
If changes ship on the order of days, you need 2(2(n)).
If changes ship on the order of weeks, you need 2(2(2(n)))
If changes ship on the order of months, you need 2(2(2(2(n))))
Your 6 person team with a consistent autodeploy loop would take 24 people to do the same amount of work, if it took days to deploy their changes. Your 10 person team that ships in weeks would need 80 people.
At cost to the company of approx 200k per engineer, that’s $3.6 million in the first example and $14 million in the second example. That’s how much your neglect of internal tools and kneejerk fear of autodeploy might be costing you.
It’s not just about engineers. The more delay you add into the process of building and shipping code, the more pathologies multiply, and you find yourselves needing to spend more and more time addressing those pathologies instead of making forward progress for the business. Longer diffs. Manual deploy processes. Bunching up multiple engineers’ diffs in a single deploy, then spending the rest of the day trying to figure out which one was at fault for the error.
Soon you need an SRE team to handle your reliability issues, build engineering specialists to build internal tools for all these engineers, managers to manage the teams, product folks to own the roadmap and project managers to coordinate all this blocking and waiting on each other…
You could have just fixed your build process. You could have just committed to continuous delivery. You would be moving more swiftly and confidently as a small, killer team than you ever could at your lumbering size.
✨✨15 minutes or bust✨✨
In 2021, how will *you* achieve the dream of CI/CD, and liberate your engineers from the shackles of pointless toil?
P.S. if you want to know my methodology for coming up with this equation, it’s called “pulled out of my ass because it sounded about right, then checked with about a dozen other technical folks to see if it aligned with their experience.”
A few weeks ago I got a question from Stephane Bjorne on twitter, about how to screen for bad managers and/or management culture.
Great question. Here are some questions to ask when interviewing to weed out bad manager culture. Ask multiple people; compare their answers.
🍸How did you become a manager? Do you enjoy it? (Never work for someone who was pressed into management, or doesn't seem happy.) https://t.co/Py5cEWLHMP
This is a great question. I’ve talked a lot about my philosophy for interviews, which boils down to equalizing the power dynamics as much as possible to reflect the reality that the candidate should be interviewing the company just as much as the reverse. You are all highly compensated subject matter experts who can find jobs relatively easily; there’s no reason all the judgment and critique should flow in a single direction.
But HOW? What questions can you ask, to get a feel for whether you will join this team and belatedly discover that you’re expected to be a jira ticket monkey, or that the manager is passive aggressive and won’t advocate for you or take a stand on anything?
Glad you asked.
First of all, backchannel like mad if you can. If you can’t, do ask the same question of multiple people and compare their answers. Pay particular attention to the different answers given by junior vs senior members of the team, and give extra weight to the experiences of any underrepresented minorities
Questions to consider asking the manager:
“How did you become a manager? Do you enjoy it?” Trust me, you never want to work for someone who was pushed into management against their will and still seems openly bitter about it.
“What do you like about your job?” Red flags include, “I was tired of being out of the loop and left out of decision-making processes.” That could be you next.
“Is management a promotion here, or a lateral move? Do people ever go back from managing to engineering? Is that considered a viable career path?”
“What kind of training do you offer new managers, and what are they evaluated on? What are YOU evaluated on?”
“Do you have a job ladder for individual contributors (ICs)? Can I see it? Do the IC levels track management levels all the way to the top, or top out at some point?”
“What does your review and promotion process look like?”
“Are your levels public or private? What is the distribution of engineers across levels, roughly?” Here you are looking for how high the IC track goes, and how many engineers exist at the upper bound. Distribution should be scant at the upper levels, roughly an order of magnitude less for each level after “senior engineer”. Do the written level descriptions seem reasonable and appealing to you? Do you see yourself in them, and feel like there is a path for growth?
“Do you have any junior or intermediate engineers, and how many? If not, why the fuck not?” Ask.
“How often do you have 1x1s with each of your direct reports? How often are the skip levels?”
And then there is the ur-question, which every one of you should ask in every single interview you ever have:
“What is the process by which someone ends up working on a particular project?” In other words, how does work get decided and allocated? Bad signs: they get flustered, don’t have a clear answer, you have no say, there is no product/design process, it’s all done via jira assignments.
Pay just as much attention to their body language and signals here as the answer they give you. Are they telling the truth, or describing their ideal world? Ask what happens when there are problems with the normal process, or how often it gets circumvented, or how you know if your work aligns with company strategy.
Questions to ask an engineer:
“What do you talk about in your 1x1s with your manager?”
“How often does your manager have career conversations with you, asking how you want to develop over the next few years?” Ideally at least a couple times a year, but really any answer is fine other than a blank stare.
“Does your manager care about you? Has working here moved your career forward? How so?”
“Have you ever been surprised by feedback you received in a review from your manager?” You should never be surprised.
Most of all, can you get the manager to tell you “no” on a thing you clearly want during the interview? (Maybe a level, or WFH schedule, or travel, etc.) Or are they squirmy, evasive, or hedging, or make you promises that they’ll look into it later, etc? Anyone who will look you straight in the eye and say “no, and here’s why”, is someone you can probably trust to be straight with you down the line, in good times and bad.
Depending on your level and career goals, it’s a good idea to ask questions to ascertain if the right gap exists for you to fill to reach your goals. Don’t join a team where five other people have the same exact skill sets as you do and are already eyeing the same role you want. (I wrote more on levels here.)
To figure out if they're a decent manager, all the above. And ask about specific situations. Any half-decent manager can BS on hypothetical stuff. – Who was the last person you promoted? Why/how? – How do you handle when X complains to you about Y? Can you give an example?
“Ask about specific situations, any half decent manager can BS on hypothetical stuff.”
“Who was the last person you promoted? Why/how?”
“How do you handle when X complains to you about Y? Can you give an example?”
“What was the best team you managed and why?”
“What is the biggest challenge the team currently has and why? What are you doing about it? Biggest strength?”
“How many people have left the team since you’ve been here?”
“Who was the last person to leave your team? Why did they leave?” (These leaving questions are tough to ask but will give you a lot of signal. Especially seeing how the manager frames it — are they playing victim, or owning up to it?)
Anyone who won’t be honest about their real personal weaknesses and struggles, probably isn’t someone you want to report to.
Thanks for the tag.
For me I want to hear from ICs: – when their manager as gone to bat for them – When their manager gave them hard to hear feedback (if its never, its a flag, all positive feedback = little growth) – when their manager coached them through a tricky situation
when their manager gave them hard to hear feedback (if it’s never, it’s a flag — all positive feedback = little growth)
when their manager coached them through a tricky situation
and from Managers:
when they went to bat for someone on their team
a team member’s growth they feel really proud of
when they got negative feedback about someone on their team and how it was handled
Obviously, you won’t be able to ask all of these questions in an hourlong slot. So ask a few, and lean in whenever they seem avoidant or uncomfortable — that’s where the juicy stuff comes from. And personally, I wouldn’t consider working somewhere that sees management as a promotion rather than a peer/support role, but that may be a high bar to clear in some parts of the world.
Good luck. Joining a team with a good manager can be one of the best ways to accelerate your career and open the door to opportunities, in part because it ensures healthy team dynamics. Workplace friction, bullying, harassment, passive aggressiveness, etc can be truly terrible, and even in the milder cases it’s still a huge distraction from your work and a big quality of life issue.
A manager’s One Job is to make sure that shit doesn’t happen. It’s really is worth trying to find a good one you can trust.
I found your blog (from Hacker News, I think) and your “Questionable Advice: The Trap of the Premature Senior” spoke to me, but I was wondering, do you have any followup advice on handling compensation in this situation? Should one be open to making less with this type of move, or is that not actually an issue?
You should ABSOLUTELY be open to making less. Consider it an investment in your long term career. Think of the extra money your company is paying you now as a hostage premium on top of your real market value. Staying isn’t what’s good for you, and that makes it hazard pay.
Your career is the single most valuable asset you have — it’s a multimillion dollar appreciating asset, and you should curate and guide it with an eye toward longevity. The first decade of your career is way, way too early to start optimizing for salary over experience.
This question was very relevant to me right now, as I recently spoke to a recruiter about a position that’s more in line with my career goals but pays less, and her immediate reaction was “don’t go backward on compensation.” But I can see the trade-off value in losing some comp to gain “better” experience.
Yeah, I think that’s terrible advice. 🙂 In general, if the compensation is fair and respectful, I think that is the absolute *worst* reason to make a job decision.
Salary is not a one-way escalator that you hop on after school, and gracefully exit at the peak upon retiring. It’s possible your recruiter’s advice was based on the assumption that your employers are likely to ask for your current salary and base their offers off of it. That used to be common practice, but is less and less so because of the fairness issues involved. Comp should be based on the value of your labor to the company, not your past comp, and in many states it is illegal to ask about your salary.
You may choose to take lower salaries at various times in your career — to work at a nonprofit or gov job, to learn new skills, in exchange for more flexibility or vacation time or titles or stock options, or as a result of moving to a different city or country.
I am not a financial advisor, but I am a big believer in retaining optionality. If the opportunity of your dreams came along with a starting salary of 150k, but every penny of your 170k salary is already committed, that’s a big loss of opportunity for you. This is a strong argument for trying to live well within your means, save religiously, and always have fuck you money in the bank. If you’re young and you get a salary bump, consider automatically diverting the raise into savings. Avoiding lifestyle inflation is the most painless way to save.
We have the unfathomable luxury of being incredibly well compensated for what we do. What is that luxury worth if we don’t use it to liberate ourselves, to facilitate happiness and fulfillment?
CI/CD pipeline, culminating with an automatic deploy to production. This should happen once per mergeset, never batching multiple engineers’ diffs in a run, and it should be over and done with in 15 minutes or less with no human intervention.
more and more time addressing those pathologies instead of making forward progress for the business. Longer diffs. Manual deploy processes. Bunching up multiple engineers’ diffs in a single deploy, then spending the rest of the day trying to figure out which one was at fault for the error.
