Engineering Manager Archetypes and Career Paths

Cross-posted from leaddev.com

Exploring seven common leadership scenarios.

If you’re looking for a very traditional breakdown of engineering manager archetypes – tech lead manager, team manager, director, and so forth – you can’t do better than Will Larson’s post, and I won’t try. I enthusiastically endorse everything he says, especially about Tech Lead Management roles being mostly a trap.

This post aims to capture a different set of archetypes: the common inflection points in a senior engineering manager’s career trajectory, what contributes to these, and some recommended next steps.

I’m of the opinion that career ‘planning’ is mostly overrated. There are just so many variables that go into what makes you feel happy, challenged, and content over the course of your life, and trying to predict how you’ll feel more than two or three years down the line is a fool’s game. You might get married, have kids, deal with immigration status or work visas, or family or health issues that suddenly throw a wrench in things. The company you work at might get acquired, go public, make some bad hires, get pummeled by changing market conditions or an economic downturn, or reorg you under somebody you absolutely loathe. The plum role you want might go to someone else. You might hitch your career to a technology that takes off like a rocket, or sputters out quietly, or suffers a series of community scandals.

Everything fails sooner or later, and most fail sooner rather than later.

And then there’s the fact that people who do plan their careers tend to plan in terms of hierarchy. They are eager to ‘make management’. Then they’re hungry to manage managers. They read ‘30 under 30’ lists and obsessively track which of their peers has climbed the ladder to director or vice president first, and their heart swells in tune with their headcount growth.

Few of us are completely immune to the siren song of such external signifiers of success. (Everybody’s mom congratulates them when they move into management – this shit runs deep in our culture.) But for most of us, nothing breaks the spell like spending time in those upper-hierarchical roles.

When you’re an engineer, it seems like managers hold all the power. Two or three years into a managerial role, you should be thoroughly cured of that illusion – aware of the constraints managers operate under and the multiple stakeholders they serve, but also newly appreciative of the powers and freedoms held by engineers, and attuned to how wielding influence and marshaling results may have little if anything to do with formal decision-making powers.

So, if you’re an ambitious person who wants a long, happy, rewarding career in technology, what’s the alternative?

Self-knowledge. Never stop working on self-awareness, being honest with yourself and others. Try to understand what truly matters to you – is it status and public acclaim? Is it spending half your life in the wilderness? Do you have a singular passion for compilers? Whatever it is, expect it to shift over the years.

  1. Do set goals for yourself – know what you want to achieve over the next few years, and have a hazy idea of where your nose is pointing five years from now.
  2. Whenever you don’t feel strongly compelled to make a particular move, act to preserve or expand optionality.

Don’t get comfortable. Once you’ve been doing any job for two or three years, it’s time for a change.

Let’s examine some archetypes and what to do if one of these applies to you. Keep in mind that these are just rules of thumb, and rules are meant to be broken. If your heart is pulling you in one direction, by all means, follow it.

The junior engineer who became an early manager

If you have worked as an engineer for less than seven-ish years before becoming a manager, you’re in risky territory. You’ll lose your fluency very quickly, and find it much harder to go back to engineering than if you had waited until you were more solidly senior. This can feel like a big compliment – and it is! – but this is your long-term career we’re talking about, not theirs, and too often those who progress early in their career aren’t warned about the downsides and the loss of optionality.

You’ll probably be fine as long as you stay at that same company, given that you know the systems well and management is all about relationships, but you may struggle when you try to find your next job, and you may also struggle finding someone willing to hire you as an engineer when you’re rusty. Technical skills buy credibility.

It’s worth noting that this seems to disproportionately happen to women, who are tapped for management because they are perceived as having better social skills, and then suffer a double penalty when they are rusty and judged ‘not technical’. If you’re a young woman, be extra cautious – make sure you ardently guard and grow your technical credibility.

The novice manager who wants to go back to engineering, six months in

When you’re deciding whether to be a manager or not, think of it as a tour of duty that will last for two to three years. Can you commit to that? If not, this probably isn’t the right time for you to try it out. If you aren’t sure, or you don’t feel like you have any idea what the job entails, don’t accept. Find ways to level up at managerial skills without taking on full responsibility for a team.

This might include taking ownership over a hiring loop – defining the role, writing a posting, designing the interview loop, training the interviewers, screening the candidates, adopting an intern or two (or even running the intern program), updating or open-sourcing your engineering job ladder, subbing for a manager while they’re out on paternity leave for a few months – any number of things. It’s okay to say no, and far better than saying yes, then backing out.

Why? First and foremost, it’s really disruptive to the team, and not fair to them. Secondly, you need to be prepared to shelve your own opinion of your work for the next two years and just keep doggedly plugging away, doing your best to learn and taking loads of feedback, without worrying too much about whether you personally think you’re doing a good job or not. You can’t base your self-esteem on how well you think you’re doing, because frankly, you’re not to be trusted. You have to gain a lot of experience and rewire a lot of neural pathways before you can once again trust your own inner voice on the topic of your job performance.

Management isn’t a promotion, it’s a change of career. And if you aren’t up for resetting the dial to ‘NOOB’ for a couple years, with all of the ups and downs, anxieties and discombobulations, then you aren’t up for management.

The manager who wants to go back to engineering, but only temporarily

One thing I hear surprisingly often from people is that they’re anxious that they’ll never get another shot at management if they step away from it – to take a new job at a new company, for instance, or to solidify their senior engineering skills. But they like management! – and are afraid of getting ‘stuck’ back in engineering.

It’s true: there are no guarantees in life. But in my experience, this fear is particularly overblown. There is a chronic shortage of good engineering managers, especially ones who genuinely enjoy the work. It’s pretty obvious when someone has the engineering manager’s skill set in their tool box; it permeates how you do your job, how you interact with your coworkers, and how you get things done. This means that it’s only a matter of time before you get tapped again. (And again.)

Engineers who have been managers tend to bat the question away over and over again – job after job, year after year. It takes more effort not to be a manager once you’re capable of it.

The manager who was forced into it and wants out…has wanted out for years

This is a tricky one, because it comes in two different guises. Sometimes it’s the person who got pushed into management unwillingly, or had a really rough time of it for a while, or maybe they have never worked in an environment where management was spoken of respectfully. For whatever reason, they still openly grump and groan about it despite the fact that they have actually come around and quite like the job now, and don’t actually want to go back to engineering after all.

Other times, the person got pushed into management unwillingly and genuinely dislikes it. Perhaps they were better as an engineer than they are as a manager, or maybe they simply never grew to love the management role enough to compensate for what they miss about engineering. If they’ve been complaining about it for years, and haven’t done anything about it, there are a few possible reasons: 1) they are genuinely getting pressured by their own boss and the organization to stay in their current role 2) they can’t bring themselves to give up the salary, the control, or the perceived status 3) there’s some other fear holding them back – for example, maybe they’re afraid it’s been too long and they’re too rusty.

In the first case, you should stay a manager, by all means, but own up to the fact that you enjoy it and find it rewarding. Even if you didn’t initially choose the gig, you’re choosing it now, so shut the hell up about being forced into it. Nobody wants to report to a manager who doesn’t want to be there, or is doing the job begrudgingly. Your complaining is toxic, and makes it impossible for you to coach others to have a healthy relationship with their roles.

In the latter case, if you genuinely don’t enjoy your job, then get the hell out. If it’s been years, then you have all the data you need. Your boss has no place forcing you into doing something you hate. You’re never going to do as well at a job you don’t enjoy as one that you do, so ultimately it’s a career-limiting move to stay where you are. Think long and hard about the distorting role that hierarchy has played in your life, and find a place where you are respected and valued for doing what you love.

Personally, I think the worst reason of all for not wanting to give up management is because it means giving up control. How do you think everyone else on your team feels? The good news is that you are ideally positioned to go back to engineering and model what a healthy power balance should look like between management and engineering – to demand transparency and model autonomy and ownership as a senior engineer.

The manager who has been managing for several years … now what?

If you’re a line manager who has been doing the job for around five years, and you feel pretty confident in your craft … what’s next?

That’s a great question. As you know, your technical credibility is grounded to some extent in your ability to do the same work as your team. So one question to ask yourself is, how wobbly are you on that front? Some managers manage to keep a hand in (but out of the critical path) the whole time; most managers do not. There is no right or wrong answer here, only an honest assessment of where you’re at. If you’re rusty, and you want to keep doing line management, it’s probably time to go back to the well for at least six months work as an engineer.

(This could be a GREAT opportunity to swap places with someone who’s questioning whether or not to be a manager full time.)

That’s definitely the choice that preserves the most optionality for you career-wise. It ensures you will still be hireable as an engineer or an engineering manager anywhere, regardless of how much technical knowledge the interview requires (and it does run the gamut, up to and including places that make you work as an engineer for six months before assuming your managerial role).

But most managers are at least slightly curious about what it would be like to move up the chain, or expand their managerial repertoire. This might be: managing teams where you are not the technical domain expert, taking on more of a hybrid product manager role, managing multiple teams, spinning off one or more subteams and managing those managers in addition to your primary team, or managing managers as a director.

Here your choices will likely be constrained by opportunities, which means that the company you work for matters A LOT. At a company with hundreds of engineers and high growth, there’s usually a nonstop trickle of reorgs, reteaming, and other opportunities opening up. If you work at a smaller company (or a place with less growth) and your bosses show no interest in leaving soon, you may need to switch companies to seek out those opportunities.

If this is important to you, you may want to consider switching earlier rather than later. You should pay extra attention to managing ‘out’ and ‘up’; building close relationships with your peers and those above you, in other words. Don’t be afraid to open your mouth and state your goals, and ask for your manager’s advice in getting there. People don’t get randomly tapped for these opportunities so much as they share their goals and are then grown into those positions over time.

The engineer who wants to be a manager, but hasn’t had the opportunity

The most demystifying thing about management is actually being a manager. To those who haven’t had the chance yet – or worse, who have seen colleague after colleague tapped for opportunities while being repeatedly overlooked– that can be cold comfort, or even infuriating to hear. If you want a shot at management, and haven’t had one yet: why not? What should you do or try differently?

Every circumstance is different, but I can offer some suggestions.

First of all, my personal belief is that in an ideal world, any senior engineer who wants to give management a try (and who demonstrates sufficient self-awareness, emotional intelligence, and respect for their colleagues), should get a shot at it. I also believe managers and engineers should both have parallel career ladders, should make equal salaries across their bands, and that technical leadership should be the purview of technical contributors. Leadership is not synonymous with management, and the more we can drain the relationships of hierarchy, the better people will be equipped to find the work that they love and find most fulfilling.

I live in the real world, so I know that most places don’t operate this way (although I believe the number is growing).

I also think that engineers tend to systematically underestimate and undervalue their own power, and they fail to inhabit and flex the power they should have. This leaves a vacuum, so managers step in to fill by default.

All that said. First, look at your company. How many opportunities are there for new managers, really? Is it too small, or not growing fast enough, or do you work in a specialty niche? Most companies aren’t bristling with new opportunities; if this is important to you, you should go work for a company that is growing fast, and you should state your ambition from the outset – yes, to the recruiter or hiring manager in the interview process.

Second, make friends with your manager and the other managers, and eagerly absorb as many managerial tasks or responsibilities as you can. Be more concerned with learning the skills and gaining the experience than with the title itself.

Third, work on your communication skills, written and spoken. Consider doing some public speaking. Run a workshop for your coworkers in something you know well. Take good meeting notes, write great technical project docs.

Fourth, ask your manager (and any other trusted senior people you work with) what you’re missing. Make it clear that you want to hear honest feedback, even if it hurts. If there’s something holding you back that you don’t know about, and someone is willing to tell you, that’s an incredible gift.

Fifth, educate yourself about diversity and inclusion issues. Stand up for others on your team who are getting talked over or having their ideas stolen. Be a leader among your peers, be someone who is willing to do what is right despite the temporary social costs.

The senior manager, director or vice president who daydreams about being an engineer again

Very often, people who were in the right place at the right time and climbed the ladder rapidly, get to the top and realize they feel restless and empty inside – they miss the intrinsic stimulation of writing code, solving problems, delighting users. Very few of them have the courage to go back.

It’s a true fact: the higher you climb the ladder, the further removed you are from the work that brings most people intrinsic joy, that feels real. Some manage to find dopamine hits in their work as a director or a vice president, but the dose is usually fainter and always on a delay.

It’s also a true fact that you’d have to go from being atop the monkey pyramid to being just another individual contributor, slinging code in the mines with the rest of them. That’s a lot for an ego to take. And it’s rough to pick up the skills again if you’ve been ten years or more away from writing code on a daily basis. Lots of people who are in their 30s, 40s or 50s feel like they have lost the mental agility to do so and therefore it might not even be an option anymore.

I don’t have an enormous sample size, but what I do have suggests the above is bullshit.

Consider my friend Molly Stamos, who was a software engineer from 1997-2001, then rapidly climbed the corporate ladder working in product, as a director, then as a vice president. She actually joined Honeycomb as our vice president of customer success, but she was burnt, and admitted to herself shortly after that she deeply missed software engineering and that was where her heart still lay. She worked in support for a while and picked up tasks from the backlog, and formally joined the engineering team in 2020, almost 20 years after she stopped coding full time. She’s freaking great at it and says she’s never been happier.

Be like Molly. Life’s too short to be miserable at work.

And P.S. most of the prestige is in your head. If you go from being an exec to being a senior engineer, I guarantee you that lots of people, especially other engineers, will actually respect you tons more.

Engineering Manager Archetypes and Career Paths

Career Development For Engineering Managers

(Cross-posted from leaddev.com)

What comes to mind when you hear the words, ‘career development for managers’?

If you’re like me, it’s one of two things: either obsessively climbing the ladder or those awful mandatory ‘trainings’ that get hastily assembled. (Otherwise known as, ‘How can we have hired so many brilliant engineers, yet none of them seem to have any idea what appropriate professional behavior looks like – and how long till we get sued?!’)

No one would claim that we have things like engineering roles, levels, and career progression locked down, but it all looks quite mature and well-understood compared to the Wild Wild West that is engineering management. Engineering at least has some basic skills (data structures, algorithms etc.) that one can study and structure an interview around, whereas for managers those basic skills (communication, relationships) come from ‘being human’.

To be fair, you can (and should!) work very hard on being a better human. All of us should – our entire life! Skills like communication, empathy, discipline and self-mastery, emotional self-regulation, self-awareness, and self-acceptance are a huge part of being an effective leader.

But let’s say you know all that. You’re a manager, you were dropped into this gig with precious little guidance (most of it contradictory), and all you want (for heaven’s sake!)  is some straightforward advice on how to grow and become a better one.

Perfect. Let’s get started.

With great power comes the responsibility to understand power dynamics

‘I totally support women/LGBTQIA+/POC in tech, but come on, it’s not as bad as some of them keep going on about. It can’t be, or I would have seen it myself by now. I’ve been in tech for ten years, and we’ve always treated every woman just the same as the guys. Besides, you know, there are two sides to every story.’

Does that sound like you? If so, please don’t become anyone’s manager until you have done a lot of listening and learning.

It’s one thing to be unaware of power dynamics and structural inequities as an individual contributor, and another when you wield formal hiring and firing powers on behalf of the org. You can really mess people up that way, and it doesn’t matter if the harm came from your ignorance instead of malice.

Build a trusted peer group of other managers and talk regularly

Get yourself some peers. You need a personal circle of other managers for mutual support, swapping stories, and talking through tricky interpersonal scenarios with strict confidentiality. You need to know these people well enough, and they need to know you and the personalities in your life well enough, that when something is bothering you it won’t take an hour of preamble just to set the stage for the story you need to tell.

I enjoyed my peer circle as an engineer. I couldn’t survive without them as a manager.

It can take some time to amass the right trusted circle, and it will shift with time, as some connections drift away and new ones emerge. Building, nurturing, and curating your circle is a continuous background process.

And it’s the number one thing you can do for your managerial career.

Give as much as you have been given

If you went to university, don’t let it fool you: our profession is fundamentally a web of apprenticeships.

This is why you should be extra generous with your time; everything you learned came from someone else.

Teach, explain, and tell stories about your own successes and failures as a manager. Helping somebody else with their problem has a way of giving you fresh eyes for your own problems, and your skills become more firmly ingrained after you teach them to others.

Invest in your public speaking skills and broadcasting skills

If public speaking terrifies you, lean into that fear and crush it like a bug. Verbal fluency in front of a group is a learnable skill, just like any other. And as a manager, you will draw on this skill 20 times a day, whether it’s running efficient meetings, holding people’s attention, representing your team well, etc.

I used to be catastrophically terrified of speaking in front of people. Back in 2015, I had to give my first internal infrastructure talk in front of maybe eight people, and for weeks in advance I woke up almost every night, sick and clammy from having nightmares about it.

Personally, I have found that a combination of practice, experience, and sometimes using anti-anxiety medication has made this an easier journey for me.

Get a therapist

Get a therapist. Get a work coach too, if work will pay for it. Yes, it’s good to work on your self-awareness, your relationship patterns, etc., but it’s also super useful to learn new metaphors and descriptive phrases from people who do this for a living. The richer your descriptive arsenal, the wider your range when connecting with people.

If you’re reluctant to do this, ask yourself honestly: couldn’t you use some better coping mechanisms and tools for emotional regulation? (Couldn’t everyone?)

Such a huge part of being a manager is making sure you don’t turn your problems into other people’s problems, or make your feelings other people’s problems. And that’s hard. It’s hard, and the ground is constantly shifting beneath your feet. Just when you think you’ve got it down, things change again.

Do a tour of duty as a manager at a big company

I list this one somewhat begrudgingly, but it’s a big one. If you really want to accelerate your career as a manager and learn lots of core skills fast: consider going to work at a Google/Facebook/Amazon, or any large and relatively faster-paced (or extremely fast-growing) tech company.

I learned so, so, so much about management (and organizational politics, and bureaucracies, and formal and informal power structures) at Facebook.

None of it was what I wanted to learn, mind you, but in retrospect it turned out to be pretty darn invaluable.

These big tech companies have their flaws, but they aren’t stupid, and they have invested a lot more into sophisticated business operations than any startup. Startups get away with so, so, so much sloppy behavior from a management and human resources perspective.

Managing a team at a big company for a year is a killer crash course in the best practices and established defaults of engineering management. If you can’t or won’t go work for a big company, try Will Larson’s book An Elegant Puzzle, which outlines a lot of the same best practices and the reasons behind them.

Overcome your fear of confrontation

If you hate confrontation, I’m sorry honey, but you have to push through that barrier ASAP. It is the one thing that will hold you back, poison your relationships, result in loss of trust, and ultimately hurt your direct reports faster than anything else. Read Radical Candor, practice with your therapist, practice with your group of peers, do whatever it takes. You have got to learn to lean into that itchy feeling and open your mouth instead of shrinking away.

Experiment with your 1:1s

Learn to have better, more interesting, more fulfilling, more varied 1:1s with your reports. This is a tough one for introverts, but it’s super worth investing some time into. I will leave ‘how’ as an exercise for the reader – there are lots of interesting blog posts out there, or you could start picking the brains of other managers who you respect.

Most of us have never had a very good manager, let alone a great manager. If you dreaded your 1:1s or were bored by them, etc., it doesn’t have to be this way. 1:1s are the great bidirectional lens into each other’s roles and lives, and with a little curiosity and experimentation, they can be much more than 30 minutes of awkward conversation or the same 3 questions every week.

Run better meetings

Running great meetings – efficient, productive, interesting, with exactly the right people present – is an understated, underestimated art. We usually only notice when a meeting is being run poorly. Train yourself to notice meetings where everyone is leaning in, eyes sparkling, devices forgotten, or even just the meetings that you never dread going to: who runs those? Pay attention to those jedi masters, and copy the hell out of them.

Volunteer to do things that are new and challenging. (Nobody else knows how to do them, either)

Put your hand up and volunteer for things. Is there a new job ladder that needs to be written or revised? Does the on call schedule need refactoring? Has anyone checked to make sure salaries are fair and not biased by race or gender?

Who better than you? 😁

And finally, having saved the hardest and most open-ended for last:

Look for ways to improve and optimize your organization as a sociotechnical system

Anyone who calls themselves a technical leader, whether engineer or manager, shares responsibility for identifying, maintaining, and optimizing the inner feedback loops that help you build and ship software. They are at the heart of how your organization delivers value and becomes a place people want to work.

Figure out how high-performing your team is (start with the DORA metrics), then begin to evaluate how much time is being wasted and consumed by wheel-spinning due to inefficiencies in your core sociotechnical feedback loops.

Try to step back from fighting all the symptoms and pathologies; instead, try to address them at the source.

This may involve:

  • Staying up to date on technical news
  • Getting your hands dirty and tinkering from time to time
  • Talking to others in the org to identify shared pain
  • Learning what really motivates each person on your team
  • Crafting a pitch that excites them and pushes their boundaries
  • Breaking down massive technical debt into bite-sized initiatives
  • Knowing when to speak and when to shut up
  • Doling out strategic praise

In conclusion

Career development as a manager isn’t about racing up the org chart at a breakneck speed. What makes someone a great, fulfilled, happy engineering manager doesn’t necessarily translate into making them a great director or VP (or a happy one). Opportunities to advance to upper management are scarce and somewhat random. There’s a lot to learn about the craft of management and technical leadership, which is a much healthier thing to focus on.

Check in with yourself regularly about whether you still enjoy your work. Swinging the pendulum back to engineering every few years is a great way to ensure you remain employable and your skills remain fresh over the long run.

Building a long and rewarding career in tech is not about chasing titles or levels, it’s about learning to follow your curiosity (and learning to love the uncertain feeling of being out of your depth.)

One final warning: managers have a tendency to slip into a pattern of only doing things or growing in ways that are useful to the org, but useless to them personally (once they change jobs or leave the company). Don’t do this! Be more selfish. Insist on building skills that are transferable and inherently valuable, not just knowing where all the bodies are buried at your current employer.

Anyone who manages to be happy in management will have to seek out fresh dopamine hits to replace those they gave up in engineering. Happy hunting.

Career Development For Engineering Managers

How To Hire An Engineering Manager

Cross-posted from leaddev.com

Why it matters to look internally first.

So, you need an engineering manager. Maybe what you need is to replace a departing manager – someone who is leaving the company, being promoted, or going back to hands-on individual contributor (IC) work. Maybe your team has gotten too large and unwieldy and is ready to split in two, amoeba-like, or you’re on the verge of hiring a few more people and spinning off another team. Maybe you are a baby startup and this engineering manager will be your first formal structure, or maybe something else entirely.

Whatever the genesis, the point is the same: you need to find a manager. And your very first fork in the road will be, “Should I hire one from the outside, or tap one of my engineers for the role?”

Here is a brief overview of the considerations involved in asking this question and making that decision.

First of all, I should confess to my biases. I am a big believer in the philosophy that management is not a promotion, but a change of career. There is a great deal of evidence that hierarchy produces worse outcomes in creative industries, of which software is decidedly one. I think that management is not and should not be synonymous with leadership; in fact, there are areas of technical leadership and implementation details where managers have no business exerting ownership. There should be technical leadership levels that track management levels as their equals in compensation, authority, and respect, but with different domains – ‘non-overlapping magisteria’, as the great Stephen Jay Gould might have said.

Engineering managers are ultimately responsible for the health of the team, the career progression of the engineers under their care, and for making sure that engineering teams are focused on the right priorities. They are the essential link between business and engineering; they must be able to translate between the two worlds many times a day. But once the decision is made that something is a priority, how it is done is the proper domain of engineers and technical leads.

Now, back to our choice. What should we do – hire from within, or hire from without?

Give preference to internal candidates

I am a firm believer that you always, always owe your existing team first crack at any and every opportunity that arises from within. It may not always be possible; you may need someone with more experience or a different skillset than you have on your team, but you owe it to your team to seriously explore the possibility, every time, before you give in and begin to explore outside possibilities.

That is an ethical stance which I happen to hold. But there are also pragmatic reasons for preferring internal candidates over external candidates where possible. Effective line management requires a manager to have a reasonably good understanding of the tech stack and the slate of current problems and tradeoffs; who better than one of the engineers working on it? And good management, in general, is all about relationships, relationships, relationships; someone coming in the door will always take longer to build those relationships and be effective than someone who already knows the lay of the land and the people who dwell in it. Internal candidates taking a lateral transfer to a management role are the most likely to succeed, in my experience.

So, what are some questions you might ask yourself when looking around for management candidates?

Who is the best engineer?

This is traditionally the first (and sometimes only) question leadership asks before ‘promoting’ (sic) said engineer to management. Unfortunately, this is the wrong question to ask yourself. There is no correlation that I’ve observed between someone’s engineering skills and management skills – no positive correlation, at any rate.

I do believe engineering managers need to have a strong technical background in order to be effective, but they don’t need to be the best engineer in the room. Many fervently mediocre senior engineers have developed into strong, admirable engineering managers – some of whom then swing the pendulum back to IC and leverage those hybrid skillsets to become supremely valuable senior technologists.

This is one big reason why it’s important to decouple leadership from management. Your best engineers are likely hungry for advancement, autonomy, and impact (just like everyone else), and you do not want them to conclude that management is the only vehicle for their ambitions – the only way they’ll ever get a seat in the room where it happens. You want them to solve your hardest technical problems if that is where their heart is. A better question to start with might be:

Who is senior enough to be a manager?

Don’t consider anyone for management until they have achieved a solid senior level and status – at least 5–7 years of hands-on engineering. Their technical judgment should be well-respected by their peers; they should know enough to know how much they don’t know, and where to trust their own opinion or not.

You won’t be doing them any favors by asking them to switch careers earlier in their tenure, and you might well be inflicting permanent damage on their career. At least half of those who try management end up not liking it, or go back to engineering at some point. If they make the leap to management too early, it will be extremely difficult for them to return to engineering a couple of years later. You might be sentencing them to a one-way trip whether they like it or not.

It will also be difficult for them to effectively manage junior engineers, who need mentorship and career development, or senior engineers, whose experience and judgment will far outstrip their own. Too many managers end up frustrating their teams by lurching between command-and-control leadership, and blind trust and abdication of leadership – not because they want to, but because they lack the technical subtlety to know when to speak or shut up.

Are any of your engineers interested?

Is anyone on your team interested in trying out the manager gig? Don’t guess; ask. Not just once, but regularly. And don’t wait until a position opens up to ask, either. Managers should have a chat about career plans with everyone on their team at least once or twice a year, and that includes asking whether they would like to try being a manager at some point.

Management is just a bundle of skills and practices, after all. You don’t have to have a formal manager title in order to be tasked with or practice things like running an interview loop, mentoring an intern, running team meetings, and writing up plans. If someone wants to become a manager at some point in their career, you should help them find opportunities to practice some of these skills and habits.

This is the quickest and easiest way for someone to find out if the work actually doesn’t interest them. And it’s a great low-risk way for you to watch them in action and observe their skills, without having to entrust the fate of an entire team to them yet.

When an opening does arise, you should already have a mental map of who on the team is likely to be interested, and how skilled they are.

Do I need to interview them, or what?

If you’re seriously considering an internal candidate for management, start by having a conversation with them – a different sort of conversation than you may have had up to this point. Start with some simple, open-ended questions, like:

“Why do you think there are so many more men than women in software roles?”

And let them talk for a while. You aren’t trying to trap them, or looking for ideological purity, but if you are going to put someone into a position of formal organizational power over others, it’s your responsibility to make sure they won’t inflict harm.

You have to have a higher bar for managers when it comes to awareness of structural racism and interpersonal power dynamics. Being a ‘really good person’ is not enough to do the job; ‘good people’ inflict immense harm every day without meaning to. You need managers who are humble, self-aware, and eager to learn – not ones who will have a meltdown and become the problem any time they mess up.

What if there are two or more who want it?

This is another great argument for treating management as a change of career, not a promotion, and for legitimizing and normalizing the idea that it’s okay to go back and forth between management and engineering, every few years.

If it’s not a promotion, then you don’t have any status to give up. If it’s an ordinary thing, there will be other opportunities. Not every opportunity exists at every company at every time, but the more you can do to lower the perceived scarcity and intensity, the more you will find people becoming managers or engineers for the right reasons rather than the wrong ones.

The team’s opinion matters more than yours

You should never inflict a manager on a team who doesn’t want to receive them in that role, no matter how qualified they seem on paper. If someone wants to be manager, your next move should be to quietly chat with each of the team members to see how they receive the idea, giving special weight to the opinions (and body language) of any underrepresented minorities on the team. Do they want to report to this person? If not, don’t push it – even a little.

Slight caveat: if the proposed candidate is a gender or racial minority, and if you suspect the team’s discomfort is related to their status, it’s worth digging in a little more to understand the situation and even possibly overrule some hesitation. As ever, trust your intuition but interrogate it, and use your best judgment.

Reasons to go ahead and open an external req

I said that internal candidates are my preference by default, but there remains a long, long list of reasons why you might decide to hire externally after all. A few of them include:

  • Your team is mostly junior-to-intermediate, and your senior folks are lukewarm on the idea of management (never push someone into it!)
  • All your existing managers are first-timers, and you really need some more experienced managers in the mix to help you formalize policies and processes as you grow
  • The team is split, and there is no consensus behind any internal candidate.

Listen to your own intuition, too: if you don’t have a great feeling about any of your options, take that seriously.

If you decide to hire a manager from the outside, the first thing you need to do is sit down and take a long hard look in the mirror. Hiring isn’t about finding the ‘best person’, it’s about crafting the strongest team. Where is your team weak? Who do you worry about? What are your strategic goals for the upcoming year or two, and what kind of leadership experience will help you get there? Do you need someone who can be very hands-on with the technology, or someone skilled at systems building, or driving transformation? Will recruiting be make-or-break for you? How diverse is your team, and what are your diversity goals?

All of these archetypes call for different types of leaders. But that’s a topic for another time.

How To Hire An Engineering Manager

Should Engineering Managers Be Technical?

(Crossposted from leaddev.com)

Last week someone asked me this question, ‘How important are management skills vs. the rest of it?’

They continued, ‘Like, shouldn’t we be able to hire people into these engineering management roles if they have management skills, but not necessarily any software engineering skills? Why do we treat one skillset as non-negotiable, and the other as easily learnable? And since we’re always telling managers not to write code or do technical work – since what managers actually do all day is go to meetings, write emails, and other managerial things – shouldn’t it be management skills that are non-negotiable rather than engineering skills?’

I have thought long and hard about this; not least because our profession has a long and inglorious history of treating technical skills as innately superior to other skillsets (and engineers as superior to other professions). It’s as though simply learning to code well is some kind of wild card ability that makes you instantly better at everything else around you. My instinct is to say that engineering skills are non-negotiable and the managerial skills can be acquired later, but am I simply manifesting that same old snobbery?

I don’t think so. But maybe! I will make my case here, and let’s see if you agree.

I don’t think you need to be the best engineer in order to be a good engineering manager (EM). I wouldn’t recommend becoming an EM until you have been working as an engineer for at least five to seven years and are comfortably senior-level, but beyond that, I see little evidence that superior engineering skills translate into greater management aptitude or better engineering outcomes. I am actually a vocal proponent of offering management opportunities to middlingly-skilled engineers.

Management is not a promotion, it’s a lateral step to a parallel track

But management itself is less of a discrete skillset and more of a superset or an adjunct discipline. It’s a coaching relationship, a derivative role, one that leverages your existing knowledge of the discipline to help teams make decisions, resolve conflicts, and develop skills and processes. Therefore, managers need to have a solid grounding in engineering fundamentals before they can usefully help others grow as engineers. Managers may also need to go back to the well periodically and engage in engineering work to refresh their capabilities.

You can’t recruit a Starbucks manager to manage a software engineering team any more than you would recruit an NFL coach to coach your Olympic gymnastics team. If you want to coach a team, you gotta know the game.

Some of you are nodding along and thinking to yourself how obvious this all sounds. But this is actually a somewhat controversial stance. Every time this topic comes up, a sizable minority of those present begin strenuously disagreeing with the case I have just made, and always for the same reason: because they have worked for managers who were non-technical, and consider them the best managers they have ever had. Seriously.

‘I had a manager who knew nothing about technology, and they were the best manager I’ve ever had. They knew their limits, they never interfered, they always took my word for it and backed me up’; ‘The manager was always paired with a tech lead (TL) who they relied on for technical stuff, and they just supported us as people’; ‘Engineers mentored each other technically – we didn’t need the manager for that.’

This experience is surprisingly common, enough that it’s worth taking seriously. And I do.

Non-technical managers may be unicorns or a split-brain

Clearly, it is possible for non-technical managers to effectively manage engineering teams. I know it is possible – I have seen it done myself, once or twice. There are two scenarios where this can work.

In the first scenario, the manager is a truly unique individual, a unicorn of sorts, who compensates for their lack of technical ability with an almost uncanny empathic, intuitive sense for who to trust; when to trust them and for how far; and usually an extensive knowledge of the domain. They overcompensate for their lack of technical skills by being stunning in every other possible way. They usually climb the org chart fairly quickly, because the higher you go, the less relevant your hands-on engineering skills become compared to your strategic ability and your skill at motivating and influencing others.

These people exist, but they are rare. You cannot plan a hiring strategy that depends on you finding and hiring them, because they are exceptions to the norm.

In the second scenario, non-technical or less-technical managers are part of an intentional organizational strategy – usually due to some quirk of who the early employees were or the organizational philosophy of the founding team. Non-technical people managers are paired with highly technical tech leads, essentially splitting the brain of an EM in half. This strategy relies on those two staying in lockstep; it has rigid leadership dynamics that I’m not super fond of, but it does work. Note that it ‘works’ by designing the entire organization around compensating for this blind spot.

If the manager and the tech lead disagree, it’s bad news. The manager has limited ability to personally evaluate or review your work and they have to outsource their judgment to the tech lead. When faced with tough decisions, the manager cannot guide the team to explore the territory and arrive at a group consensus, so the tech lead generally makes all decisions.

It’s a rather fragile state of affairs, but when you roll the dice and get the right coworkers, it can work – especially for the tech lead. Tech leads in this configuration are extremely powerful; the manager basically defers to the TL whenever the TL thinks they should, which can be quite gratifying to a senior person. Most of the vocal proponents of this model are people who miss being that TL. They tend to talk longingly about how no one ever questioned their judgment or evaluated their performance and just left them alone to do engineering as they saw fit. And they say this like it’s a good thing.

A manager’s job is tuning sociotechnical feedback loops

These days we all work on, and from within, these incredibly complex sociotechnical systems and feedback loops that are so complex, there is rarely any problem that is purely ‘social’, ‘tools’, or ‘technology’. It is all entwined and interconnected. And if you are to have a hope of improving these systems and feedback loops, you need a full toolset. You need both social and managerial skills, and technical engineering skills. Furthermore, you need to learn to wield them together.

You need to be able to look at a team that isn’t shipping very fast and figure out whether it’s because of under-skilled engineers, legacy code, a weak product culture, a leaky CI/CD pipeline, or a team that has given up and lost confidence in itself. And then you need to start influencing people to do things that visibly and materially improve the system.

If you try to separate the social from the technical, and assign different domain owners to each, you are slicing up the ownership and responsibility lengthwise. You’re setting yourself up for squabbles about whose fault it really was, instead of making the lines of responsibility and accountability crystal clear.

Good engineering managers have both social and technical skills, and they take an interest in the personal development of their engineers (or support their very senior engineers). This is the ideal situation for a team: to have a technically-skilled manager who can weigh in but knows their boundaries, doesn’t hog all the decision-making for themselves, and lets engineers own engineering decisions.

It is not your job to make all the decisions now

But too many engineering managers mess this up by not letting engineers own engineering decisions. This is especially true if they are used to being the most senior engineer on the team and so they tend to clutch onto those decisions as a manager. This is deeply disempowering and frustrating for the people on their team. It creates the conditions for engineers to speak longingly about being neglected by their nontechnical managers. It is a self-inflicted wound. Don’t do it.

Much fuss gets made about managers needing to stop writing code and contributing. I think this is close, but not quite right. Managers need to stop writing code in the critical path. They cannot allow themselves to become a bottleneck, so they shouldn’t write key features or contribute to the really interesting parts. I think it’s actually a great thing for managers to keep their hands warm with bug fixes, low pri side projects, etc. Teams really appreciate it when their manager viscerally feels their frustration with them.

So go ahead, put yourself in the on-call rotation, keep your skills sharp, all this is good and worthy. Just remember that your primary job is to be interruptible and available, so as a manager, writing code is a luxury that comes last.

Should Engineering Managers Be Technical?

“Why are my tests so slow?” A list of likely suspects, anti-patterns, and unresolved personal trauma.

Over the past couple of weeks I’ve been tweeting a LOT about lead time to deploy: the interval encompassing the time from when the code gets written and when it’s been deployed to production. Also described as “how long it takes you to run CI/CD.”

How important is this?

Fucking central.

Here is a quickie thread from this week, or just go read “Accelerate” like everybody already should have. 🙃

It’s nigh impossible to have a high-performing team with a long lead time, and becomes drastically easier with a dramatically shorter lead time.

🌷 Shorter is always better.
🌻 One mergeset per deploy.
🌹 Deploy should be automatic.

And it should clock in under 15 minutes, all the way from “merging!” to “deployed!”.

Now some people will nod and agree here, and others freak the fuck out. “FIFTEEN MINUTES?” they squall, and begin accusing me of making things up or working for only very small companies. Nope, and nope. There are no magic tricks here, just high standards and good engineering, and the commitment to maintaining your goals quarter by quarter.

If you get CI/CD right, a lot of other critical functions, behaviors, and intuitions are aligned to be comfortably successful and correct with minimal effort. If you get it wrong, you will spend countless cycles chasing pathologies. It’s like choosing to eat your vegetables every day vs choosing a diet of cake and soda for fifty years, then playing whackamole with all the symptoms manifesting on your poor, mouldering body.

Is this ideal achievable for every team, on every stack, product, customer and regulatory environment in the world? No, I’m not being stupid or willfully blind. But I suggest pouring your time and creative energy into figuring out how closely you can approximate the ideal given what you have, instead of compiling all the reasons why you can’t achieve it.

Most of the people who tell me they can’t do this are quite wrong, turns out. And even if you can’t down to 15 minutes, ANY reduction in lead time will pay out massive, compounding, benefits to your team and adjacent teams forever and ever.

So — what was it you said you were working on right now, exactly? that was so important? 🤔

“Cutting my build time by 90%!” — you

Huzzah. 🤠

So let’s get you started! Here, courtesy of my twitterfriends, is a long compiled list of Likely Suspects and CI/CD Offenders, a long list of anti-patterns, and some unresolved personal pain & suffering to hunt down and question when your build gets slow..

✨15 minutes or bust, baby!✨

 
Where it all started: what keeps you from getting under 15 minute CI/CD runs?

Generally good advice.

  • Instrument your build pipeline with spans and traces so you can see where all your time is going. ALWAYS. Instrument.
  • Order tests by time to execute and likelihood of failure.
  • Don’t run all tests, only tests affected by your change
  • Similarly, reduce build scope; if you only change front-end code, only build/test/deploy the front end, and for heaven’s sake don’t fuss with all the static asset generation
  • Don’t hop regions or zones any more than you absolutely must.
  • Prune and expire tests regularly, don’t wait for it to get Really Bad
  • Combine functionality of tests where possible — tests need regular massages and refactors too
  • Pipeline, pipeline, pipeline tests … with care and intention
  • You do not need multiple non-production environment in your CI/CD process. Push your artifacts to S3 and pull them down from production. Fight me on this
  • Pull is preferable to push. (see below)
  • Set a time elapsed target for your team, and give it some maintenance any time it slips by 25%

The usual suspects

  • tests that take several seconds to init
  • setup/teardown of databases (HINT try ramdisks)
  • importing test data, seeding databases, sometimes multiple times
  • rsyncing sequentially
  • rsyncing in parallel, all pulling from a single underprovisioned source
  • long git pulls (eg cloning whole repo each time)
  • CI rot (eg large historical build logs)
  • poor teardown (eg prior stuck builds still running, chewing CPU, or artifacts bloating over time
  • integration tests that spin up entire services (eg elasticsearch)
  • npm install taking 2-3 minutes
  • bundle install taking 5 minutes
  • resource starvation of CI/CD system
  • not using containerized build pipeline
  • …(etc)
 
Continuous deployment to industrial robots in prod?? Props, man.

Not properly separating the streams of “Our Software” (changes constantly) vs “infrastructure” (changes rarely)

  • running cloudformation to setup new load balancers, dbs, etc for an entire acceptance environment
  • docker pulls, image builds, docker pushes container spin up for tests

“Does this really go here?”

  • packaging large build artifacts into different format for distribution
  • slow static source code analysis tools
  • trying to clone production data back to staging, or reset dbs between runs
  • launching temp infra of sibling services for end-to-end tests, running canaries
  • selenium and other UX tests, transpiling and bundling assets

“Have a seat and think about your life choices.”

  • excessive number of dependencies
  • extreme legacy dependencies (things from the 90s)
  • tests with “sleep” in them
  • entirely too large frontends that should be broken up into modules

“We regret to remind you that most AWS calls operate at the pace of ‘Infrastructure’, not ‘Software'”

  • AWS CodeBuild has several minutes of provisioning time before you’re even executing your own code — even a few distinct jobs in a pipeline and you might suffer 15 min of waiting for CodeBuild to do actual work
  • building a new AMI
  • using EBS
  • spinning up EC2 nodes .. sequentially 😱
  • cool it with the AWS calls basically



A few responses were oozing with some unresolved trauma, lol.

Natural Born Opponents: “Just cache it” and “From the top!”

  • builds install correct version of toolchain from scratch each time
  • rebuilding entire project from source every build
  • failure to cache dependencies across runs (eg npm cache not set properly)

“Parallelization: the cause of, and solution to, all CI problems”

  • shared test state, which prevents parallel testing due to flakiness and non-deterministic test results
  • not parallelizing tests
 
 
I have so many questions….

Thanks to @wrd83, @sorenvind, @olitomli, @barney_parker, @dastbe, @myajpitz, @gfodor, @mrz, @rwilcox, @tomaslin, @pwyliu, @runewake2, @pdehlkefor, and many more for their contributions!

P.S. what did I say about instrumenting your build pipeline? For more on honeycomb + instrumentation, see this thread. Our free tier is incredibly generous, btw ☺️

Stay tuned for more long form blog posts on this topic. Coming soon. 🌈

charity

P.S. this blog post is the best thing i’ve ever read about reducing your build time. EVER.

“Why are my tests so slow?” A list of likely suspects, anti-patterns, and unresolved personal trauma.

How to make boba at home…without ruining any pans, making yourself ill, or ending up with a soggy, blobby mess

Last year I was diagnosed with ADHD, which was a great surprise to me (if no one else). Since then I have been trying to pay attention to things I do that might be, let’s say, outside the norm. One of those things is, apparently, food.

I tend to fixate on one food at a time. When I wake up in the morning, it’s the first and only thing I crave. When I’m hungry, I’m dying for it, and I don’t really experience cravings or desire for other foods, although I will eat them to be polite. The phase tends to last for…six months to two years? and then it shifts to something else.

The target of my appetite has been, at various times in the past: honeycrisp apples with peanut butter (I was DEVASTATED when honeycrisp season ended; other apples weren’t the same), dry cheerios with freeze-dried strawberries, chopped broccoli with sharp cheddar, a cashew chicken dish at a now-defunct Thai restaurant, etc.

One year it was manhattans (makers mark, sweet vermouth and bitters) and I seriously worried I was becoming an alcoholic. 🙈

But since September 19th, 2019, the only thing I have been interested in eating is … boba. Those little brown tapioca balls. I can rattle off to you the top boba places in every city I’ve visited since then (LA has some seriously adventurous ones). And when the world strapped in for quarantine, I was on the verge of panic. What to do??

I finally figured out how to make my own boba. This was NONTRIVIAL. It took the sacrifice of countless pans and far too many nights doubled over with nausea and stomach cramps (read my buying tips, I cannot this stress enough), and months of trial and error. But here is how to get the plump, chewy, slightly sweet boba of your dreams.

(Just the boba. Drinks are up to you. I recommend The Boba Book.)

Buying boba.

Do not buy any boba from China. Do not buy any boba labeled “quick cook”, or boba with instructions that are on the order of 5 minutes. Do not buy any flavored boba. I got violently ill from about half a dozen different brands I ordered randomly off Amazon, all made in China. Some had an odd aftertaste.

Supposedly, the Boba Guys are planning to let us buy the stuff they make domestically in California “soon”. Until then, stick to the stuff that is made of tapioca flour only, and manufactured in Taiwan or The U.S.

Also, the little balls are very fragile and turn to powder in the mail unless they are packed very tightly. This boba, from The Tea Zone is what I buy and recommend buying. Pick up some large diameter straws if you don’t have a stash at home.

Equipment.

You need a big-ass pot of boiling water. The biggest pot you’ve got. I use a big soup pot that holds like 16 or 20 quarts.

IMG_0923
Big Ass Pot

If you only have a few quarts of water, you will ruin pans. The tapioca dust turns to gummy that sticks to the sides and bottom and gets baked on like a motherfucker. You want a ratio of SHIT TONS of water to a handful or two of boba.

Cooking.

Fill it up with water to within an inch or two of the top — Bring it to a fast boil, then put your boba in — a cup or two or three, whatever you think you need. Let it boil for 20-25 minutes… only reduce the heat if you have to to keep it from boiling over.

Uncooked boba will have these little white spots in the middle. Once you see only a few of those in a sea of black pearls, turn off the heat. Let it sit in the hot water for another 20-25 minutes.

IMG_0939
Spot the uncooked boba

Then take the pot to the sink, pour off the excess water, fill it back up with cold water, swoosh it around to rinse; pour, fill, rinse a couple times til the balls are rinsed and lukewarm. You don’t have to drain them dry-dry; leave a small bit of water in the pan.

Flavoring and eating.

Add some sweetener — I like brown sugar, but honey is good too, or molasses and white sugar — and let the balls soak for another 30 minutes so they absorb the flavor. Now they are ready to eat. They will only keep for about a day, and don’t refrigerate them or they get gross.

**If you want the syrupy consistency of the gourmet boba shops, leave a little extra water in there, add the sugar, then simmer on low and STIR CONSTANTLY for 5-10 minutes or until it gets syrupy. I cannot stress this enough: rinse the boba first, and do not stop stirring, if you enjoy your pans and want to use them again

The easiest possible recipe (besides eating from the pot with a spoon) is, fill a glass 1/3 of the way with boba, add milk, add brown sugar simple syrup to taste. Add a couple ice cubes if you like your boba on the firm side. Also, try adding a little bit of rum and Frangelico for your bedtime boba.

Cheers!

IMG_9586
Boba, milk, frangelico

How to make boba at home…without ruining any pans, making yourself ill, or ending up with a soggy, blobby mess

Questionable Advice: Can Engineering Productivity Be Measured?

I follow you on Twitter and read your blog.  I particularly enjoy this post: https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/ I’m reaching out looking for some guidance.

I work as an engineering manager for a company whose non-technology leadership insists there has to be a way to measure the individual productivity of a software engineer. I have the opposite belief. I don’t believe you can measure the productivity of “professional” careers, or thought workers (ex: how do measure productivity of a doctor, lawyer, or chemist?).

For software engineering in particular, I feel that metrics can be gamed, don’t tell the whole story, or in some cases, are completely arbitrary. Do you measure individual developer productivity? If so, what do you measure, and why do you feel it’s valuable? If you don’t and share similar feelings as mine, how would you recommend I justify that position to non-technology leadership?

Thanks for your time.

Anonymous Engineering Manager

Dear Anon,

Once upon a time I had a job as a sysadmin, 100% remote, where all work was tracked using RT tasks. I soon realized that the owner didn’t have a lot of independent technical judgment, and his main barometer for the caliber of our contributions was the number of tasks we closed each day.

I became a ticket-closing machine. I’d snap up the quick and easy tasks within seconds. I’d pattern match and close in bulk when I found a solution for a group of tasks. I dove deep into the list of stale tickets looking for ones I could close as “did not respond” or “waiting for response”, especially once I realized there was no penalty for closing the same ticket over and over.

My boss worshiped me. I was bored as fuck. Sigh.

I guess what I’m trying to say is, I am fully in your camp. I don’t think you can measure the “productivity” of a creative professional by assigning metrics to their behaviors or process markers, and I think that attempting to derive or inflict such metrics can inflict a lot of damage.

In fact, I would say that to the extent you can reduce a job to a set of metrics, that job can be automated away. Metrics are for easy problems — discrete, self-contained, well-understood problems. The more challenging and novel a problem, the less reliable these metrics will be.

Your execs should fucking well know this: how would THEY like to be evaluated based on, like, how many emails they send in a day? Do they believe that would be good for the business? Or would they object that they are tasked with the holistic success of the org, and that their roles are too complex to reduce to a set of metrics without context?

This actually makes my blood boil. It is condescending as fuck for leadership to treat engineers like task-crunching interchangeable cogs. It reveals a deep misunderstanding of how sociotechnical systems are developed and sustained (plus authoritarian tendencies, and usually a big dollop of personal insecurity).

But what is the alternative?

In my experience, the “right” answer, i.e. the best way to run consistently high-performing teams, involves some combination of the following:

  • Outcome-based management that practices focusing on impact, plus
  • Team level health metrics, combined with
  • Engineering ladder and regular lightweight reviews, and
  • Managers who are well calibrated across the org, and encouraged to interrogate their own biases openly & with curiosity.

The right way to look at performance is at the team level. Individual engineers don’t own or maintain code; teams do. The team is the irreducible unit of ownership. So you need to incentivize people to think about work and spending their time cooperatively, optimizing for what is best for the team.

Some of the hardest and most impactful engineering work will be all but invisible on any set of individual metrics. You want people to trust that their manager will have their backs and value their contributions appropriately at review time, if they simply act in the team’s best interest. You do not want them to waste time gaming the metrics or courting personal political favor.

This is one of the reasons that managers need to be technical — so they can cultivate their own independent judgment, instead of basing reviews on hearsay. Because some resources (i.e. your budget for individual bonuses) are unfortunately zero-sum, and you are always going to rely on the good judgment of your engineering leaders when it comes to evaluating the relative impact of individual contributions.

“I would say that Joe’s contribution this quarter had greater impact than Jane’s. But is that really true? Jane did a LOT of mentoring and other “glue” work, which tends to be under-acknowledged as leadership work, so I just want to make sure I am evaluating this fairly … Does anyone else have a perspective on this? What might I be missing?” — a manager keeping themselves honest in calibrations

I do think every team should be tracking the 4 DORA metrics — time elapsed between merge and deploy, frequency of deploy, time to recover from outages, duration of outages — as well as how often someone is paged outside of business hours. These track pretty closely to engineering productivity and efficiency.

But leadership should do its best to be outcome oriented. The harder the problem, the more senior the contributor, the less business anyone has dictating the details of how or why. Make your agreements, then focus on impact.

This is harder on managers, for sure — it’s easier to count the hours someone spends at their desk or how many lines of code they commit than to develop a nuanced understanding of the quality and timbre of an engineer’s contributions to the product, team and the company over time. It is easier to micromanage the details than to negotiate a mutual understanding of what actually matters, commit to doing your part … and then step away, trusting them to fill in the gaps.

But we should expect this; it’s worth it. It is in those gaps where we feel trusted to act that we find joy and autonomy in our labor, where we do our best work as skilled artisans.

Questionable Advice: Can Engineering Productivity Be Measured?

Quarantine Reading Queue on the “Tiger King” Phenomenon

Last Wednesday I walked into my living room and saw three gay rednecks in hot pink shirts being married as a “throuple” on a TV screen at close range, followed by one of the grooms singing a country song about a woman feeding her husband’s remains to her tigers.

I could not look away.  What the fuck.

If you too have been rubbernecking the Tiger King — at any range — I have a book that will help you make sense of things: “Blood Rites: On The Origins and History of the Passions of War“, by Barbara Ehrenreich[1].  I re-read it last night, and here is my book report.


throuple

 

In Blood Rites, Ehrenreich asks why we sacralize war.  Not why we fight wars, or why we are violent necessarily, but why we are drawn to the idea of war, why we compulsively imbue it with an aura of honor and noble sacrifice.  If you kill one person, you’re a murderer and we shut you out from society; kill ten and you are a monster; but if you kill thousands, or kill on behalf of the state, we give you medals and write books about you.

And it’s not only about scale or being backed by state power.  The calling of war brings out the highest and finest experiences our species can know: it sings of heroism and altruism, of discipline, self-sacrifice, common ground, a life lived well in service; of belonging to something larger than one’s self.  Even if, as generations of weary returning soldiers have told us, it remains the same old butchery on the ground, the near-religious allure of war is never dented for long in the popular imagination.

What the fuck is going on?  bloodrites

Ehrenreich is impatient with the traditional scholarship, which locates the origin of war in some innate human aggression or turf wars over resources.  She is at her dryly funniest when dispatching feminist theories about violence being intrinsically male or “testosterone poisoning”, showing that the bloodthirstiest of the gods have usually been feminine.  (Although there are fascinating symmetries between girls becoming women through menstruation, and boys becoming men through … some form of culturally sanctioned ritual, usually involving bloodshed.)

Rather, she shows that our sacred feelings towards blood shed in war are the direct descendents of our veneration of blood shed in sacrifice — originally towards human sacrifice and other animal sacrifice, in a reenactment of our own ever-so-recent role inversion from prey to predator.  Prehistoric sacrifice was likely a way of exerting control over our environment and reenacting the death that gave us life through food.

In her theory, humans do not go to war because we are natural predators. Just the blink of an eye ago, on an evolutionary scale, humans were not predators by any means: we were prey.  Weak, blind, deaf, slow, clawless and naked; we scrawny, clever little apes we were easy pickings for the many large carnivores who roamed the planet.  We scavenged in the wake of predators and worshiped them as gods.  We are the nouveaux riche of predators, constantly re-asserting our dominance to soothe our insecurities.

We go to war not because we are predators, in other words, but because we are prey — and this makes us very uncomfortable!  War exists as a vestigial relic of when we venerated the shedding of blood and found it holy — as anyone who has ever opened the Old Testament can attest.  It was not until the Axial Age that religions of the world underwent a wholesale makeover into a less bloody, more universalistic set of aspirations.  ashes

When I first read this book, years ago, I remember picking it up with a roll of the eyes.  “Sounds like some overly-metaphorical liberal academic nonsense” or something like that.  But I was hooked within ten pages, my mind racing ahead with even more evidence than she marshals in this lively book.  It shifted the way I saw many things in the world.

Like horror movies, for example.  Or why cannibalism is so taboo.  How Jesus became the Son of God, the Brothers’ Grimm, the sacrament of Communion.  The primal fear of being food still resonates through our culture in so many sublimated ways.

And whether what you’re watching is “Tiger King” or the Tiger-King-watchers, it will make A LOT more sense after reading this book too.

Stay safe and don’t kill each other,

charity

IMG_6288

 

[1]  Ehrenreich is best known for her stunning book on the precariousness of the middle class, “Nickel and Dimed”, where she tried to subsist for a year only on whatever work she could get with a high school education.  Ehrenreich is a journalist, and this is a piece of science journalism, not scientific research; yet it is well-researched and scrupulously cited, and it’s worth noting that she has a PhD in biology and was once a practicing scientist.

 

 

Quarantine Reading Queue on the “Tiger King” Phenomenon

Questionable Advice: “After Being A Manager, Can I Be Happy As A Cog?”

One of my stretch goals for 2019 was to start writing an advice column.  I get a lot of questions about everything under the sun: observability, databases, career advice, management problems, what the best stack is for a startup, how to hire and interview, etc.  And while I enjoy this, having a high opinion of my own opinions and all, it doesn’t scale as well as writing essays.  I do have a (rather all-consuming) day job.

So I’d like to share some of the (edited and lightly anonymized) questions I get asked and some of the answers I have given.  With permission, of course.  And so, with great appreciation to my anonymous correspondent for letting me publish this, here is one.

Hi Charity,

I’ve been in tech for 25 years.  I don’t have a degree, but I worked my way up from menial jobs to engineering, and since then I have worked on some of the biggest sites in the world.  I have been offered a management role many times, but every time I refused.  Until about two years ago, when I said “fuck it, I’m almost 40; why not try.”

I took the job with boundless enthusiasm and motivation, because the team was honestly a mess.  We were building everything on-prem, and ops was constantly bullying developers over their supposed incompetence.  I had gone to conferences, listened to podcasts, and read enough blog posts that my head was full of “DevOps/CloudNative/ServiceOriented//You-build-it-you-run-it/ServantLeaders” idealism.  I knew I couldn’t make it any worse, and thought maybe, just maybe I could even make it better.softwarenegineeroncall_2 (1)

Soon after I took the job, though, there were company-wide layoffs.   It was not done well, and morale was low and sour.  People started leaving  for happier pastures.  But I stayed.  It was an interesting challenge, and I threw my heart and soul into it.

For two years I have stayed and grinded it out: recruiting (oh that is so hard), hiring, and then starting a migration to a cloud provider, and with the help of more and more people on the new team, slowly shifted the mindset of the whole engineering group to embrace devops best practices.  Now service teams own their code in production and are on-call for them, migrate themselves to the cloud with my team supporting them and building tools for them.  It is almost unrecognizable compared to where we were when I began managing.

A beautiful story isn’t it?  I hope you’re still reading.  🙂

Now I have to say that with my schedule full of 1:1s, budgeting, hiring, firing, publishing papers of mission statements and OKRs, shaping the teams, wielding influence, I realized that I enjoyed none of the above.  I read your 17 reasons not to be a manager, and I check so many boxes.  It is a pain in the ass to constantly listen to people’s egos, talk to them and keep everybody aligned (which obviously never happens).  And of course I am being crushed between top-down on-the-spot business decisions and bottom-up frustration of poorly executed engineering work under deadlines.  I am also destroyed by the mistrust and power games I am witnessing (or involved in, sometimes). while I long for collaboration and trust.  And of course when things go well my team gets all the praise, and when things go wrong I take all the blame.  I honestly don’t know how one can survive without the energy provided by praise and a sense of achievement.

All of the above makes me miss being an IC (Individual Contributor), where I could work for 8 hours straight without talking to anyone, build stuff, say what I wanted when I wanted, switch jobs if I wasn’t happy, and basically be a little shit like the ones you mention in your article.

Now you may say it’s obvious: I should find a new IC job in a healthier company.  You even wrote about it.  Going back to IC after two years of management is actually a good move.

But when I think about doing it, I get stuck.  I don’t know if I would be able to do it again, or if I could still enjoy it.  I’ve seen too many things, I’ve tasted what it’s like to be (sometimes) in control, and I did have a big impact on the company’s direction over time.  I like that.  If I went back to being an IC, I would feel small and meaningless, like just another cog in the machine.  And of course, being 40-ish, I will compete with all those 20-something smartasses who were born with kubernetes.

Thank you for reading.  Could you give me your thoughts on this?  In any case, it was good to get it off my chest.

Cheers,

Cog?

Dear Cog?,

Holy shitballs!  What an amazing story!  That is an incredible achievement in just two years, let alone as a rookie manager.  You deserve huge props for having the vision, the courage, and the tenacity to drive such a massive change through.

Of COURSE you’re feeling bored and restless.  You didn’t set out on a glorious quest for a life of updating mission statements and OKRs, balancing budgets, tending to people’s egos and fluffing their feelings, tweaking job descriptions, endless 1x1s and meetings meetings meetings, and the rest of the corporate middle manager’s portfolio.  You wanted something much bigger.  You wanted to change the world.  And you did!

But now you’ve done it.  What’s next?testinprod_3

First of all, YOUR COMPANY SUCKS.  You don’t once mention your leadership — where are they in all this?  If you had a good manager, they would be encouraging you and eagerly lining up a new and bigger role to keep you challenged and engaged at work.  They are not, so they don’t deserve you.  Fuck em.  Please leave.

Another thing I am hearing from you is, you harbor no secret desire to climb the managerial ranks at this time.  You don’t love the daily rhythms of management (believe it or not, some do); you crave novelty and mastery and advancement.  It sounds like you are willing to endure being a manager, so long as that is useful or required in order to tackle bigger and harder problems.  Nothing wrong with that!  But when the music stops, it’s time to move on.  Nobody should be saddled with a manager whose heart isn’t in the work.

You’re at the two year mark.  This is a pivotal moment, because it’s the beginning of the end of the time when you can easily slip back into technical work.  It will get harder and harder over the next 2-3 years, and at some point you will no longer have the option.

Picking up another technical role is the most strategic option, the one that maximizes your future opportunities as a technical leader.  But you do not seem excited by this option; instead you feel many complex and uncomfortable things.  It feels like going backwards.  It feels like losing ground.  It feels like ceding status and power.

“Management isn’t a promotion, it’s a career change.”

But if management is not a promotion, then going back to an engineering role should not feel like a demotion!  What the fuck?!

Imeplusprodt’s one thing to say that.  Whether it’s true or not is another question entirely, a question of policy and org dynamics.  The fact is that in most places, most of the power does go to the managers, and management IS a promotion.  Power flows naturally away from engineers and towards managers unless the org actively and vigorously pushes back on this tendency by explicitly allocating certain powers and responsibilities to other roles.

I’m betting your org doesn’t do this.  So yeah, going back to being an IC WILL be a step down in terms of your power and influence and ability to set the agenda.  That’s going to feel crappy, no question. We humans hate that.

Three points.

      1. You cannot go back to doing exactly what you did before, for the very simple reason that you are not the same person.  You are going to be attuned to power dynamics and ways of influencing that you never were before — and remember, leadership is primarily exercised through influence, not explicit authority.Senior ICs who have been managers are supremely powerful beings, who tend to wield outsize influence.  Smart managers will lean on them extensively for everything from shadow management and mentorship to advice, strategy, etc.  (Dumb managers don’t.  So find a smart manager who isn’t threatened by your experience.)
      2. You’re a short-timer here, remember?  Your company sucks.  You’re just renewing your technical skills and pulling a paycheck while finding a company that will treat you better, that is more aligned with your values.
      3. Lastly (and most importantly), I have a question.  Why did you need to become a manager in order to drive sweeping technical change over the past two years?  WHY couldn’t you have done it as a senior IC?  Shouldn’t technical people be responsible for technical decisions, and people managers responsible for people decisions?
        Could this be your next challenge, or part of it?  Could you go back to being an engineer, equipped with your shiny new powers of influence and mystical aura of recent management experience, and use it to organize the other senior ICs to assert their rightful ownership over technical decisions?  Could you use your newfound clout with leadership and upper management to convince them that this will help them recruit and retain better talent, and is a better way to run a technical org — for everyone?

     

I believe this is a better way, but I have only ever seen these changes happen when agitated for and demanded by the senior ICs.  If the senior ICs don’t assert their leadership, managers are unlikely to give it to them.  If managers try, but senior ICs don’t inhabit their power, eventually the managers just shrug and go back to making all the decisions.  That is why ultimately this is a change that must be driven and owned — at a minimum co-owned — by the senior individual contributors.Shared Joys, Kittens

I hope you can push back against that fear of being small and meaningless as an individual contributor.  The fact that it very often is this way, especially in strongly hierarchical organizations, does not mean that it has to be this way; and in healthy organizations it is not this way.  Command-and-control systems are not conducive to creative flourishing.  We have to fight the baggage of the authoritarian structures we inherited in order to make better ones.

Organizations are created afresh each and every day — not created for us, but by us.  Help create the organization you want to work at, where senior people are respected equally and have domains of ownership whether they manage people or technology.  If your current gig won’t value that labor, find one that will..

They exist.  And they want to hire you.

Lots of companies are DYING to hire this kind of senior IC, someone who is still hands on yet feels responsibility for the team as a whole, who knows the business side, who knows how to mentor and craft a culture and can herd cats when nec

There are companies that know how to use ICs at the strategic level, even executive level.  There are bosses who will see you not as a threat, but as a *huge asset* they can entrust with monumental work.

As a senior contributor who moves fluidly between roles, you are especially well-equipped to help shape a sociotechnical organization.  Could you make it your mission to model the kind of relationship you want to see between management and ICs, whichever side you happen to be on?  We need more people figuring out how to build organizations where management is not a promotion, just a change of career, and where going back and forth carries no baggage about promotions and demotions.  Help us.

And when you figure it out, please don’t keep it to yourself.  Expand your influence and share your findings by writing your experiences in blog posts, in articles, in talks.  Tell stories.  Show people people how much better it is this way.  Be so magnificently effective and mysteriously influential as a senior IC that all the baby engineers you work with want to grow up to be just like you.

Hope this helps.


charity

P.S. — Oh and stop fretting about “competing” with the 20-somethings kuberneteheads, you dork. You have been learning shit your whole career and you’ll learn this shit too.  The tech is the easy part.  The tech will always be the easy part.  🙂

Questionable Advice: “After Being A Manager, Can I Be Happy As A Cog?”

Outsource Your O11y: Now Roll It Out And Keep Them Happy (part 3/3)

This is part three of a three-part series of guest posts:

  1. How To Be A Champion, on how to choose a third-party vendor and champion them successfully to your security team.  (George Chamales)
  2. Get Aligned With Security, how to work with your security team to find the best possible outcome for all sides (Lilly Ryan)
  3. Now Roll It Out And Keep Them Happy, on how to operationalize your service by rolling out the integration and maintaining it — and the relationship with your security team — over the long run (Andy Isaacson)

All this pain will someday be worth it.  🙏❤️  charity + friends


“Now Roll It Out And Keep Them Happy”

This is the third in a series of blog posts; previously we analyzed the security challenges of using a third party service, and we worked together with the security team to build empathy to deliver the project.  You might want to read those first, since we are going to build on a lot of the ideas there to ship and maintain this integration.

Ready for launch

You’ve convinced the security team and other stakeholders, you’ve gotten the integration running, you’re getting promising results from dev-test or staging environments… now it’s time to move from proof-of-concept to full implementation.  Depending on your situation this might be a transition from staging to production, or it might mean increasing a feature flipper flag from 5% to 100%, or it might mean increasing coverage of an integration from one API endpoint to cover your entire developer footprint.

Taking into account Murphy’s Law, we expect that some things will go wrong during the rollout.  Perhaps during coverage, a developer realizes that the schema designed to handle the app’s event mechanism can’t represent a scenario, requiring a redesign or a hacky solution.  Or perhaps the metrics dashboard shows elevated error rates from the API frontend, and while there’s no smoking gun, the ops oncall decides to rollback the integration Just In Case it’s causing the incident.

This gives us another chance to practice empathy — while it’s easy, wearing the champion hat, to dismiss any issues found by looking for someone to blame, ultimately this poisons trust within your organization and will hamper success.  It’s more effective, in the long run (and often even in the short run), to find common ground with your peers in other disciplines and teams, and work through to solutions that satisfy everybody.

Keeping the lights on

In all likelihood as integration succeeds, the team will rapidly develop experts and expertise, as well as idiomatic ways to use the product.  Let the experts surprise you; folks you might not expect can step up when given a chance.  Expertise flourishes when given guidance and goals; as the team becomes comfortable with the integration, explicitly recognize a leader or point person for each vendor relationship.  Having one person explicitly responsible for a relationship lets them pay attention to those vendor emails, updates, and avoid the tragedy of the “but I thought *you* were” commons.  This Integration Lead is also a center of knowledge transfer for your organization — they won’t know everything or help every user come up to speed, but they can help empower the local power users in each team to ramp up their teams on the integration.

As comfort grows you will start to consider ways to change your usage, for example growing into new kinds of data.  This is a good time to revisit that security checklist — does the change increase PII exposure to your vendor?  Would the new data lead to additional requirements such as per-field encryption?  Don’t let these security concerns block you from gaining valuable insight using the new tool, but do take the chance to talk it over with your security experts as appropriate.

Throughout this organic growth, the Integration Lead remains core to managing your changing profile of usage of the vendor they shepherd; as new categories of data are added to the integration, the Lead has responsibility to ensure that the vendor relationship and risk profile are well matched to the needs that the new usage (and presumably, business value) is placing on the relationship.

Documenting the Intergation Lead role and responsibilities is critical. The team should know when to check in, and writing it down helps it happen.  When new code has a security implication, or a new use case potentially amplifies the cost of an integration, bringing the domain expert in will avoid unhappy surprises.  Knowing how to find out who to bring in, and when to bring them in, will keep your team getting the right eyes on their changes.

Security threats and other challenges change over time, too.  Collaborating with your security team so that they know what systems are in use helps your team take note of new information that is relevant to your business. A simple example is noting when your vendors publish a breach announcement, but more complex examples happen too — your vendor transitions cloud providers from AWS to Azure and the security team gets an alert about unexpected data flows from your production cluster; with transparency and trust such events become part of a routine process rather than an emergency.

It’s all operational

Monitoring and alerting is a fact of operations life, and this has to include vendor integrations (even when the vendor integration is a monitoring product.)  All of your operations best practices are needed here — keep your alerts clean and actionable so that you don’t develop pager fatigue, and monitor performance of the integration so that you don’t get blindsided by a creeping latency monster in your APIs.

Authentication and authorization are changing as the threat landscape evolves and industry moves from SMS verification codes to U2F/WebAuthn.  Does your vendor support your SSO integration?  If they can’t support the same SSO that you use everywhere else and can’t add it — or worse, look confused when you mention SSO — that’s probably a sign you should consider a different vendor.

A beautiful sunset

Have a plan beforehand for what needs to be done should you stop using the service.  Got any mobile apps that depend on APIs that will go away or start returning permission errors?  Be sure to test these scenarios ahead of time.

What happens at contract termination to data stored on the service?  Do you need to explicitly delete data when ceasing use?

Do you need to remove integrations from your systems before ending the commercial relationship, or can the technical shutdown and business shutdown run in parallel?

In all likelihood these are contingency plans that will never be needed, and they don’t need to be fully fleshed out to start, but a little bit of forethought can avoid unpleasant surprises.

Year after year

Industry best practice and common sense dictate that you should revisit the security questionnaire annually (if not more frequently). Use this chance to take stock of the last year and check in — are you getting value from the service?  What has changed in your business needs and the competitive landscape? 

It’s entirely possible that a new year brings new challenges, which could make your current vendor even more valuable (time to negotiate a better contract rate!) or could mean you’d do better with a competing service.  Has the vendor gone through any major changes?  They might have new offerings that suit your needs well, or they may have pivoted away from the features you need. 

Check in with your friends on the security team as well; standards evolve, and last year’s sufficient solution might not be good enough for new requirements.

 

Andy thinks out loud about security, society, and the problems with computers on Twitter.


 

❤️ Thanks so much reading, folks.  Please feel free to drop any complaints, comments, or additional tips to us in the comments, or direct them to me on twitter.

Have fun!  Stay (a little bit) Paranoid!!

— charity

Outsource Your O11y: Now Roll It Out And Keep Them Happy (part 3/3)