engineering management

Questionable Advice #2: How Do I Get My Team Into Observability?

mipsytipsy, , , , 1 Comment

Welcome to the second installment of my advice column! Last time we talked about the emotional impact of going back to engineering after a stint in management. If you have a question you’d like to ask, please email me or DM it to me on twitter.

Hi Charity! I hope it’s ok to just ask you this… 

I’m trying to get our company more aware of observability and I’m finding it difficult to convince people to look more into it. We currently don’t have the kind of systems that would require it much – but we will in future and I want us to be ahead of the game. 

If you have any tips about how to explain this to developers (who are aware that quality is important but don’t always advocate for it / do it as much as I’d prefer), or have concrete examples of “here’s a situation that we needed observability to solve – and here’s how we solved it”, I’d be super grateful. 

If this is too much to ask, let me know too 🙂 

I’ve been talking to Abby Bangser a lot recently – and I’m “classifying” observability as “exploring in production” in my mental map – if you have philosophical thoughts on that, I’d also love to hear them 🙂

alex_schl

 

Dear Alex,

Everyone’s systems are broken. Not just yours!

Yay, what a GREAT note!  I feel like I get asked some subset or variation of these questions several times a week, and I am delighted for the opportunity to both write up a response for you and post it for others to read.  I bet there are orders of magnitude more people out there with the same questions who *don’t* ask, so I really appreciate those who do. <3

I want to talk about the nuts and bolts of pitching to engineering teams and shepherding technical decisions like this, and I promise I will offer you some links to examples and other materials. But first I want to examine some of the assumptions in your note, because they elegantly illuminate a couple of common myths and misconceptions.

Myth #1: you don’t need observability til you have problems of scale

First of all, there’s this misconception that observability is something you only need when you have really super duper hard problems, or that it’s only justified when you have microservices and large distributed systems or crazy scaling problems.  No, no no nononono. 

There may come a point where you are ABSOLUTELY FUCKED if you don’t have observability, but it is ALWAYS better to develop with it.  It is never not better to be able to see what the fuck you are doing!  The image in my head is of a hiker with one of those little headlamps on that lets them see where they’re putting their feet down.  Most teams are out there shipping opaque, poorly understood code blindly — shipping it out to systems which are themselves crap snowballs of opaque, poorly understood code. This is costly, dangerous, and extremely wasteful of engineering time.


Ever seen an engineering team of 200, and struggled to understand how the product could possibly need more than one or two teams of engineers? They’re all fighting with the crap snowball.

Developing software with observability is better at ANY scale.  It’s better for monoliths, it’s better for tiny one-person teams, it’s better for pre-production services, it’s better for literally everyone always.  The sooner and earlier you adopt it, the more compounding value you will reap over time, and the more of your engineers’ time will be devoted to forward progress and creating value.

Myth #2: observability is harder and more technically advanced than monitoring

Actually, it’s the opposite — it’s much easier.  If you sat a new grad down and asked them to instrument their code and debug a small problem, it would be fairly straightforward with observability. Observability speaks the native language of variables, functions and API endpoints, the mental model maps cleanly to the request path, and you can straightforwardly ask any question you can come up with. (A key tenet of observability is that it gives an engineer the ability to ask any question, without having had to anticipate it in advance.)

With metrics and logging libraries, on the other hand, it’s far more complicated.you have to make a bunch of awkward decisions about where to emit various types of statistics, and it is terrifyingly easy to make poor choices (with terminal performance implications for your code and/or the remote data source).  When asking questions, you are locked in to asking only the questions that you chose to ask a long time ago. You spend a lot of time translating the relationships between code and lowlevel systems resources, and since you can’t break down by users/apps you are blocked from asking the most straightforward and useful questions entirely!  

Doing it the old way Is. Fucking. Hard.  Doing it the newer way is actually much easier, save for the fact that it is, well, newer — and thus harder to google examples for copy-pasta. But if you’re saturated in decades of old school ops tooling, you may have some unlearning to do before observability seems obvious to you.

Myth #3: observability is a purely technical solution

To be clear, you can just add an observability tool to your stack and go on about your business — same old things, same old way, but now with high cardinality!

You can, but you shouldn’t.  

These are sociotechnical systems and they are best improved with sociotechnical solutions.  Tools are an absolutely necessary and inextricable part of it.  But so are on call rotations and the fundamental virtuous feedback loop of you build it, you run it.  So are code reviews, monitoring checks, alerts, escalations, and a blameless culture.  So are managers who allocate enough time away from the product roadmap to truly fix deep technical rifts and explosions, even when it’s inconvenient, so the engineers aren’t in constant monkeypatch mode.

I believe that observability is a prerequisite for any major effort to have saner systems, simply because it’s so powerful being able to see the impact of what you’ve done.  In the hands of a creative, dedicated team, simply wearing a headlamp can be transformational.

Observability is your five senses for production.

You’re right on the money when you ask if it’s about exploring production, but you could also use words that are even more basic, like “understanding” or “inspecting”.  Observability is to software systems as a debugger is to software code.  It shines a light on the black box.  It allows you to move much faster, with more confidence, and catch bugs much sooner in the lifecycle — before users have even noticed.  It rewards you for writing code that is easy to illuminate and understand in production.

So why isn’t everyone already doing it?  Well, making the leap isn’t frictionless.  There’s a minimal amount of instrumentation to learn (easier than people expect, but it’s nonzero) and then you need to learn to see your code through the lens of your own instrumentation.  You might need to refactor your use of older tools, such as metrics libraries, monitoring checks and log lines.  You’ll need to learn another query interface and how it behaves on your systems.  You might find yourself amending your code review and deploy processes a bit.  

Nothing too terrible, but it’s all new.  We hate changing our tool kits until absolutely fucking necessary.  Back at Parse/Facebook, I actually clung to my sed/awk/shell wizardry until I was professionally shamed into learning new ways when others began debugging shit faster than I could.  (I was used to being the debugger of last resort, so this really pissed me off.)  So I super get it!  So let’s talk about how to get your team aligned and hungry for change.

Okay okay okay already, how do I get my team on board?

If we were on the phone right now, I would be peppering you with a bunch of questions about your organization.  Who owns production?  Who is on call?  Who runs the software that devs write?  What is your deploy process, and how often does it get updated, and by who?  Does it have an owner?  What are the personalities of your senior folks, who made the decisions to invest in the current tools (and what are they), what motivates them, who are your most persuasive internal voices?  Etc.  Every team is different.  <3

There’s a virtuous feedback loop you need to hook up and kickstart and tweak here, where the people with the original intent in their heads (software engineers) are also informed and motivated, i.e. empowered to make the changes and personally impacted when things are broken. I recommend starting by putting your software engineers on call for production (if you haven’t).  This has a way of convincing even the toughest cases that they have a strong personal interest in quality and understandability. 

Pay attention to your feedback loop and the alignment of incentives, and make sure your teams are given enough time to actually fix the broken things, and motivation usually isn’t a problem.  (If it is, then perhaps another feedback loop is lacking: your engineers feeling sufficiently aligned with your users and their pain.  But that’s another post.)

Technical ownership over technical outcomes

I appreciate that you want your team to own the technical decisions.  I believe very strongly that this is the right way to go.  But it doesn’t mean you can’t have influence or impact, and particularly in times like this. 

It is literally your job to have your head up, scanning the horizon for opportunities and relevant threats.  It’s their job to be heads down, focusing on creating and delivering excellent work.  So it is absolutely appropriate for you to flag something like observability as both an opportunity and a potential threat, if ignored.

If I were in your situation and wanted my team to check out some technical concept, I might send around a great talk or two and ask folks to watch it, and then maybe schedule a lunchtime discussion.  Or I might invite a tech luminary in to talk with the team, give a presentation and answer their questions.  Or schedule a hack week to apply the concept to a current top problem, or something else of that nature.

But if I really wanted them to take it fucking seriously, I would put my thumb on the scale.  I would find myself a champion, load them up with context, and give them ample time and space to skill up, prototype, and eventually present to the team a set of recommendations.  (And I would stay in close contact with them throughout that period, to make sure they didn’t veer too far off course or lose sight of my goals.)

  1. Get a champion.

    Ideally you want to turn the person who is most invested in the old way of doing things — the person who owns the ELK cluster, say, or who was responsible for selecting the previous monitoring toolkit, or the goto person for ops questions — from your greatest obstacle into your proxy warrior.  This only works if you know that person is open-minded and secure enough to give it a fair shot & publicly change course, has sufficiently good technical judgment to evaluate and project into the future, and has the necessary clout with their peers.  If they don’t, or if they’re too afraid to buck consensus: pick someone else.

  2. Give them context.  

    Take them for a long walk.  Pour your heart and soul out to them.  Tell them what you’ve learned, what you’ve heard, what you hope it can do for you, what you fear will happen if you don’t.  It’s okay to get personal and to admit your uncertainties.  The more context they have, the better the chance they will come out with an outcome you are happy with.  Get them worried about the same things that worry you, get them excited about the same possibilities that excite you.  Give them a sense of the stakes. 

    And don’t forget to tell them why you are picking them — because they are listened to by their peers, because they are already expert in the problem area, because you trust their technical judgment and their ability to evaluate new things — all the reasons for picking them will translate well into the best kind of flattery — the true kind.  

  3. Give them a deadline.

    A week or two should be plenty.  Most likely, the decision is not going to be unilaterally theirs (this also gives you a bit of wiggle room should they come back going “ah no ELK is great forever and ever”), but their recommendations should carry serious weight with the team and technical leadership.  Make it clear what sort of outcome you would be very pleased with (e.g. a trial period for a new service) and what reasons you would find compelling for declining to pursue the project (i.e. your tech is unsupported, cost prohibitive, etc).  Ideally they should use this time to get real production data into the services they are testing out, so they can actually experience and weigh the benefits, not just read the marketing copy.

As a rule of thumb, I always assume that managers can’t convince engineers to do things: only other engineers can.  But what you can do instead is set up an engineer to be your champion.  And then just sit quietly in the corner, nodding, with an interested look on your face.

The nuclear option


if you <3 prod,
prod will <3 you back

You have one final option.  If there is no appropriate champion to be found, or insufficient time, or if you have sufficient trust with the team that you judge it the right thing to do: you can simply order them to do something your way.  This can feel squicky. It’s not a good habit to get into.  It usually results in things being done a bit slower, more reluctantly, more half-assedly. And you sacrifice some of your power every time you lean on your authority to get your team to do something.

But it’s just as bad for a leader to take it off the table entirely.

Sometimes you will see things they can’t.  If you cannot wield your power when circumstances call for it, then you don’t fucking have real power — you have unilaterally disarmed yourself, to the detriment of your org.  You can get away with this maybe twice a year, tops. 

But here’s the thing: if you order something to be done, and it turns out in the end that you were right?  You earn back all the power you expended on it plus interest.  If you were right, unquestionably right in the eyes of the team, they will respect you more for having laid down the law and made sure they did the right thing.

xo

charity

Some useful resources:

 

Questionable Advice #2: How Do I Get My Team Into Observability?

Questionable Advice: “After Being A Manager, Can I Be Happy As A Cog?”

mipsytipsy, 9 Comments

One of my stretch goals for 2019 was to start writing an advice column.  I get a lot of questions about everything under the sun: observability, databases, career advice, management problems, what the best stack is for a startup, how to hire and interview, etc.  And while I enjoy this, having a high opinion of my own opinions and all, it doesn’t scale as well as writing essays.  I do have a (rather all-consuming) day job.

So I’d like to share some of the (edited and lightly anonymized) questions I get asked and some of the answers I have given.  With permission, of course.  And so, with great appreciation to my anonymous correspondent for letting me publish this, here is one.

Hi Charity,

I’ve been in tech for 25 years.  I don’t have a degree, but I worked my way up from menial jobs to engineering, and since then I have worked on some of the biggest sites in the world.  I have been offered a management role many times, but every time I refused.  Until about two years ago, when I said “fuck it, I’m almost 40; why not try.”

I took the job with boundless enthusiasm and motivation, because the team was honestly a mess.  We were building everything on-prem, and ops was constantly bullying developers over their supposed incompetence.  I had gone to conferences, listened to podcasts, and read enough blog posts that my head was full of “DevOps/CloudNative/ServiceOriented//You-build-it-you-run-it/ServantLeaders” idealism.  I knew I couldn’t make it any worse, and thought maybe, just maybe I could even make it better.softwarenegineeroncall_2 (1)

Soon after I took the job, though, there were company-wide layoffs.   It was not done well, and morale was low and sour.  People started leaving  for happier pastures.  But I stayed.  It was an interesting challenge, and I threw my heart and soul into it.

For two years I have stayed and grinded it out: recruiting (oh that is so hard), hiring, and then starting a migration to a cloud provider, and with the help of more and more people on the new team, slowly shifted the mindset of the whole engineering group to embrace devops best practices.  Now service teams own their code in production and are on-call for them, migrate themselves to the cloud with my team supporting them and building tools for them.  It is almost unrecognizable compared to where we were when I began managing.

A beautiful story isn’t it?  I hope you’re still reading.  🙂

Now I have to say that with my schedule full of 1:1s, budgeting, hiring, firing, publishing papers of mission statements and OKRs, shaping the teams, wielding influence, I realized that I enjoyed none of the above.  I read your 17 reasons not to be a manager, and I check so many boxes.  It is a pain in the ass to constantly listen to people’s egos, talk to them and keep everybody aligned (which obviously never happens).  And of course I am being crushed between top-down on-the-spot business decisions and bottom-up frustration of poorly executed engineering work under deadlines.  I am also destroyed by the mistrust and power games I am witnessing (or involved in, sometimes). while I long for collaboration and trust.  And of course when things go well my team gets all the praise, and when things go wrong I take all the blame.  I honestly don’t know how one can survive without the energy provided by praise and a sense of achievement.

All of the above makes me miss being an IC (Individual Contributor), where I could work for 8 hours straight without talking to anyone, build stuff, say what I wanted when I wanted, switch jobs if I wasn’t happy, and basically be a little shit like the ones you mention in your article.

Now you may say it’s obvious: I should find a new IC job in a healthier company.  You even wrote about it.  Going back to IC after two years of management is actually a good move.

But when I think about doing it, I get stuck.  I don’t know if I would be able to do it again, or if I could still enjoy it.  I’ve seen too many things, I’ve tasted what it’s like to be (sometimes) in control, and I did have a big impact on the company’s direction over time.  I like that.  If I went back to being an IC, I would feel small and meaningless, like just another cog in the machine.  And of course, being 40-ish, I will compete with all those 20-something smartasses who were born with kubernetes.

Thank you for reading.  Could you give me your thoughts on this?  In any case, it was good to get it off my chest.

Cheers,

Cog?

Dear Cog?,

Holy shitballs!  What an amazing story!  That is an incredible achievement in just two years, let alone as a rookie manager.  You deserve huge props for having the vision, the courage, and the tenacity to drive such a massive change through.

Of COURSE you’re feeling bored and restless.  You didn’t set out on a glorious quest for a life of updating mission statements and OKRs, balancing budgets, tending to people’s egos and fluffing their feelings, tweaking job descriptions, endless 1x1s and meetings meetings meetings, and the rest of the corporate middle manager’s portfolio.  You wanted something much bigger.  You wanted to change the world.  And you did!

But now you’ve done it.  What’s next?testinprod_3

First of all, YOUR COMPANY SUCKS.  You don’t once mention your leadership — where are they in all this?  If you had a good manager, they would be encouraging you and eagerly lining up a new and bigger role to keep you challenged and engaged at work.  They are not, so they don’t deserve you.  Fuck em.  Please leave.

Another thing I am hearing from you is, you harbor no secret desire to climb the managerial ranks at this time.  You don’t love the daily rhythms of management (believe it or not, some do); you crave novelty and mastery and advancement.  It sounds like you are willing to endure being a manager, so long as that is useful or required in order to tackle bigger and harder problems.  Nothing wrong with that!  But when the music stops, it’s time to move on.  Nobody should be saddled with a manager whose heart isn’t in the work.

You’re at the two year mark.  This is a pivotal moment, because it’s the beginning of the end of the time when you can easily slip back into technical work.  It will get harder and harder over the next 2-3 years, and at some point you will no longer have the option.

Picking up another technical role is the most strategic option, the one that maximizes your future opportunities as a technical leader.  But you do not seem excited by this option; instead you feel many complex and uncomfortable things.  It feels like going backwards.  It feels like losing ground.  It feels like ceding status and power.

“Management isn’t a promotion, it’s a career change.”

But if management is not a promotion, then going back to an engineering role should not feel like a demotion!  What the fuck?!

Imeplusprodt’s one thing to say that.  Whether it’s true or not is another question entirely, a question of policy and org dynamics.  The fact is that in most places, most of the power does go to the managers, and management IS a promotion.  Power flows naturally away from engineers and towards managers unless the org actively and vigorously pushes back on this tendency by explicitly allocating certain powers and responsibilities to other roles.

I’m betting your org doesn’t do this.  So yeah, going back to being an IC WILL be a step down in terms of your power and influence and ability to set the agenda.  That’s going to feel crappy, no question. We humans hate that.

Three points.

      1. You cannot go back to doing exactly what you did before, for the very simple reason that you are not the same person.  You are going to be attuned to power dynamics and ways of influencing that you never were before — and remember, leadership is primarily exercised through influence, not explicit authority.Senior ICs who have been managers are supremely powerful beings, who tend to wield outsize influence.  Smart managers will lean on them extensively for everything from shadow management and mentorship to advice, strategy, etc.  (Dumb managers don’t.  So find a smart manager who isn’t threatened by your experience.)
      2. You’re a short-timer here, remember?  Your company sucks.  You’re just renewing your technical skills and pulling a paycheck while finding a company that will treat you better, that is more aligned with your values.
      3. Lastly (and most importantly), I have a question.  Why did you need to become a manager in order to drive sweeping technical change over the past two years?  WHY couldn’t you have done it as a senior IC?  Shouldn’t technical people be responsible for technical decisions, and people managers responsible for people decisions?
        Could this be your next challenge, or part of it?  Could you go back to being an engineer, equipped with your shiny new powers of influence and mystical aura of recent management experience, and use it to organize the other senior ICs to assert their rightful ownership over technical decisions?  Could you use your newfound clout with leadership and upper management to convince them that this will help them recruit and retain better talent, and is a better way to run a technical org — for everyone?

     

I believe this is a better way, but I have only ever seen these changes happen when agitated for and demanded by the senior ICs.  If the senior ICs don’t assert their leadership, managers are unlikely to give it to them.  If managers try, but senior ICs don’t inhabit their power, eventually the managers just shrug and go back to making all the decisions.  That is why ultimately this is a change that must be driven and owned — at a minimum co-owned — by the senior individual contributors.Shared Joys, Kittens

I hope you can push back against that fear of being small and meaningless as an individual contributor.  The fact that it very often is this way, especially in strongly hierarchical organizations, does not mean that it has to be this way; and in healthy organizations it is not this way.  Command-and-control systems are not conducive to creative flourishing.  We have to fight the baggage of the authoritarian structures we inherited in order to make better ones.

Organizations are created afresh each and every day — not created for us, but by us.  Help create the organization you want to work at, where senior people are respected equally and have domains of ownership whether they manage people or technology.  If your current gig won’t value that labor, find one that will..

They exist.  And they want to hire you.

Lots of companies are DYING to hire this kind of senior IC, someone who is still hands on yet feels responsibility for the team as a whole, who knows the business side, who knows how to mentor and craft a culture and can herd cats when nec

There are companies that know how to use ICs at the strategic level, even executive level.  There are bosses who will see you not as a threat, but as a *huge asset* they can entrust with monumental work.

As a senior contributor who moves fluidly between roles, you are especially well-equipped to help shape a sociotechnical organization.  Could you make it your mission to model the kind of relationship you want to see between management and ICs, whichever side you happen to be on?  We need more people figuring out how to build organizations where management is not a promotion, just a change of career, and where going back and forth carries no baggage about promotions and demotions.  Help us.

And when you figure it out, please don’t keep it to yourself.  Expand your influence and share your findings by writing your experiences in blog posts, in articles, in talks.  Tell stories.  Show people people how much better it is this way.  Be so magnificently effective and mysteriously influential as a senior IC that all the baby engineers you work with want to grow up to be just like you.

Hope this helps.


charity

P.S. — Oh and stop fretting about “competing” with the 20-somethings kuberneteheads, you dork. You have been learning shit your whole career and you’ll learn this shit too.  The tech is the easy part.  The tech will always be the easy part.  🙂

Questionable Advice: “After Being A Manager, Can I Be Happy As A Cog?”

A Manager’s Bill of Responsibilities (and Rights)

mipsytipsy, , 7 Comments

Over a year and a half ago, I wrote up a post about the rights and responsibilities due any engineer at Honeycomb.  At the time we were in the middle of a growth spurt, had just hired several new engineers, and I was in the process of turning over day-to-day engmeme2engineering management over to Emily.  Writing things down helped me codify what I actually cared about, and helped keep us true to our principles as we grew.

Tacked on to the end of the post was a list of manager responsibilities, almost as an afterthought. Many people protested, “don’t managers get any rights??” (and naturally I snapped “NO!  hahahahahha”)

I always intended to circle back and write a followup post with the rights and responsibilities for managers.  But it wasn’t til recently, as we are gearing up for another hiring spurt and have expanded our managerial ranks, that it really felt like its time had come.

The time has come, the time is now, as marvin k. mooney once said.  Added the bill of rights, and updated and expanded the list of responsibilities.  Thanks Emily Nakashima for co-writing it with me.

 

Manager’s Bill of Rights

  1. You shall receive honest, courageous, timely feedback about yourself and your team, from your reports, your peers, and your leaders.  (No one is exempt from feeding the hungry hungry feedback hippo!  NOO ONNEEEE!)  🦛🦛🦛🦛🦛🦛🦛
  2. Management will be treated with the same respect and importance as individual work.  reviewmeme
  3. You have the final say over hiring, firing, and leveling decisions for your team.  It is expected that you solicit feedback from your team and peers and drive consensus where possible.  But in the end, the say is yours.
  4. Management can be draining, difficult work, even at places that do it well.  You will get tactical, strategic, and emotional support from other managers.
  5. You cannot take care of others unless you first practice self-care.  You damn well better take vacations.  (Real ones.)
  6. You have the right to personal development, career progression, and professional support.  We will retain a leadership coach for you.
  7. You do not have to be a manager if you do not want to.  No one will ever pressure you.

Manager’s Responsibilities

  • Recruit and hire and train your team. Foster a sense of solidarity and “teaminess” as well as real emotional safety.
  • Cultivate an inclusive culture and redistribute opportunity.  Fuck a pedigree.  Resist monoculture.
  • Care for the people on your team. Support them in their career trajectory, personal goals, work/life balance, and inter- and intra-team dynamics.
  • Keep an eye out for people on other teams who aren’t getting the support they need, and work with your leadership and manager peers to fix the situation. catplays
  • Give feedback early and often. Receive feedback gracefully. Always say the hard things, but say them with love.
  • Move us relentlessly forward, staying alert for rabbit-holing and work that doesn’t contribute to our goals. Ensure redundancy/coverage of critical areas.
  • Own the planning process for your team, be accountable for the goals you set. Allocate resources by communicating priorities and requesting support. Add focus or urgency where needed.
  • Own your time and attention. Be accessible. Actively manage your calendar. Try not to make your emotions everyone else’s problems (but do lean on your own manager and your peers for support).
  • Make your own personal growth and self-care a priority. Model the values and traits we want employees to pattern themselves after.
  • Stay vulnerable.

(Easier said than done, huh?)

<3 charity

Screen Shot 2019-10-30 at 8.04.07 AM

A Manager’s Bill of Responsibilities (and Rights)

Engineering Management: The Pendulum Or The Ladder

mipsytipsy, , , , , , , 62 Comments

Last night I was out with a dear friend who has been an engineering manager for a year now, and by two drinks in I was rattling off a long list things I always say to newer engineering managers.

Then I remembered: I should write a post! It’s one of my goals this year to write more long form instead of just twittering off into the abyss.Buffy Jaguar 3.5x5

There’s a piece I wrote two years ago, The Engineer/Manager Pendulum,  which is probably my all time favorite.  It was a love letter to a friend who I desperately wanted to see go back to engineering, for his own happiness and mental health.  Well, this piece is a sequel to that one.

It’s primarily aimed at new managers, who aren’t sure what their career options look like or how to evaluate the opportunities that come their way, or how it may expand or shrink their future opportunities.

The first fork in the manager’s path

Every manager reaches a point where they need to choose: do they want to manage engineers (a “line manager”), or do they want to try to climb the org chart? — manage managers, managers of other managers, even other divisions; while Does Not Kill Us Puppy UPDATEDbeing “promoted” from manager to senior manager, director to senior director, all the way up to VP and so forth.   Almost everyone’s instinct is to say “climb the org chart”, but we’ll talk about why you should be critical of this instinct.

They also face a closely related question: how technical do they wish to stay, and how badly do they care?

Are you an “engineering MANAGER” or an “ENGINEERING manager”?

These are not unlike the decisions every engineer ends up making about whether to go deep or go broad, whether to specialize or be a generalist.  The problem is that both engineers and managers often make these career choices with very little information — or even awareness that they are doing it.

And managers in particular then have a tendency to look up ten years later and realize that those choices, witting or unwitting, have made them a) less employable  and b) deeply unhappy.

Lots of people have the mindset that once they become an engineering manager, they should just go from gig to gig as an engineering manager who manages other engineers: that’s who they are now.  But this is actually a very fragile place to sit long-term, as we’ll discuss further on in this piece.

But let’s start at to the beginning, so I can speak to those of you who are considering management for the very first time.

“So you want to try engineering management.”

COOL! I think lots of senior engineers should try management, maybe even most senior engineers.  It’s so good for you, it makes you better at your job. (If you aren’t a senior engineer, and by that I mean at least 7+ years of engineering experience, be very wary; know this isn’t usually in your best interest.)

Hopefully you have already gathered that management is a career change, not a promotion, and you’re aware that nobody is very good at it when they first start.

That’s okay! It takes a solid year or two to find new rhythms and reward mechanisms before you can even begin to find your own voice or trust your judgment. Management problems look easy, deceptively so.  Reasons this is hard include:

  1. Most tech companies are absolutely abysmal at providing any sort of training or structure to help you learn the ropes and find your feet.
  2. Even if they do, you still have to own your own career development.  If learning to be a good engineer was sort of like getting your bachelor’s, learning to be a good manager is like getting your PhD — much more custom to who you are.
  3. It will exhaust you mentally and emotionally in the weirdest ways for much longer than you think it should.  You’ll be tired a lot, and you’ll miss feeling like you’re good at something (anything).

This is because you need to change your habits and practices, which in turn will actually change who you are.  This takes time.  Which is why …

The minimum tour of duty as a new manager is two years.

If you really want to try being a manager, and the opportunity presents itself, do it!  But only if you are prepared to fully commit to a two year long experiment.

Root Causes DolphinCommit to it like a proper career change. Seek out new peers, find new heroes. Bring fresh eyes and a beginner’s mindset. Ask lots of questions. Re-examine every one of your patterns and habits and priorities: do they still serve you? your team?

Don’t even bother thinking about in terms of whether you “enjoy managing” for a while, or trying to figure out if you are are any good at it. Of course you aren’t any good at it yet.  And even if you are, you don’t know how to recognize when you’ve succeeded at something, and you haven’t yet connected your brain’s reward systems to your successes.  A long stretch of time without satisfying brain drugs is just the price of admission if you want to earn these experiences, sadly.

It takes more than one year to learn management skills and wire up your brain to like it.  If you are waffling over the two year commitment, maybe now is not the time.  Switching managers too frequently is disruptive to the team, and it’s not fair to make them report to someone who would rather be doing something else or isn’t trying their ass off.

It takes about 3-5 years for your skills to deteriorate.

So you’ve been managing a team for a couple years, and it’s starting to feel … comfortable?  Hey, you’re pretty good at this!  Yay!

With a couple of years under your belt as a line manager, you now have TWO powerful skill sets.  You can build things, AND you can organize people into teams to build even bigger things. Right now, both sets are sharp.  You could return to engineering pretty easily, or keep on as a manager — your choice.

But this state of grace doesn’t last very long. Your technical skills stop advancing when you become a manager, and instead begin eroding.  Two years in, you aren’t the effective tech lead you once were; your information is out of date and full of gaps, the hard parts are led by other people these days.

More critically, your patterns of mind and habits shift over time, and become those of a manager, not an engineer.  Consider how excited an engineer becomes at the prospect of a justifiable greenfield project; now compare to her manager’s glum reaction as she instinctively winces at having to plan for something so reprehensibly unpredictable and difficult to estimate.  It takes time to rewire yourself back.

If you like engineering management, your tendency is to go “cool, now I’m a manager”, and move from job to job as an engineering manager, managing team after team of engineers.  But this is a trap.  It is not a sound long term plan.  It leads too many people off to a place they never wanted to end up: technically sidelined.

Sunglasses Tiger Debugger 3.3x5

Why can’t I just make a career out of being a combo tech lead+line manager?

One of the most common paths to management is this: you’re a tech lead, you’re directing ever larger chunks of technical work, doing 1x1s and picking up some of the people stuff, when your boss asks if you’d like to manage the team.  “Sure!”, you say, and voila — you are an engineering manager with deep domain expertise.

But if you are doing your job, you begin the process of divesting yourself of technical leadership responsibilities starting immediately.  Your own technical development should screech to a halt once you become a manager, because you have a whole new career to focus on learning.

Your job is to leverage that technical expertise to grow your engineers into great senior engineers and tech leads themselves.  Your job is not to hog the glory and squat on the hard problems yourself, it’s to empower and challenge and guide your team.  Don’t suck up all the oxygen: you’ll stunt the growth of your team.

But your technical knowledge gets dated, and your skills atrophy..  The longer it’s been since you worked as an engineer, the harder it will be to switch back.  It gets real hard around three years, and five years seems like a tipping point.[1]

And because so much of your credibility and effectiveness as an engineering leader comes from your expertise in the technology that your team uses every day, ultimately you will be no longer capable of technical leadership, only people management.

On being an “engineering manager” who only does people management

I mean, there’s a reason we don’t lure good people managers away from Starbucks to run engineering teams.  It’s the intersection and juxtaposition of skill sets that gives engineering managers such outsize impact.

The great ones can make a large team thrum with energy.  The great ones can break down a massive project into projects that challenge (but do not overwhelm) a dozen or more engineers, from new grads to grizzled veterans, pushing everyone to grow.  The great ones can look ahead and guess which rocks you are going to die on if you don’t work to avoid them right now.

The great ones are a treasure: and they are rare.  And in order to stay great, they regularly need to go back to the well to refresh their own hands-on technical abilities.

Pointless Ice Cream 3x2.5There is an enormous demand for technical engineering leaders — far more demand than supply.  The most common hackaround is to pair a people manager (who can speak the language and knows the concepts, but stopped engineering ages ago) with a tech lead, and make them collaborate to co-lead the team.  This unwieldy setup often works pretty well.

But most of those people managers didn’t want or expect to end up sidelined in this way when they were told to stop engineering.

If you want to be a pure people manager and not do engineering work, and don’t want to climb the ladder or can’t find a ladder to climb, more power to you.  I don’t know that I’ve met many of these people in my life.  I have met a lot of people in this situation by accident, and they are always kinda angsty and unhappy about it.  Don’t let yourself become this person by accident.  Please.

Which brings me to my next point.

You will be advised to stop writing code or engineering.

✨

Fuck

That.

 ✨

Everybody’s favorite hobby is hassling new managers about whether or not they’ve stopped writing code yet, and not letting up until they say that they have.  This is a terrible, horrible, no-good VERY bad idea that seems like it must originally have been a botched repeating of the correct advice, which is:

Stop writing code and engineering

in the critical path

Can you spot the difference?  It’s very subtle.  Let’s run a quick test:

  • Authoring a feature?  ⛔️
  • Covering on-call when someone needs a break?  ✅
  • Diving on the biggest project after a post mortem?  ⛔️
  • Code reviews?  ✅
  • Picking up a p2 bug that’s annoying but never seems to become top priority?  ✅
  • Insisting that all commits be gated on their approval?  ⛔️
  • Cleaning up the monitoring checks and writing a library to generate coverage?  ✅

The more you can keep your hands warm, the more effective you will be as a coach and a leader.  You’ll have a richer instinct for what people need and want from you and each other, which will help you keep a light touch.  You will write better reviews and resolve technical disputes with more authority.  You will also slow the erosion and geriatric creep of your own technical chops.

I firmly believe every line manager should either be in the on call rotation or pinch hit liberally and regularly, but that’s a different post.

Technical Leadership track

If you  love technology and want to remain a subject-matter expert in designing, building and shipping cutting-edge technical products and systems, you cannot afford to let yourself drift too far or too long away from hands-on engineering work.  You need to consciously cultivate your path , probably by practicing some form of the engineer/manager pendulum.

If you love managing engineers — if being a technical leader is a part of your identity that you take great pride in, then you must keep up your technical skills and periodically DIstrust Kittens 2.5x3invest in your practice and renew your education.  Again: this is simply the price of admission.  You need to renew your technical abilities, your habits of mind, and your visceral senses around creating and maintaining systems.  There is no way to do this besides doing it.  If management isn’t a promotion, then returning to hands-on work isn’t a demotion, either.  Right?

One warning: Your company may be great, but it doesn’t exist for your benefit.  You and only you can decide what your needs are and advocate for them.  Remember that next time your boss tries to guilt you into staying on as manager because you’re so badly needed, when you can feel your skills getting rusty and your effectiveness dwindling.  You owe it to yourself to figure out what makes you happy and build a portfolio of experiences that liberate you to do what you love.  Don’t sacrifice your happiness at the altar of any company.  There are always other companies.

Honestly, I would try not to think of yourself as a manager at all: you are an “engineering leader” performing a tour of duty in management.  You’re pursuing a long term strategy towards being a well-respected technologist, someone who can sling code, give informed technical guidance and explain in detail customized for to anyone at any level of sophistication.

Organizational Leadership Track

Most managers assume they want to climb the ladder.  Leveling up feels like an achievement, and that can feel impossible to resist.

Resist it.  Or at least, resist doing it unthinkingly.  Don’t do it because the ladder is there and must be climbed.  Know as much as you can about what you’re in for before you decide it’s what you want.

Here are a few reasons to think critically about climbing the ladder to director and executive roles.

  1. Your choices shrink. There are fewer jobs, with more competition, mostly at bigger companies.  (Do you even like big companies?)
  2. You basically need to do real time at a big company where they teach effective management skills, or you’ll start from a disadvantage.
  3. Bureaucracies are highly idiosyncratic, skills and relationships may or may not transfer with you between companies. As an engineer you could skip every year or two for greener pastures if you landed a crap gig.  An engineer has … about 2-3x more leeway in this regard than an exec does.  A string of short director/exec gigs is a career ender or a coach seat straight to consultant life.
  4. You are going to become less employable overall.  The ever-higher continuous climb almost never happens, usually for reasons you have no control over.  This can be a very bitter pill.
  5. Your employability becomes more about your “likability” and other problematic things.  Your company’s success determines the shape of your career much more than your own performance.  (Actually, this probably begins the day you start managing people.)
  6. Your time is not your own. Your flaws are no longer cute. You will see your worst failings ripple outward and be magnified and reflected.  (Ditto, applies to all leaders but intensifies as you rise.)
  7. You may never feel the dopamine hit of “i learned something, i fixed something, i did something” that comes so freely as an I.C.  Some people learn to feel satisfaction from managery things, others never do.  Most describe it as a very subdued version of the thrill you get from building things.
  8. You will go home tired every night, unable to articulate what you did that day. You cannot compartmentalize or push it aside. If the project failed for reasons outside your control, you will be identified with the failure anyway.
  9. Nobody really thinks of you as a person anymore, you turn into a totem for them to project shit on. (Things will only get worse if you hit back.)  Can you handle that?  Are you sure?
  10. It’s pretty much a one-way trip.

Sure, there are compensating rewards.  Money, power, impact.  But I’m pointing out the negatives because most people don’t stop to consider them when they start saying they want to try managing managers.  Every manager says that.

The mere existence of a ladder compels us all to climb.

I know people who have climbed, gotten stuck, and wished they hadn’t. I know people who never realized how hard it would be for them to go back to something they loved doing after 5+ years climbing the ladder farther and farther away from tech.  I know some who are struggling their way back, others who have no idea how or where to start.  For those who try, it is hard.  

You can’t go back and forth from engineering to executive, or even director to manager, in the way you can traverse freely between management and engineering as a technologist.

I just want more of you entering management with eyes wide open.  That’s all I’m saying.

If you don’t know what you want, act to maximize your options.

Engineering is a creative act. Managing engineers will require your full attentive and authentic self. You will be more successful if you figure out what that self is, and honor its needs.  Try to resist the default narratives about promotions and titles and roles, they have nothing to do with what satisfies your soul.  If you have influence, use it to lean hard against things like paying managers more than ICs of the same level.[2]

It’s totally normal not to know who you want to be, or have some passionate end goal.  It’s great to live your life and work your work and keep an eye out for interesting opportunities, and see what resonates.  It’s awesome when you get asked to step up and opportunistically build on your successes.

If you want a sustainable career in tech, you are going to need to keep learning your whole life. The world is changing much faster than humans evolved to naturally adapt, so you need to stay a little bit restless and unnaturally hungry to succeed in this industry.

The best way to do that is to make sure you a) know yourself and what makes you happy, b) spend your time mostly in alignment with that. Doing things that make you happy give you energy. Doing things that drain you are antithetical to your success. Find out what those things are, and don’t do them. 

Don’t be a martyr, don’t let your spending habits shackle you, and don’t build things that trouble your conscience.

And have fun.

Yours in inverting $(allthehierarchies),
charity.

img_5680

 

[1] Important point: I am not saying you can’t pick up the skills and patience to practice engineering again.  You probably can!  But employers are extremely reluctant to pay you a salary as an engineer if you haven’t been paid to ship code recently.  The tipping point for hireability comes long before the tipping point for learning ability, in my experience.

[2] It is in no one’s best interest for money to factor into the decision of whether to be a manager or not.  Slack pays their managers LESS than engineers of the same level, and I think this is incredibly smart: sends a strong signal of servant leadership.

 

Engineering Management: The Pendulum Or The Ladder