Friday Deploy Freezes Are Exactly Like Murdering Puppies

VOICEOVER: “Previously, on twitter …”

So, that happened.

I hadn’t seen anyone say something like this in quite a while.  I remember saying things like this myself as recently as, oh, 2016, but I thought the zeitgeist had moved on to continuous delivery.

Which is not to say that Friday freezes don’t happen anymore, or even that they shouldn’t; I just thought that this was no longer seen as a badge of responsibility and honor, rather a source of mild embarrassment.  (Much like the fact that you still don’t automatedly restore your db backups and verify them every night.  Do you.)

So I responded with an equally hyperbolic and indefensible claim:

Now obviously, OBVIOUSLY, reassigning all your developer cycles is probably a terrible idea.  You don’t get 100x parallel efficiency if you put 100 developers on a single problem.  So I thought it was clear that this said somewhat tongue in cheek, serious-but-not-really.  I was wrong there too.

So let me explain.

There’s nothing morally “wrong” with Friday freezes.  But it is a costly and cumbersome bandage for a problem that you would be better served to address directly.  And if your stated goal is to protect people’s off hours, this strategy is likely to sabotage that goal and cause them to waste far more time and get woken up much more often, and it stunts your engineers’ technical development on top of that.

Fear is the mind-killer.

Fear of deploys is the ultimate technical debt.  How much time does your company waste, between engineers:

  • waiting until it is “safe” to deploy,
  • batching up changes into bigger changes that are decidedly unsafe to deploy,
  • debugging broken deploys that had many changes batched into them,Does Not Kill Us Puppy UPDATED
  • waiting nervously to get paged after a deploy goes out,
  • figuring out if now is a good time to deploy or not,
  • cleaning up terrible deploy-related catastrophuckes

Anxiety related to deploys is the single largest source of technical debt in many, many orgs.  Technical debt, lest we forget, is not the same as “bad code”.  Tech debt hurts your people.

Saying “don’t push to production” is a code smell.  Hearing it once a month at unpredictable intervals is concerning.  Hearing it EVERY WEEK for an ENTIRE DAY OF THE WEEK should be a heartstopper alarm.  If you’ve been living under this policy you may be numb to its horror, but just because you’re used to hearing it doesn’t make it any less noxious.

If you’re used to hearing it and saying it on a weekly basis, you are afraid of your deploys and you should fix that.

If you are a software company, shipping code is your heartbeat.  Shipping code should be as reliable and sturdy and fast and unremarkable as possible, because this is the drumbeat by which value gets delivered to your org.

Deploys are the heartbeat of your company.

Every time your production pipeline stops, it is a heart attack.  It should not be ok to go around nonchalantly telling people to halt the lifeblood of their systems based on something as pedestrian as the day of the week.

Why are you afraid to push to prod?  Usually it boils down to one or more factors:

  • your deploys frequently break, and require manual intervention just to get to a good state
  • your test coverage is not good, your monitoring checks are not good, so you rely on users to report problems back to you and this trickles in over daysfaith
  • recovering from deploys gone bad can regularly cause everything to grind to a halt for hours or days while you recover, so you don’t want to even embark on a deploy without 24 hours of work day ahead of you
  • your deploys are painfully slow, and take hours to run tests and go live.

These are pretty darn good reasons.  If this is the state you are in, I totally get why you don’t want to deploy on Fridays.  So what are you doing to actively fix those states?  How long do you think these emergency controls will be in effect?

The answers of “nothing” and “forever” are unacceptable.  These are eminently fixable problems, and the amount of drag they create on your engineering team and ability to execute are the equivalent of five-alarm fires.

Fix. That.  Take some cycles off product and fix your fucking deploy pipeline.

If you’ve been paying attention to the DORA report or Accelerate, you know that the way you address the problem of flaky deploys is NOT by slowing down or adding roadblocks and friction, but by shipping more QUICKLY.

Science says: ship fast, ship often.

Deploy on every commit.  Smaller, coherent changesets transform into debuggable, understandable deploys.  If we’ve learned anything from recent research, it’s that velocity of deploys and lowered error rates are not in tension with each other, they actually reinforce each other.  When one gets better, the other does too.

So by slowing down or batching up or pausing your deploys, you are materially contributing to the worsening of your own overall state.

If you block devs from merging on Fridays, then you are sacrificing a fifth of your velocity and overall output.  That’s a lot of fucking output.Screen Shot 2019-02-05 at 7.02.43 AM

If you do not block merges on Fridays, and only block deploys, you are queueing up a bunch of changes to all get shipped days later, long after the engineers wrote the code and have forgotten half of the context.  Any problems you encounter will be MUCH harder to debug on Monday in a muddled blob of changes than they would have been just shipping crisply, one at a time on Friday.  Is it worth sacrificing your entire Monday?  Monday-Tuesday?  Monday-Tuesday-Wednesday?

Good judgment matters more than rules.

I am not saying that you should make a habit of shipping a large feature at 4:55 pm on Friday and then sauntering out the door at 5.  For fucks sake.  Every engineer needs to learn and practice good technical judgment around deploy hygiene.  LIke,

  • icecream_ninesDon’t ship before you walk out the door on *any* day.
  • Don’t ship big, gnarly features right before the weekend, if you aren’t going to be around to watch them.
  • Instrument your code, and go and LOOK at the damn thing once it’s live.
  • Use feature flags and other tools that separate turning on code paths from deploys.

But you don’t need rules for this; in fact, rules actually inhibit the development of good judgment!

Most deploy-related problems are readily obvious, if the person who has the context for the change in their heads goes and looks at it.

But if you aren’t looking for them, then sure — you probably won’t find out until user reports start to trickle in over the next few days.

So go and LOOK.

Stop shipping blind.  Actually LOOK at what you ship.

I mean, if it takes 48 hours for a bug to show up, then maybe you better freeze deploys on Thursdays too, just to be safe!  🙄

I get why this seems obvious and tempting.  The “safety” of nodeploy Friday is realized immediately, while the costs are felt later later.  They’re felt when you lose Monday (and Tuesday) to debugging the big blob deplly.  Or they get amortized out over time.  Or you experience them as sluggish ship rates and a general culture of fear and avoidance, or learned helplessness, and the broad acceptance of fucked up situations as “normal”.

But if recovering from deploys is long and painful and hard, then you should fix that.  If you don’t tend to detect reliability events until long after the event, you should fix that.  If people are regularly getting paged on Saturdays and Sundays, they are probably getting paged throughout the night, too.  You should fix that.

On call paging events should be extremely rare.  There’s no excuse for on call being something that significantly impacts a person’s life on the regular.  None.Root Causes Dream Bunny 4x4

I’m not saying that every place is perfect, or that every company can run like a tech startup.  I am saying that deploy tooling is systematically underinvested in, and we abuse people far too much by paging them incessantly and running them ragged, because we don’t actually believe it can be any better.

It can.  If you work towards it.

Devote some real engineering hours to your deploy pipeline, and some real creativity to your processes, and someday you too can lift the Friday ban on deploys and relieve your oncall from burnout and increase your overall velocity and productivity.

On virtue signaling

Finally, I heard from a alarming number of people who admitted that Friday deploy bans were useless or counterproductive, but they supported them anyway as a purely symbolic gesture to show that they supported work/life balance.

This makes me really sad.  I’m … glad they want to support work/life balance, but surely we can come up with some other gestures that don’t work directly counter to their goals of  life/work balance.

Recovery: building a healthy deploy culture

Ways to begin recovering from a toxic deploy culture:

  • Have a deploy philosophy, make sure everybody knows what it is.  Be consistent.
  • Build and deploy on every set of committed changes.  Do not batch up multiple people’s commits into a deploy.
  • Train every engineer so they can run their own deploys, if they aren’t fully automated.  Make every engineer responsible for their own deploys.
  • (Work towards fully automated deploys.)
  • Every deploy should be owned by the developer who made the changes that are rolling out.  Page the person who committed the change that triggered the deploy, not whoever is oncall.
  • Set expectations around what “ownership” means.  Provide observability tooling so they can break down by build id and compare the last known stable deploy with the one rolling out.
  • Never accept a diff if there’s no explanation for the question, “how will you know Graph Everything, Kittenswhen this code breaks?  how will you know if the deploy is not behaving as planned?”  Instrument every commit so you can answer this question in production.
  • Shipping software and running tests should be fast.  Super fast.  Minutes, tops.
  • It should be muscle memory for every developer to check up on their deploy and see if it is behaving as expected, and if anything else looks “weird”.
  • Practice good deploy hygiene using feature flags.  Decouple deploys from feature releases.  Empower support and other teams to flip flags without involving engineers.

Each deploy should be owned by the developer who made the code changes.  But your deploy pipeline needs to have a team that owns it too.  I recommend putting your most experienced, senior developers on this problem to signal its high value.

You can find more tips for boring deploys in my piece on why shipping software should not be scary.

Good teams ship often.

Ultimately, I am not dogmatic about Friday deploys.  Truly, I’m not.  If that’s the only lever you have to protect your time, use it.  But call it and treat it like the hack it is.  It’s a gross workaround, not an ideal state.

Don’t let your people settle into the idea that it’s some kind of moral stance instead of a butt-ugly hack.  Because if you do you will never, ever get rid of it.

Remember: a team’s maturity and efficiency can be represented by how long it takes to get their shit into users’ hands after they write it.  Ship it fast, while it’s still fresh in your developers’ heads.  Ship one change set at a time, so you can swiftly debug and revert them.  I promise your lives will be so much better.  Every step helps.  ❤

charity.

IMG_3768

Friday Deploy Freezes Are Exactly Like Murdering Puppies

On pain, careers, and doing things the hard way.

Part 1

Seven years ago I was working on backend infra for mobile apps at Parse, resenting MongoDB and its accursed single write lock per replica with all my dirty, blackened soul.  That’s when Miles Ward asked me to give a customer testimonial for MongoDB at AWS reinvent.

It was my first time EVER speaking in public, and I had never been more terrified.  I have always been a writer, not a talker, and I was pathologically afraid of speaking in public, or even having groups of people look at me.  I scripted every word, memorized my lines, even printed it all out just in case my laptop didn’t work.  I had nightmares every night.  For three months I woke up every night in a cold sweat, shaking.

And I bombed, completely and utterly.  The laptop DIDN’T work, my limbs and tongue froze, I was shaking so badly I could hardly read my printout, and after I rushed through the last sentences I turned and stumbled robotically off the stage, fully unaware that people were raising their hands and asking questions.  I even tripped over the microphone cord in my haste to escape the stage.

Afterwards I burned with unpleasantries — fear, anger, humiliation, rage at being so bad at anything.  It was excruciating.  For the next two years I sought out every opportunity I could get to talk at a meetup, conference, anything.  I got a prescription for propranolol to help manage the physical symptoms of panic.   I gave 17 more talks that year, spending most nights and weekends working on them or rehearsing, and 21 the year after that.  I hated every second of it.

I hated it, but I burned up my fear and aversion as fuel.  Until around 18 months later, when I realized that I no longer had nightmares and had forgotten to pack my meds for a conference.  I brute forced my way through to the other side, and public speaking became just an ordinary skill or a tool like any other.

part 2

I was on a podcast last week where the topic was career journeys.  They asked me what piece of career advice I would like to give to people.  I promptly said that following your bliss is nice, but I think it’s important to learn to lean into pain.

“Pain is nature’s teacher,” I said.  Feedback loops train us every day, mostly unconsciously.  We feel aversion for pain, and we enjoy dopamine hits, and out of those and other brain chemicals our habits are made.  All it takes is a little tolerance for discomfort and a some conscious tweaking of those feedback loops, and you can train yourself to achieve big things without even really trying.

But then I hesitated.  Yes, leaning in to pain has done well for me in my career.  But that is not the whole story, it leaves off some important truths.  It has also hurt me and held me back.

Misery is not a virtue.  Pain is awful.  That’s why it’s so powerful and primal.  It’s a pre-conscious mechanism, an acute response that kicks in long before your conscious mind.  Even just the suggestion of pain (or memory of past trauma) will train you to twist and contort around to avoid it.

When you are in pain, your horizons shrink.  Your vision narrows, you curl inward. You have to expend enormous amounts of energy just moving forward through the day inch by inch.

Everything is hard when you’re in pain.  Your creative brain shuts down.  Basic life functions become impossible tests.  You have to spend so much time compensating for your reduced capacity that learning new things is nearly impossible.  You can’t pick up on subtle signals when your nerves are screaming in agony.  And you grow numb over time, as they die off from sheer exhaustion.

part 3

I am no longer the CEO of honeycomb.

I never wanted to be CEO; I always fiercely wanted a technical role.  But it was a matter of company survival, and I did my best.  I wasn’t a great CEO, although we did pretty well at the things I am good at or care about.  But I couldn’t expand past them.

I hated every second of it.  I cried every single day for the first year and a half.  I tried to will myself into loving a role I couldn’t stand, tried to brute force my way to success like I always do.  It didn’t get better.  My ability to be present and curious and expansive withered.  I got numb.

Turns out not every problem can be powered through on a high pain tolerance.  The collateral damage starts to rack up.  Sometimes the only way to succeed is to redefine success.

Pain is a terrific teacher, but pain is an acute response.  Chronic pain will hijack your reward pathways, your perspective, your relationships, and every other productive system and leave them stunted.

Leaning in to pain can be powerful if you have the agency and ability to change it, or practice it to mastery, or even just adapt your own emotional responses to it.  If you don’t or you can’t, leaning in to pain will kill you.  Having the wisdom to know the difference is everything.  Or so I’m learning.

From here on out I’ll be in the CTO seat.  I don’t know what that even means yet, but I guess we’ll find out.  Stay tuned.  ❤

charity

img_7678

On pain, careers, and doing things the hard way.

Engineering Management: The Pendulum Or The Ladder

Last night I was out with a dear friend who has been an engineering manager for a year now, and by two drinks in I was rattling off a long list things I always say to newer engineering managers.

Then I remembered: I should write a post! It’s one of my goals this year to write more long form instead of just twittering off into the abyss.Buffy Jaguar 3.5x5

There’s a piece I wrote two years ago, The Engineer/Manager Pendulum,  which is probably my all time favorite.  It was a love letter to a friend who I desperately wanted to see go back to engineering, for his own happiness and mental health.  Well, this piece is a sequel to that one.

It’s primarily aimed at new managers, who aren’t sure what their career options look like or how to evaluate the opportunities that come their way, or how it may expand or shrink their future opportunities.

The first fork in the manager’s path

Every manager reaches a point where they need to choose: do they want to manage engineers (a “line manager”), or do they want to try to climb the org chart? — manage managers, managers of other managers, even other divisions; while Does Not Kill Us Puppy UPDATEDbeing “promoted” from manager to senior manager, director to senior director, all the way up to VP and so forth.   Almost everyone’s instinct is to say “climb the org chart”, but we’ll talk about why you should be critical of this instinct.

They also face a closely related question: how technical do they wish to stay, and how badly do they care?

Are you an “engineering MANAGER” or an “ENGINEERING manager”?

These are not unlike the decisions every engineer ends up making about whether to go deep or go broad, whether to specialize or be a generalist.  The problem is that both engineers and managers often make these career choices with very little information — or even awareness that they are doing it.

And managers in particular then have a tendency to look up ten years later and realize that those choices, witting or unwitting, have made them a) less employable  and b) deeply unhappy.

Lots of people have the mindset that once they become an engineering manager, they should just go from gig to gig as an engineering manager who manages other engineers: that’s who they are now.  But this is actually a very fragile place to sit long-term, as we’ll discuss further on in this piece.

But let’s start at to the beginning, so I can speak to those of you who are considering management for the very first time.

“So you want to try engineering management.”

COOL! I think lots of senior engineers should try management, maybe even most senior engineers.  It’s so good for you, it makes you better at your job. (If you aren’t a senior engineer, and by that I mean at least 7+ years of engineering experience, be very wary; know this isn’t usually in your best interest.)

Hopefully you have already gathered that management is a career change, not a promotion, and you’re aware that nobody is very good at it when they first start.

That’s okay! It takes a solid year or two to find new rhythms and reward mechanisms before you can even begin to find your own voice or trust your judgment. Management problems look easy, deceptively so.  Reasons this is hard include:

  1. Most tech companies are absolutely abysmal at providing any sort of training or structure to help you learn the ropes and find your feet.
  2. untrueEven if they do, you still have to own your own career development.  If learning to be a good engineer was sort of like getting your bachelor’s, learning to be a good manager is like getting your PhD — much more custom to who you are.
  3. It will exhaust you mentally and emotionally in the weirdest ways for much longer than you think it should.  You’ll be tired a lot, and you’ll miss feeling like you’re good at something (anything).

This is because you need to change your habits and practices, which in turn will actually change who you are.  This takes time.  Which is why …

The minimum tour of duty as a new manager is two years.

If you really want to try being a manager, and the opportunity presents itself, do it!  But only if you are prepared to fully commit to a two year long experiment.

Root Causes DolphinCommit to it like a proper career change. Seek out new peers, find new heroes. Bring fresh eyes and a beginner’s mindset. Ask lots of questions. Re-examine every one of your patterns and habits and priorities: do they still serve you? your team?

Don’t even bother thinking about in terms of whether you “enjoy managing” for a while, or trying to figure out if you are are any good at it. Of course you aren’t any good at it yet.  And even if you are, you don’t know how to recognize when you’ve succeeded at something, and you haven’t yet connected your brain’s reward systems to your successes.  A long stretch of time without satisfying brain drugs is just the price of admission if you want to earn these experiences, sadly.

It takes more than one year to learn management skills and wire up your brain to like it.  If you are waffling over the two year commitment, maybe now is not the time.  Switching managers too frequently is disruptive to the team, and it’s not fair to make them report to someone who would rather be doing something else or isn’t trying their ass off.

It takes about 3-5 years for your skills to deteriorate.

So you’ve been managing a team for a couple years, and it’s starting to feel … comfortable?  Hey, you’re pretty good at this!  Yay!

With a couple of years under your belt as a line manager, you now have TWO powerful skill sets.  You can build things, AND you can organize people into teams to build even bigger things. Right now, both sets are sharp.  You could return to engineering pretty easily, or keep on as a manager — your choice.

But this state of grace doesn’t last very long. Your technical skills stop advancing when you become a manager, and instead begin eroding.  Two years in, you aren’t the effective tech lead you once were; your information is out of date and full of gaps, the hard parts are led by other people these days.

More critically, your patterns of mind and habits shift over time, and become those of a manager, not an engineer.  Consider how excited an engineer becomes at the prospect of a justifiable greenfield project; now compare to her manager’s glum reaction as she instinctively winces at having to plan for something so reprehensibly unpredictable and difficult to estimate.  It takes time to rewire yourself back.

If you like engineering management, your tendency is to go “cool, now I’m a manager”, and move from job to job as an engineering manager, managing team after team of engineers.  But this is a trap.  It is not a sound long term plan.  It leads too many people off to a place they never wanted to end up: technically sidelined.

Sunglasses Tiger Debugger 3.3x5

Why can’t I just make a career out of being a combo tech lead+line manager?

One of the most common paths to management is this: you’re a tech lead, you’re directing ever larger chunks of technical work, doing 1x1s and picking up some of the people stuff, when your boss asks if you’d like to manage the team.  “Sure!”, you say, and voila — you are an engineering manager with deep domain expertise.

But if you are doing your job, you begin the process of divesting yourself of technical leadership responsibilities starting immediately.  Your own technical development should screech to a halt once you become a manager, because you have a whole new career to focus on learning.

Your job is to leverage that technical expertise to grow your engineers into great senior engineers and tech leads themselves.  Your job is not to hog the glory and squat on the hard problems yourself, it’s to empower and challenge and guide your team.  Don’t suck up all the oxygen: you’ll stunt the growth of your team.

But your technical knowledge gets dated, and your skills atrophy..  The longer it’s been since you worked as an engineer, the harder it will be to switch back.  It gets real hard around three years, and five years seems like a tipping point.[1]

And because so much of your credibility and effectiveness as an engineering leader comes from your expertise in the technology that your team uses every day, ultimately you will be no longer capable of technical leadership, only people management.

On being an “engineering manager” who only does people management

I mean, there’s a reason we don’t lure good people managers away from Starbucks to run engineering teams.  It’s the intersection and juxtaposition of skill sets that gives engineering managers such outsize impact.

The great ones can make a large team thrum with energy.  The great ones can break down a massive project into projects that challenge (but do not overwhelm) a dozen or more engineers, from new grads to grizzled veterans, pushing everyone to grow.  The great ones can look ahead and guess which rocks you are going to die on if you don’t work to avoid them right now.

The great ones are a treasure: and they are rare.  And in order to stay great, they regularly need to go back to the well to refresh their own hands-on technical abilities.

Pointless Ice Cream 3x2.5There is an enormous demand for technical engineering leaders — far more demand than supply.  The most common hackaround is to pair a people manager (who can speak the language and knows the concepts, but stopped engineering ages ago) with a tech lead, and make them collaborate to co-lead the team.  This unwieldy setup often works pretty well.

But most of those people managers didn’t want or expect to end up sidelined in this way when they were told to stop engineering.

If you want to be a pure people manager and not do engineering work, and don’t want to climb the ladder or can’t find a ladder to climb, more power to you.  I don’t know that I’ve met many of these people in my life.  I have met a lot of people in this situation by accident, and they are always kinda angsty and unhappy about it.  Don’t let yourself become this person by accident.  Please.

Which brings me to my next point.

You will be advised to stop writing code or engineering.

Fuck

That.

 ✨

Everybody’s favorite hobby is hassling new managers about whether or not they’ve stopped writing code yet, and not letting up until they say that they have.  This is a terrible, horrible, no-good VERY bad idea that seems like it must originally have been a botched repeating of the correct advice, which is:

Stop writing code and engineering

in the critical path

Can you spot the difference?  It’s very subtle.  Let’s run a quick test:

  • Authoring a feature?  ⛔️
  • Covering on-call when someone needs a break?  ✅
  • Diving on the biggest project after a post mortem?  ⛔️
  • Code reviews?  ✅
  • Picking up a p2 bug that’s annoying but never seems to become top priority?  ✅
  • Insisting that all commits be gated on their approval?  ⛔️
  • Cleaning up the monitoring checks and writing a library to generate coverage?  ✅

The more you can keep your hands warm, the more effective you will be as a coach and a leader.  You’ll have a richer instinct for what people need and want from you and each other, which will help you keep a light touch.  You will write better reviews and resolve technical disputes with more authority.  You will also slow the erosion and geriatric creep of your own technical chops.

I firmly believe every line manager should either be in the on call rotation or pinch hit liberally and regularly, but that’s a different post.

Technical Leadership track

If you  love technology and want to remain a subject-matter expert in designing, building and shipping cutting-edge technical products and systems, you cannot afford to let yourself drift too far or too long away from hands-on engineering work.  You need to consciously cultivate your path , probably by practicing some form of the engineer/manager pendulum.

If you love managing engineers — if being a technical leader is a part of your identity that you take great pride in, then you must keep up your technical skills and periodically DIstrust Kittens 2.5x3invest in your practice and renew your education.  Again: this is simply the price of admission.  You need to renew your technical abilities, your habits of mind, and your visceral senses around creating and maintaining systems.  There is no way to do this besides doing it.  If management isn’t a promotion, then returning to hands-on work isn’t a demotion, either.  Right?

One warning: Your company may be great, but it doesn’t exist for your benefit.  You and only you can decide what your needs are and advocate for them.  Remember that next time your boss tries to guilt you into staying on as manager because you’re so badly needed, when you can feel your skills getting rusty and your effectiveness dwindling.  You owe it to yourself to figure out what makes you happy and build a portfolio of experiences that liberate you to do what you love.  Don’t sacrifice your happiness at the altar of any company.  There are always other companies.

Honestly, I would try not to think of yourself as a manager at all: you are an “engineering leader” performing a tour of duty in management.  You’re pursuing a long term strategy towards being a well-respected technologist, someone who can sling code, give informed technical guidance and explain in detail customized for to anyone at any level of sophistication.

Organizational Leadership Track

Most managers assume they want to climb the ladder.  Leveling up feels like an achievement, and that can feel impossible to resist.

Resist it.  Or at least, resist doing it unthinkingly.  Don’t do it because the ladder is there and must be climbed.  Know as much as you can about what you’re in for before you decide it’s what you want.

Here are a few reasons to think critically about climbing the ladder to director and executive roles.

  1. Your choices shrink. There are fewer jobs, with more competition, mostly at bigger companies.  (Do you even like big companies?)
  2. You basically need to do real time at a big company where they teach effective management skills, or you’ll start from a disadvantage.
  3. Bureaucracies are highly idiosyncratic, skills and relationships may or may not transfer with you between companies. As an engineer you could skip every year or two for greener pastures if you landed a crap gig.  An engineer has … about 2-3x more leeway in this regard than an exec does.  A string of short director/exec gigs is a career ender or a coach seat straight to consultant life.
  4. You are going to become less employable overall.  The ever-higher continuous climb almost never happens, usually for reasons you have no control over.  This can be a very bitter pill.
  5. Your employability becomes more about your “likability” and other problematic things.  Your company’s success determines the shape of your career much more than your own performance.  (Actually, this probably begins the day you start managing people.)
  6. Your time is not your own. Your flaws are no longer cute. You will see your worst failings ripple outward and be magnified and reflected.  (Ditto, applies to all leaders but intensifies as you rise.)
  7. You may never feel the dopamine hit of “i learned something, i fixed something, i did something” that comes so freely as an I.C.  Some people learn to feel satisfaction from managery things, others never do.  Most describe it as a very subdued version of the thrill you get from building things.
  8. You will go home tired every night, unable to articulate what you did that day. You cannot compartmentalize or push it aside. If the project failed for reasons outside your control, you will be identified with the failure anyway.
  9. Nobody really thinks of you as a person anymore, you turn into a totem for them to project shit on. (Things will only get worse if you hit back.)  Can you handle that?  Are you sure?
  10. It’s pretty much a one-way trip.

Sure, there are compensating rewards.  Money, power, impact.  But I’m pointing out the negatives because most people don’t stop to consider them when they start saying they want to try managing managers.  Every manager says that. parasite

The mere existence of a ladder compels us all to climb.

I know people who have climbed, gotten stuck, and wished they hadn’t. I know people who never realized how hard it would be for them to go back to something they loved doing after 5+ years climbing the ladder farther and farther away from tech.  I know some who are struggling their way back, others who have no idea how or where to start.  For those who try, it is hard.  

You can’t go back and forth from engineering to executive, or even director to manager, in the way you can traverse freely between management and engineering as a technologist.

I just want more of you entering management with eyes wide open.  That’s all I’m saying.

If you don’t know what you want, act to maximize your options.

Engineering is a creative act. Managing engineers will require your full attentive and authentic self. You will be more successful if you figure out what that self is, and honor its needs.  Try to resist the default narratives about promotions and titles and roles, they have nothing to do with what satisfies your soul.  If you have influence, use it to lean hard against things like paying managers more than ICs of the same level.[2]

gpsun2It’s totally normal not to know who you want to be, or have some passionate end goal.  It’s great to live your life and work your work and keep an eye out for interesting opportunities, and see what resonates.  It’s awesome when you get asked to step up and opportunistically build on your successes.

If you want a sustainable career in tech, you are going to need to keep learning your whole life. The world is changing much faster than humans evolved to naturally adapt, so you need to stay a little bit restless and unnaturally hungry to succeed in this industry.

The best way to do that is to make sure you a) know yourself and what makes you happy, b) spend your time mostly in alignment with that. Doing things that make you happy give you energy. Doing things that drain you are antithetical to your success. Find out what those things are, and don’t do them. 

Don’t be a martyr, don’t let your spending habits shackle you, and don’t build things that trouble your conscience.

And have fun.

Yours in inverting $(allthehierarchies),
charity.

img_5680

 

[1] Important point: I am not saying you can’t pick up the skills and patience to practice engineering again.  You probably can!  But employers are extremely reluctant to pay you a salary as an engineer if you haven’t been paid to ship code recently.  The tipping point for hireability comes long before the tipping point for learning ability, in my experience.

[2] It is in no one’s best interest for money to factor into the decision of whether to be a manager or not.  Slack pays their managers LESS than engineers of the same level, and I think this is incredibly smart: sends a strong signal of servant leadership.

 

Engineering Management: The Pendulum Or The Ladder

Software Sprawl, The Golden Path, and Scaling Teams With Agency

gplanis

Stop me if you’ve heard this one before.

The company is growing like crazy, your engineering team keeps rising to the challenge, and you are ferociously proud of them.  But some cracks are beginning to show, and frankly you’re a little worried.  You have always advocated for engineers to have broad latitude in technical decisions, including choosing languages and tools.  This autonomy and culture of ownership is part of how you have successfully hired and retained top talent despite the siren song of the Faceboogles.

But recently you saw something terrifying that you cannot unsee: your company is using all the languages, all the environments, all the databases, all the build tools.  Shit!!!  Your ops team is in full revolt and you can’t really blame them.  It’s grown into an unsupportable nightmare and something MUST be done, but you don’t know what or how — let alone how to solve it while retaining the autonomy and personal agency that you all value so highly.

gpredwedd

I hear a version of this everywhere I’ve gone for the past year or two.  It’s crazy how often.  I’ve been meaning to write my answer up for ages, and here it (finally) is.

gpathcartoon

First of all: you aren’t alone.  This is extremely common among high-performing teams, so congratulations.  Really!

There actually seems to be a direct link between teams that give engineers lots of leeway to own their technical decisions and that team’s ability to hire and retain top-tier talent, particularly senior talent.   Everything is a tradeoff, obviously, but accepting somewhat more chaos in exchange for a stronger sense of individual ownership is usually the right one, and leads to higher-performing teams in the long run.

Second, there is actually already a well-trod path out of this hole to a better place, and it doesn’t involve sacrificing developer agency.  It’s fairly simple!  Just five short steps, which I will describe to you now.

gpjava

How to build a golden path and reverse software sprawl

  1. Assemble a small council of trusted senior engineers.
  2. Task them with creating a recommended list of default components for developers to use when building out new services.  This will be your Golden Path, the path of convergence (and the path of least resistance).
  3. Tell all your engineers that going forward, the Golden Path will be fully supported by the org.  Upgrades, patches, security fixes; backups, monitoring, build pipeline; deploy tooling, artifact versioning, development environment, even tier 1 on call support.  Pave the path with gold.  Nobody HAS to use these components … but if they don’t, they’re on their own.  They will have to support it themselves.
  4. Work with team leads to draw up an umbrella plan for adopting the Golden Path for their current projects as well as older production services, as much as is reasonable or possible or desirable.  Come up with a timeline for the whole eng org to deprecate as many other tools as possible.  Allocate real engineering time to the effort.  Hell, make a party out of it!
  5. After the cutoff date (and once things have stabilized), establish a regular process for reviewing and incorporating feedback about the blessed Path and considering any proposed changes, additions or removals.

There you go.  That’s it.  Easy, right??

(It’s not easy.  I never said it was easy, I said it was simple.  👼🏼)

Your engineers are currently used to picking the best tool for the job by optimizing locally.  What data store has a data model that is easiest for them to fit to their needs?  Which language is fastest for I/O throughput?  What are they already proficient in?  What you need to do is start building your muscles for optimizing globally.  Not in isolation of other considerations, but in conjunction with them.  It will always be a balancing act between optimizing locally for the problem at hand and optimizing globally for operability and general sanity.

(Oh, incidentally, requiring an engineer to write up a proposal any time they want to use a non-standard component, and then defend their case while the council grills them in person — this will be nothing but good for them, guaran-fucking-teed.)

Let’s go into a bit more detail on each of the five points.  But quick disclaimer: this is not a prescription.  I don’t know your system, your team, your cultural land mines or technical interdependencies or anything else about your situation.  I am just telling stories here.

gpjon

1. Assemble your council

Three is a good number for a council.  More than that gets unwieldy, and may have trouble reaching consensus.  Less than three and you run into SPOFs.  You never want to have a single person making unilateral decisions because a) the decision-making process will be weaker, b) it sets that person up for too much interpersonal friction, and c) it denies your other engineers the opportunity to practice making these kinds of decisions.

  • Your council members need technical breadth more than depth, and should be widely respected by engineers.
  • gprain7At least one member should have a long history with the company so they know lots of stupid little details about what’s been tried before and why it failed.
  • At least one member should be deeply versed in practical data and operability concerns.
  • They should all have enough patience and political skill to drive consensus for their decisions.  Absolutely no bombthrowers.

If you’re super lucky, you just tap the three senior technologists who immediately come to mind … your mind and everyone else’s.  If you don’t have this kind of automatic consensus, you may want to let teams or orgs nominate their own representative so they feel they have some say.

gpcss

2.  Task the council with defining a Golden Path

gpsun2

Your council cannot vanish for a week and then descend from the mountain lugging lists engraved on stone tablets.  The process of discovery and consensus is what validates the result.

The process must include talking to and gathering feedback from your engineers, talking to experts outside the company, talking to teams at other companies who are farther along using that technology, coming up with detailed pro/con lists and reasons for their choices.  Maybe sometimes it includes prototyping something or investigating the technical depths … but yeah no mostly it’s just the talking.

You need your council members to have enough political skill to handle these conversations deftly, building support and driving consensus through the process.  Everybody doesn’t have to love the outcome, but it shouldn’t be a *surprise* to anyone by the end.

gphappy

3.  Know where you’re going

Your council should create a detailed written plan describing which technologies are going to be supported … and a stab at what “supported” means.  (Ask the experts in each component what the best practices are for backups, versioning, dependency management, etc.)

You might start with something like this:

* Backend lang: Go 1.11           ## we will no longer be supporting
backend scripting languages
* Frontend lang: ReactJS v 16.5
* Primary db: Aurora v 2.0        ## Yes, we know postgres is "better", 
but we have many mysql experts and 0 pg experts except the one guy 
who is going to complain about this.  You know who you are.
* Deploy pipeline: github -> jenkins + docker -> S3 -> custom k8s 
deploy tooling
* Message broker: kafka v 2.10, confluent build
* Mail: SES
* .... etc

Circulate the draft regularly for feedback, especially with eng managers.  Some team reorganization will probably be necessary to bear the new weight of your support specifications, and managers will need some lead time to wrangle this.

This is also a great time to reconceive of the way on call works at your company.  But I am not going to go into all that here.

gpbutt2

4. Set a date, draft a plan: go!

Get approval from leadership to devote a certain amount of time to consolidating your stack and paying down a lump sum of tech debt.  It depends on your stage of decay, gprainbut a reasonable amount of time might be “25% of engineering time for three months“.  Whatever you agree to, make sure it’s enough to make the world demonstrably better for the humans who run it; you don’t want to leave them with a tire fire or you’ll blow your credibility.

The council and team leads should come up with a rough outer estimate for how long it would take to rewrite everything and move the whole stack on to the Golden Stack.  (It’s probably impossible and/or would take years, but that’s okay.)  Next, look for the quick wins or swollen, inflamed pain points.

  • If you are running two pieces of functionally similar software, like postgres and mysql, can you eliminate one?
  • If you are managing something yourself that AWS could manage for you (e.g. postfix instead of SES, or kafka instead of kinesis), can you migrate that?
  • If you are managing anything yourself that is not core to your business value, in fact, you should try to not manage it.
  • If you are running any services by hand on an AWS instance somewhere, could you try using a service?
  • If you are running your own monitoring software, etc … can you not?
  • If you have multiple versions of a piece of software, can you upgrade or consolidate on one version?

gpdied

The hardest parts are always going to be the ones around migrating data or rewriting components.  Not everything is worth doing or can afford to be done in the time span of your project time, and that’s okay.

Next, brainstorm up some carrots.  Can you write templates so that anybody who writes a service using your approved library, magically gets monitoring checks without having to configure anything?  Can you write a wrapper so they get a bunch of end-to-end tests for free?  Anything you can do to delight people or save them time and effort by using your preferred components is worth considering.  gps8

(By the way, if you don’t have any engineers devoted to internal tooling, you’re probably way overdue at this point.)

Pay down as much debt as you can, but be pragmatic: it’s better to get rid of five small things than one large thing, from a support perspective.  Your main goal is to shrink the number of types of software your team has to support, particularly databases.

Do look for ways to make it fun, like … running a competition to see who can move the most tools to AWS in a week, or throwing a hack week party, or giving dorky prizes like trophies that entitle you to put your manager on call instead of you for a day, etc.

gpcersei

5. Make the process sustainable

After your target date has come and gone, you probably want to hold a post mortem retrospective and do lots of listening.  (Well — first might I recommend a bubble bath and a bottle of champagne?  But then a post mortem.)

Nothing is ever fixed forever.  The company’s needs are going to expand and contract, gpsrsand people will come and go, because change is the only constant.  So you need to bake some flex into your system.  How are you going to handle the need for changes to the Golden Path?  Monthly discussions?  An email list?  Quarterly meetings with a formal agenda?  I’ve seen people do all of these and more, it doesn’t really matter afaict.

Nobody likes a cabal, though, so the original council should gradually rotate out.  I recommend replacing one person at a time, one per quarter, and rotating in another senior engineer in their place.  This provides continuity while giving others a chance to learn these technical and political skills.

In the end, engineers are still free to use any tool or component at any time, just like before, only now they are solely responsible for it, which puts pressure on them not to do it unless REALLY necessary.  So if someone wants to propose adding a new tool to the default golden path, they can always add it themselves and gain some experience in it before bringing it to the council to discuss a formal place for it.

gplinux

That’s all folks

See, wasn’t that simple?

(It’s never simple.)

I dearly wish more people would write up their experiences with this sort of thing in detail.  I think engineering teams are too reluctant to show their warts and struggles to the world — or maybe it’s their executives who are afraid?  Dunno.

Regardless, I think it’s actually a highly effective recruiting tool when teams aren’t afraid to share their struggles.  The companies that brag about how awesome they are are the ones who come off looking weak and fragile.  Whereas you can always trust the ones gpwvwho are willing to laugh about all the ways they screwed up.  Right?

In conclusion, don’t feel like an asshole for insisting on some process here.  There should be friction around adding new components to your stack.  (Add in haste, repent at leisure, as they say.)  Anybody who argues with you probably needs to be exposed to way, way more of the support load for that software.  That’s my professional opinion.

Anyway.  You win or you die.  Good luck with your sprawl.

charity

IMG_2433

 

 

Software Sprawl, The Golden Path, and Scaling Teams With Agency

How to Run a Tech Leadership Skill Share

“How can I learn to be a better manager?  I have no idea what I’m doing.”

“I’m a tech lead, and responsible for shipping large projects, but all of my power is informal.  How do I get people to listen to me and do what I need them to?”

“As a manager, I don’t feel safe talking to anyone about my work problems.  What if it gets passed around as gossip?”

Good questions.

colorful-rainbow

Leadership is such a weird thing.[1]  Leadership is something every one of us does more and more of as we get older and more experienced, but mostly you learn leadership lessons from trial and error and failing a lot.  Which is too bad when you’re doing something you really care about, for the first time.

(Like starting a company.)

I’ve read books on leadership.  I’ve been semi-consensually subjected to management training, I’ve had coaches, I’ve tried therapy and mentors.  Most of this has been impressively (and expensively) unhelpful.

There’s only one thing that has reliably accelerated my development as a leader or manager, and that is forming bonds and swapping stories with my peers.

Stories are power tools.

IMG_1413A story is a tool.  The more stories you have about how other people have solved a problem like yours, the more tools you have.

People are very complicated puzzles, and the more tools you have the more likely you are to find a tool that works.

Unlike management books which speak in abstractions and generalities, stories are real and specific.  When you have the storyteller in front of you, you can drill down and find out more about how the situation was like your own or not, and what they wish they’d done in retrospect.

Details matter.  Context matters.

Sometimes all you really need is a sympathetic ear to listen and make murmuring noises of encouragement while you work it out yourself out loud.  Sometimes they have grappled with a similar situation and can tell you how it all worked out, what they wish they’d known.  Sometimes they will cut you off and tell you to quit feeling sorry for yourself or sabotaging yourself.. if you’re lucky.

Peers?? Why not a ‘mentor’?

No insult meant to anyone who gets a lot out of mentoring, but it isn’t really my bag.  I’ve always had.. let’s say issues with authority.  Which is a nice way of saying “never met a power structure I didn’t simultaneously want to crush and invert”.  So IIMG_0304 prefer the framing of “peers” over even the relatively tame hierarchy of the mentor-mentee relationship.

I mean, one-way relationships are fucked up.  Lots of my peers are more junior than me and some are more senior, yet somehow we all manage to be givers as well as takers.  And if you’re both giving support and receiving it, then what the fuck do you need different roles like mentor and mentee?

I don’t want to be someone’s mentor.  I want to be their friend and to sometimes be helpful.  I don’t want to be someone’s “mentee” either, that makes me feel like their charity case (ha ha).

But friends and peers?  Those just make my life better and awesomer.

From each according to their ability, to each according to their need.

IMG_4766The first year or two of honeycomb, I had a small list of friends who I got dinner or drinks with once a month like clockwork.  Most of them were founders or execs or had been at one point, so they knew how depressed I was without needing to say so.

They listened to my stories (even though I was terrible company), and shared plenty of their own.  They just kept showing up, reminding me to sleep, asking if they could help, not taking it personally whatever state I was in.

These friendships carried me through some dark times.  When Christine’s role required her to level up at leadership skills, I encouraged her to get some peers too.  And that’s when I began to realize some of the limitations of the 1×1 model: it’s very time consuming, and doesn’t scale well.IMG_3881

But hey! scaling problems are fun.  😀   I decided to pull together a peer group where people could come together and give and get support all at the same time.

(Actually two groups.  One for me, one for Christine, so we could complain about each other in proper peace and privacy.)

Practicing vulnerability, establishing intimacy.

It took some time to assemble the right groups, but then we met weekly for 6 weeks straight, and after that roughly once a month for a year.   The six starter weeks were intended to help us practice vulnerability and establish intimacy in a compressed time frame.

IMG_3697Last week was our one-year birthday.

There’s something sterile about management books and leadership material, something that makes it hard for me to emotionally connect my problems to the solutions they preach.  I want advice from someone who knows me in all my strengths and weaknesses, who knows what advice I can take and perform authentically and what I can’t.  Context matters.  Who we are matters.

As word has started to get around about the group, sometimes people ask me about joining or how to form a group of their own.  Turns out lots of people are hungry to get better at leadership, and there are precious few resources.

That’s why I decided to write up and publish my notes.  Everything I learned along the way about how to run a tech leadership skill swap — the logistics, the facilitation, the homework, the ground rules.  Who to invite.  Recommended reading lists.

Here it is: https://github.com/charity/tech-leads-skill-share/.  [2]

(It’s a little rough, but I positively cannot spare any more time.)

This shit is hard.  You need a posse.

Would you like to run a tech leads skill swap?  Please tell me if you do, I would love to know!  IMG_5638I’m happy to help you get started with a phone call, if you want.

All I ask is that you try to pull together a posse that’s at least 50% women, queers, and other marginalized folks.

Good luck. ~charity

IMG_6044[1] I take a pretty expansive view of leadership.  For example, an intern might exercise leadership in the vaunted area of database backups — just by volunteering to own backups, reliably performing said backups and serving as a point of coordination and education for how we do backups here at $corp.

If you have expertise and people rely on you for it, this is a legit form of influence and power … in other words, that’s leadership.

[2] A HUGE thanks to Rachel Chalmers for scribing a first draft of these notes, and to Kris for running the other group and contributing the homework sheet and stories of a related group at twitter.

How to Run a Tech Leadership Skill Share

Shipping Software Should Not Be Scary

On twitter this week, @srhtcn noted that “Many incidents happen during or right after release” and asked for advice on ways to fix this.

And he’s right!  Rolling out new software is the proximate cause for the overwhelming majority of incidents, at companies of all sizes.  Upgrading software is both a necessity and a minor insanity, considering how often it breaks things.

Image result for deploy production memeI’m not going to recap the history of continuous integration and delivery, suffice it to say that we now know that smaller and more frequent changes are much safer than larger and less frequent changes.

But it’s still risky.  And most issues are still caused by humans and our pesky need for “improvements”.  So what can be done?

It’s not ok for software releases to be scary and hazardous

First of all: If releasing is risky for you, you need to fix that.  Make this a priority.  Track your failures, practice post mortems, evaluate your on call practices andImage result for test in production meme culture.  Know if you’re getting better or worse.  This is a project that will take weeks if not months until you can be confident in the results.

You have to fix it though, because these things are self-reinforcing.  If shipping changes is scary and fraught, people will do it less and it will get even MORE scary and treacherous.

Likewise, if you turn it into a non-cortisol inducing event and set expectations, engineers will ship their code more often in smaller diffs and therefore break the world less.

Fixing deploys isn’t about eliminating errors, it’s about making your pipeline resilient to errors.  It’s fundamentally about detecting common failures and recovering from them, without requiring human intervention.

Value your tools more

As an short term patch, you should run deploys in the mornings or whenever everyone is around and fresh.  Then take a hard look at your deploy pipeline.

In too many organizations, deploy code is a technical backwater, an accumulation of crufty scripts and glue code, forked gems and interns’ earnest attempts to hack up Capistrano.  It usually gives off a strong whiff of “sloppily evolved from many 2 am patches with no code review”.Image result for test in production meme

This is insane.  Deploy software is the most important software you have.  Treat it that way: recruit an owner, allocate real time for development and testing, bake in metrics and track them over time.

If it doesn’t have an owner, it will never improve.  And you will need to invest in frequent improvements even after you’re over this first hump.

  • Signal high organizational value by putting one of your best engineers on it.
  • Recruit help from the design side of the house as well.  The “right” thing to do must be the fastest, easiest thing to do, with friendly prompts and good docs.  No “shortcuts” for people to reach for at the worst possible time.  You need user research and design here.  Image result for deploy production meme
  • Track how often deploys fail and why.  Managers should pay close attention to this metric, just like the one for people getting interrupted or woken up, and allocate time to fixing this early whenever it sags.  Before it gets bad.
  • Allocate real time for development, testing, and training — don’t expect the work to get shoved into people’s “spare time” or post mortem cleanup time.  Make sure other managers understand the impact of this work and are on board.  Make this one of your KPIs.Image result for deploy production meme

In other words, make deploy tools a first class citizen of your technical toolset.  Make the work prestigious and valued — even aspirational.  If you do performance reviews, recognize the impact there.

(Btw, “how we hardened our deploys” is total Velocity-bait (&& other practitioner conferences) as well as being great for recruiting and general visibility in blog post form.  People love these stories; there definitely aren’t enough of them.)

Turn software engineers into software owners

The canonical CI/CD advice starts with “ship early, ship often, ship smaller change sets”.  That’s great advice: you should definitely do those things.  But they are covered plenty elsewhere.  What’s software ownership?

Software ownership is the natural end state of DevOps.  Software engineers, operations engineers, platform engineers, mobile engineers — everyone who writes code should be own the full lifecycle of their software.

Software owners are people who:

  1. Write codeImage result for deploy production meme
  2. Can deploy and roll back their own code
  3. Are able to debug their own issues in prod (via instrumentation, not ssh)

If you’re lacking any one of those three ingredients, you don’t have ownership.

Why ownership?  Because software ownership makes for better engineers, better software, and a better experience for customers.  It shortens feedback loops and means the person debugging is usually the person with the most context on what has recently changed.

Some engineers might balk at this, but you’ll be doing them a favor.  We are all distributed systems engineers now, and distributed systems require a much higher level of operational literacy.  May as well start today.

Fail fast, fix fast

This is about shifting your mindset from one of brittleness and a tight grip, to one of flexibility where failures are no big deal because they happen all the time, don’t impact users, and give everyone lots of practice at detecting and recovering from them.

Here are a few of the best practices you should adopt with this practice.

Make operability a high-value skill set.  Never promote someone to “senior engineer” if they can’t deploy and debug their Image result for test in production memeown code.

Software engineers don’t have to become operational experts.  They do need to know the bare basics of instrumentation, deploy/revert, and debugging.

Everyone who puts software in production needs to understand and feel responsible for the full lifecycle of their code, not just how it works in their IDE.

Baking: it’s not just for cookies

Shipping something to production is a process of incrementally gaining confidence, not a switch you can flip.

You can’t trust code until it’s been in prod a while, Image result for deploy production memeuntil you’ve seen it perform under a wide range of load and concurrency scenarios, in lots of partial failure modes.  Only over time can you develop confidence in it not being terrible.

Nothing is production except production.  Don’t rely on never failing; expect failure, embrace failure.  Practice failure!  Build guard rails around your production systems to help you find and fix problems quickly.

The changes you need to make your pipeline more resilient are roughly the same changes you need to safely test in production.  These are a few of your guard rails.

  • Use feature flags to switch new code paths on and offImage result for test in production meme
  • Build canaries for your deploy process, so you can promote releases gracefully and automatically to larger subsets of your traffic as you gain confidence in them
  • Create cohorts.  Deploy to internal users first, then any free tier, etc in order of ascending importance.  Don’t jump from 10% to 25% to 50% and then 100% — some changes are related to saturating backend resources, and the 50%-100% jump will kill you.
  • Have robots check the health of your software as it rolls out to decide whether to promote the canary.  Over time the robot checks will mature and eventually catch a ton of problems and regressions for you.

The quality of code is not knowable before it hits production.  You may able to spot some problems, but you can never guarantee a lack of then.  It takes time to bake a new release and gain incremental confidence in new code.

In summary.

  1. Get someone to own the deploy software
  2. Value the work
  3. Create a culture of software ownership
  4. LOOK at what you’ve done after you do it
  5. Be suspicious of new versions until they prove themselves

Image result for deploy production meme

Two blog posts in one weekend!  That’s definitely never happened before.  Thanks to Baron for asking me to draft this up following the weekend’s twitter thread: https://twitter.com/mipsytipsy/status/1030340072741064704.

 

 

Shipping Software Should Not Be Scary

On Engineers and Influence

(Based on yesterday’s tweetstorm and the ensuing conversation, https://twitter.com/mipsytipsy/status/1029608573217587201)

Let’s talk about influence. As an engineer, how do you get influence? What does influence look like, what is it rooted in, how do you wield it or lose it? How is it different from the power and influence you might have as a manager?[0]

This often comes up in the context of ICs who desperately want to become managers in order to have more access to information and influence over decisions. This is a bad signal, though it’s sadly very common.

When that happens, you need to do some soul-searching. Does your org make space for senior ICs to lead and own decisions? Do you have an IC track that runs parallel to the manager track at least as high as director level? Are they compensated equally? Do youImage result for engineer software meme individual contributorhave a career ladder? Are your decision-making processes mysterious to anyone who isn’t a manager? Don’t assume what’s obvious to you is obvious to others; you have to ask around.

If so, it’s probably their own personal baggage speaking. Maybe they don’t believe you. Maybe they’ve only worked in orgs where managers had all the power. Maybe they’ve even worked in lots of places that said the exact same things as you are saying about how ICs can have great impact, but it was all a lie and now they’re burned. Maybe they aren’t used to feeling powerful for all kinds of reasons.

Regardless, people who want to be managers in order to perpetuate a bad power structure are the last people you want to be managers.[1]

But what does engineering influence look like?  How do your powers manifest?

I am going to avoid discussing the overlapping and interconnected issues of gender, race and class, let’s just acknowledge that it’s much more structurally difficult for some to wield power than for others, ok?

The power to create

Doing is the engineering superpower. We create things with just a laptop and our brain! It’s incredible! We don’t have to constantly convince and cajole and coerce others into building on our behalf, we can just build.

This may seem basic, but it matters. Creation is the ur-power from which all our forms of power flow. Nothing gets built unless we agree to build it (which makes this an ethical issue, too).

Facebook had a poster that said “CODE WINS ARGUMENTS”. Problematic in many ways, absolutely. But how many times have you seen a technical dispute resolved by who wasImage result for code wins arguments facebook willing to do the work? Or “resolved” one way.. then reversed by doing? Doing ends debates. Doing proves theories. Doing is powerful. (And “doing” doesn’t only mean “write code”.)

Furthermore, building software is a creative activity, and doing it at scale is an intensely communal one. As a creative act, we are better builders when we are motivated and inspired and passionate about our work (as compared to say, chopping wood). And as a collaborative act, we do better work when we have high trust and social cohesion.

Engineering ability and judgment, autonomy and sense of purpose, social trust and cooperative behaviors: this is the raw stuff of great engineering. Everybody has a mode or two that they feel most comfortable and authoritative operating from: we can group these roughly into archetypes.

(Examples drawn from some of the stupendously awesome senior engineers I’ve gotten to work with over the years, as well as the ways I loved to fling my weight around as an engineer.)

Archetypes of influence

  • “Doing the work that is desperately hard and desperately needed — and often desperately dull.” SOC2 compliance, backups and restores, terrifying refactors, any auth integration ever: if it’s moving the business forward, they don’t give a shit how dull the work is. If you are this engineer, you have a deep well of respect and gratitude.
  • Debugger of last resort.” Often the engineer who has been there the longest or originally built the system. If you are helpful and cheerful with your history and context, this is a huge asset. (People tend to wildly overestimate this person’s indispensability, actually; please don’t encourage this.)Image result for engineer software meme manager
  • The “expert” archetype is closely related. If you are the deep subject matter expert in some technology component, you have a shit ton of influence over anything that uses or touches that component. (You should stay up on impending changes to retain your edge.)
  • There are people who deliver a bafflingly powerful firehose of sustained output, sometimes making headway on multiple fronts at once. Some work long hours, others just have an unerring instinct for how to maximize impact (this sometimes maps to junior/senior manifestations). Nobody wants to piss off those people. Their consent is critical for … everything. Their participation will often turbo charge a project or pull a foundering effort over the finish line.

Not all influence is rooted in raw technical strength or output.  Just a few of the wide variety of creative/collaborative/interpersonal strengths:

  • Some engineers are infinitely curious, and have a way of consistently sniffing a few steps ahead of the pack. They might seem to be playing around with something pointless, and you want to scold them; then they save your ass from total catastrophe. You learn to value their playing around.
  • Some engineers solve problems socially, by making friends and trading tips and fixes and favors in the industry. Don’t underestimate social debugging, it’s often the quickest path to the right answer.Image result for influence meme
  • Some are dazzlingly lazy and blow your mind with their elegant shortcuts and corners correctly cut.
  • Some are recruiting magnets, and it’s worth paying their salary just for all the people who want to work with them again.
  • Some are skilled at driving consensus among stakeholders.
  • Some are killer explainers and educators and storytellers.
  • Some are the senior engineer everyone silently wants to grow up to be.
  • Some can tell such an inspiring story of tomorrow that everyone will run off to make it so.
  • Some teach by turning code reviews into a pedagogical art form.
  • Some make everyone around them somehow more productive and effective. Some create relentless forward momentum. Some are good at saying no.

And there are a few special wells of power that bear calling out as such.

  • Engineers who have been managers are worth their weight in gold.  They can translate business goals for junior engineers in their native language with impeccable credibility (something managers never really have, esp in junior engineers’ eyes.). They make strong tech leads, they can carve up projects into components that challenge but do not overwhelm each contributor while hitting deadlines.
  • Some engineers are a royal pain in the ass because they questionImage result for engineer software meme individual contributor and challenge every system and hierarchy. But these are sharp, powerful rocks that can polish great teams. Though they do require a strong manager, to channel t
    heir energy towards productive dialogue and improvement and keep them from pissing off the whole team.
  • And let’s not forget engineers who are on call. If you have a healthy on call culture,your ownership over production creates a deep, deep well of power and moral authority — to make demands, drive change, to prioritize. On call should not be a shit salad served up to those who can’t refuse, it should be a badge of honor and seriousness shouldered by every engineer who ships code. (And it should not be miserable or regularly life-impacting.)

… I could go on all day. Engineering is such a powerful role and skill set. It’s definitely worth unpacking where your own influence comes from, and understanding how others perceive your strengths.

Most forms of power boil down to “influence, wielded”.

But just banging out code is not enough. You may have credibility, but having it is not the same as using it. To transform influence into power you have to use it.  And the way you use it is by communicating.

What’s locked up in your head has no impact on the rest of us.  You have to get it out.

You can do this in lots of ways: by writing, in 1x1s, conversations with small groups, openly recruiting allies, convincing someone with explicit authority, broadcasting inImage result for engineer software meme individual contributorpublic, etc.

Because engineering is a creative activity, authoritarian power is actually quite brittle and damaging. The only sustainable forms of power are so-called “soft powers” like influencing and inspiring, which is why good managers use their soft power freely and hard power sparingly/with great reluctance. If your leadership invokes authority on the regular, that’s an antipattern.[2]

If you don’t speak up, you don’t have the right to sit and fume over your lack of influence. And speaking up does mean being vulnerable — and sometimes wrong — in front of other people.

This is not a zero-sum game.

Most of you have far more latent power than you realize or are used to wielding, because you don’t feel powerful or don’t recognize what you do in those terms.

Managers may have hard power and authority, but the real meaty decisions about technical delivery and excellence are more properly made by the engineers closest to them. These belong properly to the doers, in large part because they are the ones who have to support the consequences of these decisions.Image result for engineer software meme individual contributor

Power tends to flow towards managers because they are privy to more information. That makes it important to hire managers who are aware of this and lean against it to push power back to others.

In the same way that submissives have ultimate power in healthy BDSM relationships, engineers actually have the ultimate power in healthy teams. You have the ultimate veto: you can refuse to create.  Demand is high for your skills.  You can usually afford to look for better conditions. Many of you probably should.

And when technical and managerial priorities collide, who wins? Ideally you work together to find the best solution for the business and the people. The teams that feel 🔥on fire🔥 always have tight alignment between the two.

Pick your battles.

One final thought. You can have a lot of say in what gets built and how it gets built, if you cultivate your influence and spend it wisely. But you can’t have a say in everything. It doesn’t work that way.

Think of it like @mcfunley’s famous “innovation tokens”, but for attention and fucks given.
Image result for engineer software meme
The more you use your influence for good outcomes, the more you build up over time, yes … but it’s a precision tool, not background noise. Imagine someone trying to give you a massage by laying down on your whole back instead of pushing their elbow or hand into knots and trigger points. A too-broad target will diffuse your force and limit your potential impact.

Spend your attention tokens wisely.

And once you have influence, don’t forget to use it on behalf of others. Pay attention to those who aren’t being heard, and amplify their voices. Give your time, lend your patronage and credibility, and most of all teach the skills that have made you powerful to others who need them.

charity

P.S. I owe a huge debt to all the awesome senior engineers i’ve gotten to work with.  Mad love to you all.  ❤
Image result for influence meme

  • [0] I successfully answered one (1) of these questions before running out of steam.  Later. 
  • [1] Sheepish confession: this is why I became a manager.
  • [2] It’s also a bad sign if they won’t grant any explicit authority to the people they hold responsible for outcomes. I’m talking about relatively healthy orgs here, not pathological ones where people (often women) are told they don’t need promotions or explicit authority, they should just use their “soft power” — esp when the hard forms of power aligned against with them. That’s setting you up for failure.
  • [3] Some people seem caught off guard by my use of “power” to signal anything other than explicit granted powers by the org. This doesn’t make any sense to me. I find it too depressing and disempowering to think of power as merely granted authority. It doesn’t map to how I experience the world, either. Individual clout is a thing that waxes and wanes and only exists in relation to others’. I’ve seen plenty of weak managers pushed around by strong personalities (which is terrible too).
On Engineers and Influence