Shipping Software Should Not Be Scary

On twitter this week, @srhtcn noted that “Many incidents happen during or right after release” and asked for advice on ways to fix this.

And he’s right!  Rolling out new software is the proximate cause for the overwhelming majority of incidents, at companies of all sizes.  Upgrading software is both a necessity and a minor insanity, considering how often it breaks things.

Image result for deploy production memeI’m not going to recap the history of continuous integration and delivery, suffice it to say that we now know that smaller and more frequent changes are much safer than larger and less frequent changes.

But it’s still risky.  And most issues are still caused by humans and our pesky need for “improvements”.  So what can be done?

It’s not ok for software releases to be scary and hazardous

First of all: If releasing is risky for you, you need to fix that.  Make this a priority.  Track your failures, practice post mortems, evaluate your on call practices andImage result for test in production meme culture.  Know if you’re getting better or worse.  This is a project that will take weeks if not months until you can be confident in the results.

You have to fix it though, because these things are self-reinforcing.  If shipping changes is scary and fraught, people will do it less and it will get even MORE scary and treacherous.

Likewise, if you turn it into a non-cortisol inducing event and set expectations, engineers will ship their code more often in smaller diffs and therefore break the world less.

Fixing deploys isn’t about eliminating errors, it’s about making your pipeline resilient to errors.  It’s fundamentally about detecting common failures and recovering from them, without requiring human intervention.

Value your tools more

As an short term patch, you should run deploys in the mornings or whenever everyone is around and fresh.  Then take a hard look at your deploy pipeline.

In too many organizations, deploy code is a technical backwater, an accumulation of crufty scripts and glue code, forked gems and interns’ earnest attempts to hack up Capistrano.  It usually gives off a strong whiff of “sloppily evolved from many 2 am patches with no code review”.Image result for test in production meme

This is insane.  Deploy software is the most important software you have.  Treat it that way: recruit an owner, allocate real time for development and testing, bake in metrics and track them over time.

If it doesn’t have an owner, it will never improve.  And you will need to invest in frequent improvements even after you’re over this first hump.

  • Signal high organizational value by putting one of your best engineers on it.
  • Recruit help from the design side of the house as well.  The “right” thing to do must be the fastest, easiest thing to do, with friendly prompts and good docs.  No “shortcuts” for people to reach for at the worst possible time.  You need user research and design here.  Image result for deploy production meme
  • Track how often deploys fail and why.  Managers should pay close attention to this metric, just like the one for people getting interrupted or woken up, and allocate time to fixing this early whenever it sags.  Before it gets bad.
  • Allocate real time for development, testing, and training — don’t expect the work to get shoved into people’s “spare time” or post mortem cleanup time.  Make sure other managers understand the impact of this work and are on board.  Make this one of your KPIs.Image result for deploy production meme

In other words, make deploy tools a first class citizen of your technical toolset.  Make the work prestigious and valued — even aspirational.  If you do performance reviews, recognize the impact there.

(Btw, “how we hardened our deploys” is total Velocity-bait (&& other practitioner conferences) as well as being great for recruiting and general visibility in blog post form.  People love these stories; there definitely aren’t enough of them.)

Turn software engineers into software owners

The canonical CI/CD advice starts with “ship early, ship often, ship smaller change sets”.  That’s great advice: you should definitely do those things.  But they are covered plenty elsewhere.  What’s software ownership?

Software ownership is the natural end state of DevOps.  Software engineers, operations engineers, platform engineers, mobile engineers — everyone who writes code should be own the full lifecycle of their software.

Software owners are people who:

  1. Write codeImage result for deploy production meme
  2. Can deploy and roll back their own code
  3. Are able to debug their own issues in prod (via instrumentation, not ssh)

If you’re lacking any one of those three ingredients, you don’t have ownership.

Why ownership?  Because software ownership makes for better engineers, better software, and a better experience for customers.  It shortens feedback loops and means the person debugging is usually the person with the most context on what has recently changed.

Some engineers might balk at this, but you’ll be doing them a favor.  We are all distributed systems engineers now, and distributed systems require a much higher level of operational literacy.  May as well start today.

Fail fast, fix fast

This is about shifting your mindset from one of brittleness and a tight grip, to one of flexibility where failures are no big deal because they happen all the time, don’t impact users, and give everyone lots of practice at detecting and recovering from them.

Here are a few of the best practices you should adopt with this practice.

Make operability a high-value skill set.  Never promote someone to “senior engineer” if they can’t deploy and debug their Image result for test in production memeown code.

Software engineers don’t have to become operational experts.  They do need to know the bare basics of instrumentation, deploy/revert, and debugging.

Everyone who puts software in production needs to understand and feel responsible for the full lifecycle of their code, not just how it works in their IDE.

Baking: it’s not just for cookies

Shipping something to production is a process of incrementally gaining confidence, not a switch you can flip.

You can’t trust code until it’s been in prod a while, Image result for deploy production memeuntil you’ve seen it perform under a wide range of load and concurrency scenarios, in lots of partial failure modes.  Only over time can you develop confidence in it not being terrible.

Nothing is production except production.  Don’t rely on never failing; expect failure, embrace failure.  Practice failure!  Build guard rails around your production systems to help you find and fix problems quickly.

The changes you need to make your pipeline more resilient are roughly the same changes you need to safely test in production.  These are a few of your guard rails.

  • Use feature flags to switch new code paths on and offImage result for test in production meme
  • Build canaries for your deploy process, so you can promote releases gracefully and automatically to larger subsets of your traffic as you gain confidence in them
  • Create cohorts.  Deploy to internal users first, then any free tier, etc in order of ascending importance.  Don’t jump from 10% to 25% to 50% and then 100% — some changes are related to saturating backend resources, and the 50%-100% jump will kill you.
  • Have robots check the health of your software as it rolls out to decide whether to promote the canary.  Over time the robot checks will mature and eventually catch a ton of problems and regressions for you.

The quality of code is not knowable before it hits production.  You may able to spot some problems, but you can never guarantee a lack of then.  It takes time to bake a new release and gain incremental confidence in new code.

In summary.

  1. Get someone to own the deploy software
  2. Value the work
  3. Create a culture of software ownership
  4. LOOK at what you’ve done after you do it
  5. Be suspicious of new versions until they prove themselves

Image result for deploy production meme

Two blog posts in one weekend!  That’s definitely never happened before.  Thanks to Baron for asking me to draft this up following the weekend’s twitter thread: https://twitter.com/mipsytipsy/status/1030340072741064704.

 

 

Shipping Software Should Not Be Scary

On Engineers and Influence

(Based on yesterday’s tweetstorm and the ensuing conversation, https://twitter.com/mipsytipsy/status/1029608573217587201)

Let’s talk about influence. As an engineer, how do you get influence? What does influence look like, what is it rooted in, how do you wield it or lose it? How is it different from the power and influence you might have as a manager?[0]

This often comes up in the context of ICs who desperately want to become managers in order to have more access to information and influence over decisions. This is a bad signal, though it’s sadly very common.

When that happens, you need to do some soul-searching. Does your org make space for senior ICs to lead and own decisions? Do you have an IC track that runs parallel to the manager track at least as high as director level? Are they compensated equally? Do youImage result for engineer software meme individual contributorhave a career ladder? Are your decision-making processes mysterious to anyone who isn’t a manager? Don’t assume what’s obvious to you is obvious to others; you have to ask around.

If so, it’s probably their own personal baggage speaking. Maybe they don’t believe you. Maybe they’ve only worked in orgs where managers had all the power. Maybe they’ve even worked in lots of places that said the exact same things as you are saying about how ICs can have great impact, but it was all a lie and now they’re burned. Maybe they aren’t used to feeling powerful for all kinds of reasons.

Regardless, people who want to be managers in order to perpetuate a bad power structure are the last people you want to be managers.[1]

But what does engineering influence look like?  How do your powers manifest?

I am going to avoid discussing the overlapping and interconnected issues of gender, race and class, let’s just acknowledge that it’s much more structurally difficult for some to wield power than for others, ok?

The power to create

Doing is the engineering superpower. We create things with just a laptop and our brain! It’s incredible! We don’t have to constantly convince and cajole and coerce others into building on our behalf, we can just build.

This may seem basic, but it matters. Creation is the ur-power from which all our forms of power flow. Nothing gets built unless we agree to build it (which makes this an ethical issue, too).

Facebook had a poster that said “CODE WINS ARGUMENTS”. Problematic in many ways, absolutely. But how many times have you seen a technical dispute resolved by who wasImage result for code wins arguments facebook willing to do the work? Or “resolved” one way.. then reversed by doing? Doing ends debates. Doing proves theories. Doing is powerful. (And “doing” doesn’t only mean “write code”.)

Furthermore, building software is a creative activity, and doing it at scale is an intensely communal one. As a creative act, we are better builders when we are motivated and inspired and passionate about our work (as compared to say, chopping wood). And as a collaborative act, we do better work when we have high trust and social cohesion.

Engineering ability and judgment, autonomy and sense of purpose, social trust and cooperative behaviors: this is the raw stuff of great engineering. Everybody has a mode or two that they feel most comfortable and authoritative operating from: we can group these roughly into archetypes.

(Examples drawn from some of the stupendously awesome senior engineers I’ve gotten to work with over the years, as well as the ways I loved to fling my weight around as an engineer.)

Archetypes of influence

  • “Doing the work that is desperately hard and desperately needed — and often desperately dull.” SOC2 compliance, backups and restores, terrifying refactors, any auth integration ever: if it’s moving the business forward, they don’t give a shit how dull the work is. If you are this engineer, you have a deep well of respect and gratitude.
  • Debugger of last resort.” Often the engineer who has been there the longest or originally built the system. If you are helpful and cheerful with your history and context, this is a huge asset. (People tend to wildly overestimate this person’s indispensability, actually; please don’t encourage this.)Image result for engineer software meme manager
  • The “expert” archetype is closely related. If you are the deep subject matter expert in some technology component, you have a shit ton of influence over anything that uses or touches that component. (You should stay up on impending changes to retain your edge.)
  • There are people who deliver a bafflingly powerful firehose of sustained output, sometimes making headway on multiple fronts at once. Some work long hours, others just have an unerring instinct for how to maximize impact (this sometimes maps to junior/senior manifestations). Nobody wants to piss off those people. Their consent is critical for … everything. Their participation will often turbo charge a project or pull a foundering effort over the finish line.

Not all influence is rooted in raw technical strength or output.  Just a few of the wide variety of creative/collaborative/interpersonal strengths:

  • Some engineers are infinitely curious, and have a way of consistently sniffing a few steps ahead of the pack. They might seem to be playing around with something pointless, and you want to scold them; then they save your ass from total catastrophe. You learn to value their playing around.
  • Some engineers solve problems socially, by making friends and trading tips and fixes and favors in the industry. Don’t underestimate social debugging, it’s often the quickest path to the right answer.Image result for influence meme
  • Some are dazzlingly lazy and blow your mind with their elegant shortcuts and corners correctly cut.
  • Some are recruiting magnets, and it’s worth paying their salary just for all the people who want to work with them again.
  • Some are skilled at driving consensus among stakeholders.
  • Some are killer explainers and educators and storytellers.
  • Some are the senior engineer everyone silently wants to grow up to be.
  • Some can tell such an inspiring story of tomorrow that everyone will run off to make it so.
  • Some teach by turning code reviews into a pedagogical art form.
  • Some make everyone around them somehow more productive and effective. Some create relentless forward momentum. Some are good at saying no.

And there are a few special wells of power that bear calling out as such.

  • Engineers who have been managers are worth their weight in gold.  They can translate business goals for junior engineers in their native language with impeccable credibility (something managers never really have, esp in junior engineers’ eyes.). They make strong tech leads, they can carve up projects into components that challenge but do not overwhelm each contributor while hitting deadlines.
  • Some engineers are a royal pain in the ass because they questionImage result for engineer software meme individual contributor and challenge every system and hierarchy. But these are sharp, powerful rocks that can polish great teams. Though they do require a strong manager, to channel t
    heir energy towards productive dialogue and improvement and keep them from pissing off the whole team.
  • And let’s not forget engineers who are on call. If you have a healthy on call culture,your ownership over production creates a deep, deep well of power and moral authority — to make demands, drive change, to prioritize. On call should not be a shit salad served up to those who can’t refuse, it should be a badge of honor and seriousness shouldered by every engineer who ships code. (And it should not be miserable or regularly life-impacting.)

… I could go on all day. Engineering is such a powerful role and skill set. It’s definitely worth unpacking where your own influence comes from, and understanding how others perceive your strengths.

Most forms of power boil down to “influence, wielded”.

But just banging out code is not enough. You may have credibility, but having it is not the same as using it. To transform influence into power you have to use it.  And the way you use it is by communicating.

What’s locked up in your head has no impact on the rest of us.  You have to get it out.

You can do this in lots of ways: by writing, in 1x1s, conversations with small groups, openly recruiting allies, convincing someone with explicit authority, broadcasting inImage result for engineer software meme individual contributorpublic, etc.

Because engineering is a creative activity, authoritarian power is actually quite brittle and damaging. The only sustainable forms of power are so-called “soft powers” like influencing and inspiring, which is why good managers use their soft power freely and hard power sparingly/with great reluctance. If your leadership invokes authority on the regular, that’s an antipattern.[2]

If you don’t speak up, you don’t have the right to sit and fume over your lack of influence. And speaking up does mean being vulnerable — and sometimes wrong — in front of other people.

This is not a zero-sum game.

Most of you have far more latent power than you realize or are used to wielding, because you don’t feel powerful or don’t recognize what you do in those terms.

Managers may have hard power and authority, but the real meaty decisions about technical delivery and excellence are more properly made by the engineers closest to them. These belong properly to the doers, in large part because they are the ones who have to support the consequences of these decisions.Image result for engineer software meme individual contributor

Power tends to flow towards managers because they are privy to more information. That makes it important to hire managers who are aware of this and lean against it to push power back to others.

In the same way that submissives have ultimate power in healthy BDSM relationships, engineers actually have the ultimate power in healthy teams. You have the ultimate veto: you can refuse to create.  Demand is high for your skills.  You can usually afford to look for better conditions. Many of you probably should.

And when technical and managerial priorities collide, who wins? Ideally you work together to find the best solution for the business and the people. The teams that feel 🔥on fire🔥 always have tight alignment between the two.

Pick your battles.

One final thought. You can have a lot of say in what gets built and how it gets built, if you cultivate your influence and spend it wisely. But you can’t have a say in everything. It doesn’t work that way.

Think of it like @mcfunley’s famous “innovation tokens”, but for attention and fucks given.
Image result for engineer software meme
The more you use your influence for good outcomes, the more you build up over time, yes … but it’s a precision tool, not background noise. Imagine someone trying to give you a massage by laying down on your whole back instead of pushing their elbow or hand into knots and trigger points. A too-broad target will diffuse your force and limit your potential impact.

Spend your attention tokens wisely.

And once you have influence, don’t forget to use it on behalf of others. Pay attention to those who aren’t being heard, and amplify their voices. Give your time, lend your patronage and credibility, and most of all teach the skills that have made you powerful to others who need them.

charity

P.S. I owe a huge debt to all the awesome senior engineers i’ve gotten to work with.  Mad love to you all.  <3
Image result for influence meme

  • [0] I successfully answered one (1) of these questions before running out of steam.  Later. 
  • [1] Sheepish confession: this is why I became a manager.
  • [2] It’s also a bad sign if they won’t grant any explicit authority to the people they hold responsible for outcomes. I’m talking about relatively healthy orgs here, not pathological ones where people (often women) are told they don’t need promotions or explicit authority, they should just use their “soft power” — esp when the hard forms of power aligned against with them. That’s setting you up for failure.
  • [3] Some people seem caught off guard by my use of “power” to signal anything other than explicit granted powers by the org. This doesn’t make any sense to me. I find it too depressing and disempowering to think of power as merely granted authority. It doesn’t map to how I experience the world, either. Individual clout is a thing that waxes and wanes and only exists in relation to others’. I’ve seen plenty of weak managers pushed around by strong personalities (which is terrible too).
On Engineers and Influence

An Engineer’s Bill of Rights (and Responsibilities)

Power has a way of flowing towards people managers over time, no matter how many times you repeat “management is not a promotion, it’s a career change.”

It’s natural, like water flowing downhill.  Managers are privy to performance reviews and other personal information that they need to do their jobs, and they tend to be more practiced communicators.  Managers facilitate a lot of decision-making and routing of people and data and things, and it’s very easy to slip into making the all decisions rather than empowering people to make them.  Sometimes you want to just hand out assignments and order everyone to do as told.  (er, just me??)

But if you let all the power drift over to the engineering managers, pretty soon it doesn’t look so great to be an engineer.  Now you have people becoming managers for all the wrong reasons, or everyone saying they want to be a manager, or engineers just tuning out and turning in their homework (or quitting).  We all want autonomy and impact, we all crave a seat at the table.  You need to work harder to save those seats for non-managers.

So, in the spirit of the enumerated rights and responsibilities of our musty Constitution, here are some of the commitments we make to our engineers at Honeycomb — and some of the expectations we have for managering and engineering roles.  Some of them mirror each other, and others are very different.

(Incidentally, I find it helpful to practice visualizing the org chart hierarchies upside down — placing managers below their teams as support structure rather than perched atop.)

 

izeng

Engineer’s Bill of Rights

  1. You should be free to go heads down and focus, and trust that your manager will tap you when you are needed (or would want to be included).
  2. We will invest in you as a leader, just like we invest in managers.  Everybody will have opportunities to develop their leadership and interpersonal skills.
  3. Technical decisions must remain the provenance of engineers, not managers.
  4. You deserve to know how well you are performing, and to hear it early and often if you aren’t meeting expectations.
  5. On call should not substantially impact your life, sleep, or health (other than carrying your devices around).  If it does, we will fix it.
  6. Your code reviews should be turned around in 24 hours or less, under ordinary circumstances.
  7. You should have a career path that challenges you and contributes to your personal life goals, with the coaching and support you need to get there.
  8. You should substantially choose your own work, in consultation with your manager and based on our business goals.  This is not a democracy, but you will have a voice in our planning process.
  9. You should be able to do your work whether in or out of the office. When you’re working remotely, your team will loop you in and have your back.

Engineer’s responsibilities

  • Make forward progress on your projects every week. Be transparent.
  • Make forward progress on your career every quarter.  Push your limits.
  • Build a relationship of trust and mutual vulnerability with your manager andcateng team, and invest in those relationships.
  • Know where you stand: how well are you performing, how quickly are you growing?
  • Develop your technical judgment and leadership skills.  Own and be accountable for engineering outcomes.  Ask for help when you need it, give help when asked.
  • Give feedback early and often, receive feedback gracefully.  Practice both saying no and hearing no.  Let people retract and try again if it doesn’t come out quite right.
  • Own your time and actively manage your calendar.  Spend your attention tokens mindfully.

Manager’s responsibilities

  • Recruit and hire and train your team.  Foster a sense of solidarity and “teaminess” as well as real emotional safety.
  • Care for every engineer on your team.  Support them in their career trajectory, personal goals, work/life balance, and inter- and intra-team dynamics.
  • Give feedback early and often. Receive feedback gracefully. Always say the hard things, but say them with love.
  • Move us relentlessly forward, watching out for overengineering and work that doeasshatsn’t contribute to our goals.  Ensure redundancy/coverage of critical areas.
  • Own the quarterly planning process for your team, be accountable for the goals you set.  Allocate resources by communicating priorities and recruiting eng leads.  Add focus or urgency where needed.
  • Own your time and attention. Be accessible. Actively manage your calendar.  Try not to make your emotions everyone else’s problems (but do lean on your own manager and your peers for support).
  • Make your own personal growth and self-care a priority. Model the values and traits we want our engineers to pattern themselves after.
  • Stay vulnerable.

I’d love to hear from anyone else who has a list like this.

 

asleepatwork

 

 

 

An Engineer’s Bill of Rights (and Responsibilities)

Post-mortem: feminist advice meltdown (March 2nd)

Okay!  As of today it’s been one week since I wrote some advice and the internet exploded in my face, so now it’s time to do what I always do: post mortem that shit.

This is going to be long.  I erred by making my first post too short, so I’m going to ship $(allthedetail) this time.  Duly warned.

ragrets

What happened?

Around 8 am on Friday, March 2nd, after pulling an all-nighter, I decided to pound out a quick blog post that has been on my todo list forever: the only advice I feel equipped to give on how to succeed in tech.

My advice, in brief, was this:

  1. as a junior engineer, tough it out.  work hard, learn everything, earn your stripes.
  2. stay technical.  don’t get sucked into an offramp unless you are god damn sure you want out for good.
  3. once you are senior, use your power to advocate for others and fuck that shit up.

Money, power, credibility.  This is the best way I know how to earn these things.  This is what worked for me and most of the senior technical women I know and admire.

First of all: I don’t think there should be anything controversial at all about this advice.  It’s good advice, if a bit bluntly put.  Pick your battles, show strategic impact, leverage your influence into power and use that power to fuck shit up in the manner of your choosing.

The fact is, we are far too chickenshit about telling young women straight up how to succeed at work.  We praise them for all kinds of dumb shit and second shift work and emotional labor that has little if any strategic impact to the bottom line, and wonder why they’re burned out and resentful.

We live in a fallen world.  I didn’t make it this way, I just want to help you level up to be a powerful destroyer being so you can make it better.

So I hit “publish”.

Around 9:30 am, Camille Fournier gave me a bunch of unsolicited criticism. Unfortunately, due to some sour personal history with Camille I was extremely not disposed to receive this from her.  I can be a resentful little shit: as soon as she told me to change it in certain ways,  it was the last fucking thing in the world I was going to do.

For a few hours, all the feedback was good. People liked my advice to stay technical (“god I wish someone had told me that 15 years ago”) and my pointing out the loophole that lets women advocate for each other without being penalized.

A few people nailed what I was trying to say even better than I did:

But by the end of the day I was receiving a steady stream of angry tweets from people I had never heard of, with objections that seemed puzzling and ridiculous to me.

They were acting as though the sum total of my advice had been ordering bullied and abused people to just shut up and tough it out.  Soon I was getting tweets accusing me of trashing all diversity work, trashing all women, only being out for myself and my own career, erasing sexual assault, being insensitive and destructive to people of color, and on and on.feamale

People were subtweeting me like crazy, or DM’ing me telling me how much they liked my piece but were afraid to say so in public. Others were harassing my engineering managers and people who follow me.

I have never received textual scrutiny of this type before, where every single word was turned over and macerated and peered at for evidence of traitorous views.  It sucks.  (And it’s pretty hypocritical, to say the least … some of these same women who were gleefully bashing me for clumsy words remain good friends with men who are actual known harassers and abusers of women.)

Lots of people wanted me to take the post down immediately, or publish a retraction or correction immediately. Some prominent feminists publicly chided me and refused to talk to me until I repented of my sins.  🙄

https://twitter.com/vaurorapub/status/970544808086331392

Let’s be clear. I have no problem admitting my errors and making amends. I do it all the fucking time. But I am disinclined to grovel before a howling mob.  It wasn’t even clear to me what I had done wrong, given all the contradictory noises.

So I decided to wait a week before responding, so I could talk to people and figure out what to take away from the mess.

(Also last week: traveled to multiple continents, flew a few dozen hours, wrote multiple talks, delivered presentations at various conferences and meetups, visited and pitched to potential customers, managed a handful of teams, fit 1x1s in between hops and time zones and you know just tried to do my fucking job while dealing with crazed nuts screaming abuse at me online.)

I had a couple of hard but helpful conversations with people like Alice Goldfuss and Courtney Nash, who took the time to walk me through ways that what I wrote may be misinterpreted or wrongly received. This feedback can mostly be bucketed into the following categories:

  • “Assume the reader knows nothing about you and considers you hostile until proven otherwise.” Well shit, I am not used to writing defensively.  I live my life in high trust, high transparency environments and prefer it that way.
  • Your advice doesn’t apply to $x.”  True!  I didn’t bracket it in layers of padding — “this is just what worked for me”, “may not apply to every situation” — because I thought that was freaking obvious.
  • It sounds like you are shit talking all diversity efforts.” No, but I was waving vaguely in the direction of some very cynical and tired feelings on the subject. I’m pretty over corporate diversity issues and pinkwashing that doesn’t expand opportunity or share power.
  • It sounds like you are shitting on all women.” Oof. This is the one that is really painful, because this is the one I have been working hard on for close to 20 years… and should have seen coming. I did intend to put some space between myself and women in tech, because I don’t exactly identify as a woman.. exactly.  I grew up fundamentalist and misogynist af, and have been working hard to recover from that ever since I left home at 15.
  • Maybe you shouldn’t give advice to women at all.” Courtney challenged me on whether I should speak to women, given my ambivalence wrt my own gender identification. Which is an interesting question that I have pondered a lot.

This was all desperately inevitable and predictable, however, and I made some unforced errors. So let’s talk about what I do and don’t regret about all this, and what I would or would not do differently.

witch

Regrets/No regrets

NO REGRETS: giving the advice. It’s good advice, it needed to be said. I’m tired of seeing women burn themselves out on shitty corporate diversity work that only diverts their energy from amassing real power and strategic impact.  Not sorry.

REGRETS: I was sloppy about waving in the direction of my gender issues. I intended to put some space between myself and “women’s issues”, because I don’t exactly identify as a woman, exactly, and I have always felt uncomfortable in women’s spaces. Given the historic devaluation of women’s spaces and issues, I should have been clearer. I am sorry.

NO REGRETS: I think it’s fine for me to give advice to women if they ask, which they do. After all I was raised as a woman, have always been read and treated as one, and assumed there was no other option for 30+ years. I get to speak.  Not sorry.

SEMI-REGRETS: I still can’t figure how anyone managed to project into my piece that I was slamming all diversity work. I said somewhat colorfully that a lot of the advice didn’t work for me and wasn’t my favorite thing to dwell on, in the same grouchy grumbly tone that I use when bitching about query planners and terraform variable interpolation. I don’t think this would have been a big deal if the frenzy hadn’t gotten whipped up, but if anyone genuinely felt hurt or dismissed by it, I can be sorry for that.

REGRETS: the impact it had on my poor engineering managers and other people who work with me. They are still being asked to denounce me or defend me and their decision to work with me. So deeply not ok. I am sorry — but that’s really on you, internet assholes.

BIGGEST REGRETS: any accidental cover given to misogynists. By far the most annoying thing about the brouhaha has been when men with toxic views compliment me because they think I’m agreeing with them.  I am NOT, so get off me.  Sorry not sorry.

SADDEST REGRETS: my plummeting opinion of the feminist internet trash mob. I am a feminist and damn proud of it, but I am also disgusted by the hyperperformative boundary policing of certain self-proclaimed “tech feminists”. If your great joy in life is roving the interwebs looking for any toes pressing a line so you can rapturously castigate them and shun them until they have licked your boots and begged for forgiveness … if you love performing elaborate outrage rituals and whipping up a frenzy of whispers or a witch hunt… then:

laralittleFuck. The Fuck. Off. You are an embarrassment. This is about your ego, and your manufactured grievance machines are Not Helping.

I honestly thought these feminist pile-on mobs were a right-wing fantasy, and I’m sad that I was wrong. I’m also pretty sad about all the folks who know me and have every reason to know better.  In my world you check in with your friends before leaping to judgment, and you help teach each other when you’re being stupid. A pretty dismal number of people I would have called friends just leapt excitedly into the fray passing judgment.

So now I know more about who my friends are.

richer

In conclusion

Why even stick my neck out? I guessed something might go wrong, I just didn’t know what. So why?

Because I want to help, dammit. The farther I get in my career the more time I spend pondering how to bring others along with me, how to open the gates a little wider.

I’ve gotten to do a few things. I have tried to create an equitable, respectful working environment where everyone can do their best work, with managers who are passionate about diversity and strong where I am weak.

But … I have felt very often alienated by the messaging and attempts to help women.  I can’t be the only one who responds more to a strategic message than an empathetic one, who feels condescended to and patronized by the mainstream corporate efforts.

I can’t be the only one who feels simmering resentment every time I get held up as a successful “woman in tech” (the world’s worst participation trophy). I don’t want a fucking consolation prize. I want to sweep the competition, I want to change the world. I can’t be the only one who hungers for power, money and credibility.

I know I’m not, actually. I know because they are telling me. The response has been at least 100-1 positive in private — from junior women especially — thanking me for being brutally honest and treating them like adults, like equals.  (I’ve been told there are armies of women who feel dreadfully hurt but too afraid to say so.  Pity if true, as they say.)

There has always been tension between the people who see the world as it is and fight to succeed in it, and the people who opt out and refuse to participate because it’s compromised.  The world needs us both.  So shut the fuck up and let the kids pick for themselves.

And maybe stop persecuting the people who stand with you.

charity.

P.S. check out Jen Andre’s eloquent restatement of it all.  it’s so great.

nobodies

Charity todo items

  • If I ever again write anything about women or diversity, have someone I trust proof before publishing
  • Remember how much of my audience doesn’t know shit about me, and won’t or can’t assume the best of my intentions
  • Wrap statements in exception handlers about this being my experience blah blah
  • Try not to let people get under my skin and spark a personal reaction
  • Derive somewhat less pleasure from smacking down assholes on the internet, even when they deserve it. [ASPIRATIONAL] [WONTFIX]
Post-mortem: feminist advice meltdown (March 2nd)

Money, power and credibility

I don’t really do “women stuff”.  I don’t really identify with any gender and I find a lot of the advice to be condescending and overly delicate, and it’s just a really boring thing to think and talk about. For me.IMG_9847

But I’m feeling guilty after turning down a bunch of requests to do shit for International Women’s Day next week.  So I’m gonna do a thing I’ve been avoiding doing for years, and write down my (deeply problematic but practical) advice.

  1. Toughen up.  For your first 10 years or 3 jobs in the industry, you’re a junior contributor.  You need them way more than they need you, so suck it up.  Try not to dwell on the bullshit.  Work hard and level up and always angle for more money and power when you can.
  2. Stay technical.  There are a thousand paved ramps out of engineering roles and only a few hard paths back in.  Technical excellence is currency in this industry, even more so if your credibility is gonna get challenged again and again.  So don’t stop engineering til you’re great at it.
  3. Use your power for good.  Once you become a senior contributor — and i’m not talking bullshit titles but real seniority, when people are coming to you for help far more than you go to them — then you can afford to get sensitive.  …On behalf of others.  The research convincingly shows that women get punished for advocating for themselves, but not for advocating for others.  It’s a sweet loophole, use it.

If you feel like table flipping out of tech, just remember the rest of the world is at LEAST as sexist as tech is, but without the money and power and ridiculous life-coddling.  Where exactly do you think you’re going to go?

Don’t quit tech: quit your job.  There are LOTS of tolerable-to-great companies out there.  If you stay and suffer, you’re just rewarding the shitholes with your presence.  Don’t reward the shitholes any more than you can help it.

Learn shit, save your money, amass great power.  Then use it to fuck shit up.

IMG_9885

Money, power and credibility

Company Values: A Resentful Odyssey

I have a very cynical reaction to the word “values”, especially in the context of corporate heartentities. At best it’s a disingenuous marketing campaign, usually it’s more like a red blaring light shining on their degenerate hypocrisies and weakest aspirations.

But we’ve been doing a lot of hiring. And one day I realized that most of our top futurehoneycandidates were asking the same two questions: 1) how do technical decisions get made, and 2) what are our company values?

After a few rounds of stuttering and sounding like an idiot, I decided it was time to mayyyybe stop sounding like an idiot and come up with an answer.

But it’s hard to do something you don’t believe in, let alone something as cheesy and heart-on-your-sleeve as write a company values statement. So first I had to talk myself into believing it was worth doing. Which went something like this:

  1. Candidates I respect seem to think this thing matters, therefore it must matter to me too.
  2. … Fuck.
  3. Well, some are worse than others.  The ones I feel cynical about are the worst.  (Facebook’s were a running joke because nobody believed them)
  4. I didn’t hate Linden Lab’s values .. until we stopped believing in them.  Hmm, so what was valuable about them?
  5. Well, people used them to help resolve conflicts and make decisions.  Nice.
  6. Ok, what else do I hate? Values that are overly broad or include their opposite (“we work hard AND play harder”, “we’re empathetic BUT tough-minded”), are overly generic or too obvious (“no assholes” — duh), are unmemorable laundry lists, or too angelic and earnest (this list is not going to get you laid, capitalist scum).
  7. Ok. So. Our values should be particular (they should not apply just as well to any other company), they should help make decisions and resolve conflict (if they aren’t useful/if we don’t use them then what’s the point), they should be pithy and a bit snarky (an aesthetic choice, just a spoonful of bile to help the medicine go down).heart

I started jotting down values fodder on my phone, while walking back and forthbee-thinking between home and work, which is how I do all my writing these days.  I spent a couple weeks spewing notes out.  It was a mess.

Then I sat down with Ginsu, my wizard of a COO, because I knew[*] he had the unique power to sift through my ramblings and craft a pithy message from vast effluvia.  After bouncing it around a bit and soliciting everyone’s feedback, we were left with this list, which we quietly back-posted.

What I love about the list that it is specific, actionable, and truly echoes things we say every day to each other (“Everything is an experiment”, “Fast and mostly-right is better than slow and perfect”, and “We hire adults”), as well as bringing bits of our heritage (“Feedback is a gift”is lifted from Facebook; “Do it with style” comes from Linden Lab).bee-debuggy

I like that I overhear people repeating the phrases to each other as they do their work and argue and urge each other on. I even love that there are huge, known flaws with it (“do it with style” notoriously does not scale) because that reminds me this is a living document, that we have committed to its care and feeding and regular revisioning.

I even love that you may read it and think, “This place is not for me.” If a place is for everyone, it is not for anyone in particular. I am okay with being a particular place, for particular people at a particular time in our lives. Specificity elicits passion in a way that generic never can.

Nothing lasts forever. As we close this round of funding and the end of the beginning chapter of this company, it’s nice to take a breath, pause, and put a stamp on it.
heart
This is what works for us now. It won’t work for us forever, and that’s okay. We are who we are, and when we change, our values will too.

We hire adults. We got this.

charity

[*] he made me, obviously

Company Values: A Resentful Odyssey

The Engineer/Manager Pendulum

Lately I’ve been doing some career counseling for people off Twitter (long story). The central drama for many people goes something like this:

“I’m a senior engineer, but I’m thinking about being a manager. I really like engineering, but I feel like I’m just solving the same problems over and over and it seems like the real problems are people problems. I have to be a manager to get promoted. I hope it isn’t terrible, once I make the switch. I hear it’s terrible.”

I’ve been meaning to write this post for a while. There’s a lot but let’s start with: Fuck the whole idea that only managers get career progression. And fuckkkk the idea you have to choose a “lane” and grow old there.  I completely reject this kind of slotting.

“Your advice is bad and you should feel bad”:

The best frontline eng managers in the world are the ones that are never more than 2-3 years removed from hands-on work, full time down in the trenches. The best individual contributors are the ones who have done time in management.

unicorns-are-jerksAnd the best technical leaders in the world are often the ones who do both. Back and forth.  Like a pendulum.

I’ve done this a few times myself now; start out as an early or first infra engineering hire, build the stack, then build the team, then manage the team, then … leave and start it all over again. I get antsy, I get restless. I start to feel like I know what I’m doing (… a telltale sign something’s wrong).

It’s a good cycle for people who like early stage companies, or have ADD. But I don’t see people talking about it as a career path. So I’m here to advocate for it, as an intentional and awesome way of life.

(h/t to @sarahmei who was tweetstorming this up at the EXACT SAME TIME as i was writing this.  Yes Virginia, internet feminists ARE linked by a mystical hive brain.)

On being a manager (of technical projects)

Promoting managers from within means you get those razor sharp skills from the people who just built the thing. That gives them credibility, while they struggle with their newly achieved incompetence in a different role.

helpfulcat

That’s one of the only ways you can achieve the temporary glory of a hybrid manager+tech lead. This is an unstable combination, because your engineering skills and context-sharpness are decaying the longer you do it.

You can only really improve at one of these things at a time: engineering or management.  And if you’re a manager, your job is to get better at management.  Don’t try to cling to your former glory.

Management is highly interruptive, and great engineering — where you’re learning things — requires blocking out interruptions. You can’t do these two opposite things at once.  As a manager, it is your job to be available for your team, to be interrupted. It is your job to choose to hand off the challenging assignments, so that your engineers can get better at engineering.

On being a tech lead (of people):

Conversely: the best tech leads in the world are always the ones who done time in management. This is not because they’re always the best programmers or debuggers; it’s because they know how to get shit done, which means they know how to communicate and manage other people.

A tech lead is a manager … but their first priority is achieving the task at hand, not grooming and minding the humans who work on it.

They still need the full manager toolset.  They’ll need to know how to rally people and teams and motivate them, or how to triage and restart a stalled project that everybody dreads. They still need to connect the dots between business objectives and technical objectives, and break down big objectives rainbow-swooshinto components. They need to be able to size up a junior engineer’s ability and craft a meaningful assignment, one that pushes their boundaries without crushing them … then do the same for another twenty contributors. This is management work, from the slightly shifted perspective of “Get Thing X Done” not “care for these people”.

So these tech leads usually spend more time in meetings than building things, and they will bitch about it but do it anyway, because writing code is not the best use of their time.  Tech is the easy part, herding humans is the harder part.

Senior engineers who have both these toolsets are the kind of tech leads you can build an org around, or a company around. They get shit done. And they are rare.

Almost all of them have spent considerable time in management.

The Pendulum

We don’t talk about this nearly enough: the immense breadth and strength that accrues to engineers who make a practice of going back and forth.

Being an IC is like reverse-engineering how a company works with very little Rainbow_dash_12_by_xpesifeindx-d5giyirinformation. A lot of things seem ridiculous, or pointless or inefficient from the perspective of a leaf node. .

Being a manager teaches you how the business works.  It also teaches you how people work. You will learn to have uncomfortable conversations. You will learn how to still get good work out of people who are irritated, or resentful, or who hate your guts.  You will learn how to resolve conflicts, dear god will you ever learn to resolve conflicts.  (Actually you’ll learn to YEARN for conflicts because straightforward conflict is usually better than all the other options.) You’ll go home exhausted every day and unable to articulate anything you actually did.  But you did stuff.rainbow-clouds

You’ll miss the dopamine hit of fixing something or solving something.  You’ll miss it desperately.

One last thing about management. There’s a myth that makes it really hard for people to stop managing, even when it makes them and everyone around them miserable.  And that’s the idea that management is a promotion.

Management is NOT a promotion.

Seriously, fuck that so hard. It is SUCH an insidious myth, and it leads to so many people managing even though they hate managing and have no business managing, and also starves the senior eng pool of the great mentors and elder wizards we need.

Management is not a promotion, management is a change of profession. And you will be bad at it for a long time after you start doing it.  If you don’t think you’re bad at it, you aren’t doing your job.

Managing because it feeds your ego is a terrific way to be sure that your engineers get to report to someone miserable and resentful, someone who should really be writing code

charity-gofmt
my feelings on having to only manager OR engineer for the rest of my life

or finding something else that brings them joy.

 

There’s nothing worse than reporting to someone forced into managing.  Please don’t be one of the reasons people burn out hard on tech.

It isn’t a promotion, so you don’t have any status to give up. Do it as long as it makes you happy, and the people around you happy. Then stop. Go back to building things. Wait til you get that itch again.

Then do it all over again. <3

The Engineer/Manager Pendulum

DevOps vs SRE: delayed coverage of the dumbest war

Last week was the West Coast Velocity conference.  I had a terrific time — I think it’s the
best Velocity I’ve been to yet.  I also slipped in quite late, the evening before last, to catch Gareth’s session on DevOps vs SRE.

had to catch it, because Gareth Rushgrove (of DevOps Weekly glory) was taunting @lusis and me about it on the Internet.

rainbowdropletAnd it was worth it!   Holy crap, this was such a fun barnburner of a talk, with Gareth schizophrenically arguing both for and against the key premise of the talk, which was about “Google Infrastructure for Everyone Else (GIFEE)” and whether SRE is a) the highest, noblest goal that we should all aspire towards, or b) mostly irrelevant to anyone outside the Google confines.

Which Gareth won?  Check out the slides and judge for yourself.  🙃

 

At some point in his talk, though, Gareth tossed out something like “Charity probably already has a blog post on this drafted up somewhere.”  And I suddenly remembered “Fuck!  I DO!”  it’s been sitting in my Drafts for months god dammit.

So this is actually a thing I dashed off back in April, after CraftConf.  Somebody asked me for my opinion on the internet — always a dangerous proposition — and I went off on a bit of a rant about the differences and similarities between DevOps and SRE, as philosophies and practices.
colorful-rainbow

Time passed and I forgot about it, and then decided it was too stale.  I mean who really wants to read a rehash of someone’s tweetstorm from two months ago?

Well Gareth, apparently.

Anyway: enjoy.


SRE vs DevOps: TWO PHILOSOPHIES ENTER, BOTH ARE PHENOMENALLY SUCCESSFUL AND MUTUALLY DUBIOUS OF ONE ANOTHER


 

So in case you haven’t noticed, Google recently published a book about Site Reliability Engineering: How Google Runs Production Systems.  It contains some really terrific wisdom on how to scale both systems and orgs.  It contains chapters written by dear friends of mine.  It’s a great book, and you should buy it and read it!

Rainbow-Umbrella-Z-5_5It also has some really fucking obnoxious blurbs.  Things like about how “ONLY GOOGLE COULD HAVE DONE THIS”, and an whiff of snobbery throughout the book as though they actually believe this (which is far worse if true).

You can’t really blame the poor blurb’ers, but you can certainly look askance at a massive systems engineering org when it seems as though they’ve never heard of DevOps, or considered how it relates to SRE practices, and may even be completely unaware of what the rest of the industry has been up to for the past 10-plus years.  It’s just a little weird.

So here, for the record, is what I said about it.

 

Google is a great company with lots of terrific engineers, but you can only say they are THE

sre
The Google SRE Bible

BEST at what they do if you’re defining what they do tautologically, i.e. “they are the best at making Google run.”  Etsyans are THE BEST at running Etsy, Chefs are THE BEST at building Chef, because … that’s what they do with their lives.

Context is everything here.  People who are THE BEST at Googling often flail and flame out in early startups, and vice versa.  People who are THE BEST at early-stage startup engineering are rarely as happy or impactful at large, lumbering, more bureaucratic companies like Google.  People who can operate equally well and be equally happy at startups and behemoths are fairly rare.

And large companies tend to get snobby and forget this.  They stop hiring for uniquerainbow-swirl strengths and start hiring for lack of weaknesses or “Excellence in Whiteboard Coding Techniques,” and congratulate themselves alot about being The Best.  This becomes harmful when it translates into to less innovation, abysmal diversity numbers, and a slow but inexorable drift into dinosaurdom.

Everybody thinks their problems are hard, but to a seasoned engineer, most startup problems are not technically all that hard.  They’re tedious, and they are infinite, but anyone can figure this shit out.  The hard stuff is the rest of it: feverish pace, the need to reevaluate and reprioritize and reorient constantly, the total responsibility, the terror and uncertainty of trying to find product/market fit and perform ten jobs at once and personally deliver to your promises to your customers.

rainbow-cloud-dropletAt a large company, most of the hardest problems are bureaucratic.  You have to come to terms with being a very tiny cog in a very large wheel where the org has a huge vested interest in literally making you as replicable and replaceable as possible.  The pace is excruciatingly slow if you’re used to a startup.  The autonony is … well, did I mention the politics?  If you want autonomy, you have to master the politics.

 

Everyone.  Operational excellence is everyone’s job.  Dude, if you have a candidate come in and they’re a jerk to your office manager or your cleaning person, don’t fucking hire that person because having jerks on  your team is an operational risk (not to mention, you know, like moral issues and stuff).

But the more engineering-focused your role is, the more direct your impact will be on operational outcomes.

As a software engineer, developing strong ops chops makes you powerful.  It makes you better at debugging and instrumentation, building resiliency and observability into your own systems and interdependent systems, and building systems that other people can come along and understand and maintain long after you’re gone.rainbow-shade

As an operations engineer, those skills are already your bread and butter.  You can increase your power in other ways, like by leveling up at software engineering skills like test coverage and automation, or DBA stuff like query optimization and storage engine internals, or by helping the other teams around you level up on their skills (communication and persuasion are chronically underrecognized as core operations engineering skills).

This doesn’t mean that everyone can or should be able to do everything.  (I can’t even SAYrainbow-dot-ball the words “full stack engineer” without rolling my eyes.)  Generalists are awesome!  But past a certain inflection point, specialization is the only way an org can scale.

It’s the only way you make room for those engineering archetypes who only want to dive deep, or who really really love refactoring, or who will save the world then disappear for weeks.  Those engineers can be incredibly valuable as part of a team … but they are most valuable in a large org where you have enough generalists to keep the oars rowing along in the meantime.

So, back to Google.  They’ve done, ahem, rather  well for themselves.  Made shitbuckets of money, pushed the boundaries of tech, service hardly ever goes down.  They have operational demands that most of us never have seen and never will, and their engineers are definitely to be applauded for doing a lot of hard technical and cultural labor to get there.

So why did this SRE book ruffle a few feathers?

Mostly because it comes off a little tone deaf in places.  I’m not personally pissed off by
the google SRE book, actually, just a little bemused at how legitimately unaware they seem to be about … anything else that the industry has been doing over the past 10 years, in terms of cultural transformation, turning sysadmins into better engineers, sharing on-call rotations, developing processes around empathy and cross-functionality, engineering best practices, etc.

devops
DevOps for the rest of us

If you try and just apply Google SRE principles to your own org according to their prescriptive model, you’re gonna be in for a really, really bad time.

However, it happens that Jen Davis and Katherine Daniels just published a book called Effective DevOps, which covers a lot of the same ground with a much more varied and inclusive approach.  And one of the things they return to over and over again is the power of context, and how one-size-fits-all solutions simply don’t exist, just like unisex OSFA t-shirts are a dirty fucking lie.

Google insularity is … a thing.  On the one hand it’s great that they’re opening up a bit!rainbow-umbrella-clipart-1 On the other hand it’s a little bit like when somebody barges onto a mailing list and starts spouting without skimming any of the archives.  And don’t even get me started on what happens when you hire long, longterm ex-Googlers back into to the real world.

So, so many of us have had this experience of hiring ex-Googlers who automatically assume that the way Google does a thing is CORRECT, not just contextually appropriate.   Not just right for Google, but right for everyone, always.  Which is just obviously untrue.  But the reassimilation process can be quite long and exhausting when the Kool-Aid is so strong.

Because yeah, this is a conversation and a transformation that the industry has been having for a long time now.  Compared with the SRE manifesto, the DevOps philosophy is much more crowd-sourced, more flexible, and adaptable to organizations of all stages of developments, with all different requirements and key business differentiators, because it’s benefited from loud, mouthy contributors who aren’t all working in the same bubble all along.

And it’s like Google isn’t even aware this was happening, which is weird.

Orrrrrr, maybe I’m just a wee bit annoyed that I’ve been drawn into this position rainbow-dot-ballof having to defend “DevOps”, after many excellent years spent being grumpy about the word and the 10000010101 ways it is used and abused.

(Tell me again about your “DevOps Engineering Team”, I dare you.)

P.S. I highly encourage you to go read the epic hours-long rant by @matthiasr that kicked off the whole thing.  some of which I definitely endorse and some of which not, but I think we could go drink whiskey and yell about this for a week or two easy breezy  <3

Anyway what the fuck do I know, I’ve never worked in the Google lair, so maybe I am just under-equipped to grasp the true glory, majesty and superiority of their achievements over us all.

Or maybe they should go read Katherine and Jen’s book and interact with the “UnGoogled” once in a while.  ☺️

colorful-rainbow

 

DevOps vs SRE: delayed coverage of the dumbest war

WTF is operations? #serverless

I just got back from the very first ever @serverlessconf in NYC.  I have a soft spot for well-curated single-track conferences, and the organizers did an incredible job.  Major kudos to @iamstan and team for pulling together such a high-caliber mix of attendees as well as presenters.

I’m really honored that they asked me to speak.  And I had a lot of fun delivering my talk!  But in all honesty, I turned it down a few times — and then agreed, and then backed out, and then agreed again at the very last moment.  I just had this feeling like the attendees weren’t going to want to hear what I was gonna say, or like we weren’t gonna be speaking the same language.

Rainbow_dash_12_by_xpesifeindx-d5giyirWhich … turned out to be mmmmostly untrue.  To the organizers’ credit, when I expressed this concern to them, they vigorously argued that they wanted me to talk *because* they wanted a heavy dose of real talk in the mix along with all the airy fairy tales of magic and success.

 

So #serverless is the new cloud or whatever

Hi, I’m grouchy and I work with operations and data and backend stuff.  I spent 3.5 years helping Parse grow from a handful of apps to over a million.  Literally building serverless before it was cool TYVM.

So when I see kids saying “the future is serverless!” and “#NoOps!” I’m like okay, that’s cute.  I’ve lived the other side of this fairytale.  I’ve seen what happens when application developers think they don’t have to care about the skills associated with operations engineering.  When they forget that no matter how pretty the abstractions are, you’re still dealing with dusty old concepts like “persistent state” and “queries” and “unavailability” and so forth, or when they literally just think they can throw money at a service to make it go faster because that’s totally how services work.

I’m going to split this up into two posts.  I’ll write up a recap of my talk in a sec, but first let’s get some things straight.  Like words.  Like operations.

What is operations?

Let’s talk about what “operations” actually means, in the year 2016, assuming a reasonably high-functioning engineering environment.

Screen Shot 2016-05-30 at 4.29.09 PM

At a macro level, operational excellence is not a role, it’s an emergent property.  It is how you get shit done.

Operations is the sum of all of the skills, knowledge and values that your company has built up around the practice of shipping and maintaining quality systems and software.  It’s your implicit values as well as your explicit values, habits, tribal knowledge, reward systems.  Everybody from tech support to product people to CEO participates in your operational outcomes, even though some roles are obviously more specialized than others.

Saying you have an ops team who is solely responsible for reliability is about as silly as saying that “HR defines and owns our company culture!”  No.  Managers and HR professionals may have particular skills and responsibilities, but culture is an emergent property and everyone contributes (and it only takes a couple bad actors to spoil the bushel).

Thinking about operational quality in terms of “a thing some other team is responsible for” is just generally not associated with great outcomes.  It leads to software engineers who are less proficient or connected to their outcomes, ops teams who get burned out, and an overall lower quality of software and services that get shipped to customers.

These are the specialized skill sets that I associate with really good operations engineers.  Do these look optional to you?

Screen Shot 2016-05-30 at 4.41.21 PM

It depends on your mission, but usually these are not particularly optional.  If you have customers, you need to care about these things.  Whether you have a dedicated ops team or not.  And you need to care about the tax it imposes on your humans too, especially when it comes to the cognitive overhead of complex systems.

So this is my definition of operations.  It doesn’t have to be your definition.  But I think it is a valuable framework for helping us reason about shipping quality software and healthy teams.  Especially given the often invisible nature of operations labor when it’s done really well.  It’s so much easier to notice and reward shipping shiny features than “something didn’t break”.

The inglorious past

Don’t get me wrong — I understand why “operations” has fallen out of favor in a lot of crowds.  I get why Google came up with “SRE” to draw a line between what they needed and what the average “sysadmin” was doing 10+ years ago.

Ops culture has a number of well-known and well-documented pathologies: hero/martyr complexes, risk aversion, burnout, etc.  I understand why this is offputting and we need to fix it.

Also, historically speaking, ops has attracted a greater proportion of nontraditional oddballs who just love debugging and building things — fewer Stanford CS PhDs, more tinkerers and liberal arts majors and college dropouts (hi).  And so they got paid less, and had less engineering discipline, and burned themselves out doing too much ad hoc labor.Rainbow_Dash_3.png

But — this is no longer our overwhelming reality, and it is certainly not the reality we are hurtling towards.  Thanks to the SRE movement, and the parallel and even more powerful & diverse open source DevOps movement, operations engineers are … engineers.  Who specialize in infrastructure.  And there’s more value than ever in empathy and fluid skill sets, in engineers who are capable of moving between disciplines and translating between specialties.  This is where the “full-stack developer” buzzword comes from.  It’s annoying, but reflects a real craving for generalist skill sets.

The BOFH stereotype is dead.  Some of the most creative cultural and technical changes in the technical landscape are being driven by the teams most identified with operations and developer tooling.  The best software engineers I know are the ones who consistently value the impact and lifecycle of the code they ship, and value deployment and instrumentation and observability.  In other words, they rock at ops stuff.

The Glorious Future

And so I think it’s time to bring back “operations” as a term of pride.  As a thing that is valued, and rewarded.  As a thing that every single person in an org understands as being critical to success.  Every organization has unique operational needs, and figuring out what they are and delivering on them takes a lot of creativity and ingenuity on both the cultural and technical front.

“Operations” comes with baggage, no doubt.  But I just don’t think that distance and denial are aRainbow_Dash_in_flightn effective approach for making something better, let alone trash talking and devaluing the skill sets that you need to deliver quality services.

You don’t make operational outcomes magically better by renaming the team “DevOps” or “SRE” or anything else.  You make it better by naming it and claiming it for what it is, and helping everyone understand how their role relates to your operational objectives.

And now that I have written this blog post I can stop arguing with people who want to talk about “DevOps Engineers” and whether “#NoOps” is a thing and maybe I can even stop trolling them back about the nascent “#NoDevs” movement.  (Haha just kidding, that one is too much fun.)

Part 2: Operations in a #Serverless World

 

WTF is operations? #serverless

How to survive an acquisition

Last week Facebook announced that they will be shutting down Parse.

 

I have so, so many feelings.  But this isn’t about my feelings now.  This is about what I wish I had known when we got acquired.

When they told us we were getting acquired, it felt like a wrecking ball to the gut.  Nothing against Facebook, it just wasn’t what I signed up for.  Big company, bureaucracy, 3-4 hour daily commute, and goddammit Parse was my *baby*.  What the fuck was this big company going to do to my baby?

Here’s the first thing I wish I had known: this is normal.  Trauma is the norm.  Acquisitions are *always* massively disruptive and upsetting, even the good ones.

This was all compounded by the jarring disconnect between internal and external perceptions of the acquisition.  The surreality of being written up in the press and having everyone act like this was the greatest thing that ever happened to us, versus the shock and dismay and confusion that many of us were actually experiencing.

Here are a few more things I wish I had understood.

You don’t own your product anymore.  Your product is now “strategic alignment”.  Look at yourself in the mirror and repeat that five times every morning.

Your customers are not your customers anymore.  Your customer is now your corporate overlord.  You can resist this and try to serve your old customers first, but it will wear your engineering team out and eventually drive you mad.

Cultures will clash.  Yours will lose.  You *must* learn the tribal and hierarchal games of the new org if you want to succeed.  They don’t make sense to you, and you didn’t choose into them, but you must learn them anyway if you want to succeed.

Assume good intent.  Aggressively assume good intent.  If a big company buys up your tiny startup, lots of people are going to condescend to you and assume they know how to solve your own problems better than you do.  In retrospect, I realize that a lot of tiny acquired startups *don’t* know what the fuck they’re doing and so the behemoth just gets used to assuming that everyone is like that.  Just grit your teeth and take it.

And if you honestly can’t see yourself ever embracing the new parent company, whether for cultural or ethical or technical reasons, you should leave sooner rather than later.  (UNLESS it’s for the golden handcuffs in which case for god’s sake try to emotionally disengage and do not be a people manager.)

One of Facebook’s internal slogans is something like “always do what’s best for Facebook”.  I never gave a shit about Facebook’s mission; I cared about Parse’s.  I remain incredibly proud of everything we accomplished for Parse, MongoDB, and RocksDB.  If only our strategic alignments had, well, aligned.

 

How to survive an acquisition