startup

How to Throw A Company Offsite In A “Post-COVID” World

mipsytipsy5 Comments

Earlier this month we had our first Honeycomb all-hands offsite in three years … our first one since February of 2020, before the plague. It was wonderful and glorious and silly and energizing and so, so SO much fun. It was a potent reminder of the reality that no virtual activity can compare with the energy of being physically present with people you care about.

I was talking with Paul Biggar last month, telling him about all the things we were doing both to create a safe environment and to ease the pangs of re-entry for a bunch of people who haven’t done this in years. Paul observed that it seemed like most companies are either 1) not gathering at all or 2) barreling forward as though COVID didn’t exist, and that there aren’t many stories about groups assembling safely.

So I said I would write one, if we pulled it off. And now that it’s been long enough that we can confidently say nobody came down with COVID at our offsite, here it is.

Offsites: luxury or necessity?

Honeycomb has always been proud to be a distributed company, even before the plague hit. It’s part of our belief system that this is just a better way of doing business.

But being distributed doesn’t mean that in-person connection doesn’t matter. It matters even more when you are all remote most of the time.

Getting together in human meatspace is expensive, it’s annoying, it’s awkward and uncomfortable and inconvenient … and it is necessary. It’s not optional or a “nice-to-have,” it is a ✨critical ingredient✨ of the recipe that makes high-performing distributed teams work. Spending time together face to face is the yeast in our bread, the bitters in our Old Fashioneds.

By the end of 2022, it had become clear that we are stuck with COVID for the long-haul. It was long past time to gather and get to know each other in person. But if we were going to do it, we had to do our best to mitigate the risks and adapt to the world we live in now. What precautions to take? Where to even start?

When and where, and why??

COVID concerns were part of the planning from day one. There were several constraints where to hold the event, which we immediately dubbed “Swarm” (of course). 🙃

  • Moderate weather. We wanted to be able to let people congregate outdoors as much as possible, as an additional insurance policy for the extra-anxious.
  • “Centrally located” was the original goal, but most of the country is just too cold in February. We settled for a destination that everyone could fly to direct.
  • Legal safety. We needed to go to a state w/  here all of our employees felt safe, not targeted by various legal jurisdictions.
  • We did NOT care about glamorous locations. We knew we were going to spend our time at the hotel, focused on each other, not the locale.

In the end we chose Los Angeles — the the oh-so-glamorous LAX Airport Marriott, to be precise. 🌴🐠\\🍹

We originally scheduled the event for mid-January, to kick off the new year, but ended up witthlizadelaying five weeks into 2023 in case there was a wave of post-holiday COVID infections.

Preparing for the event

Our COVID safety plans are designed to maximize attendance, give people enough information to properly manage their own risk, and provide an extra-safe outdoor alternative for as many activities as possible.

We tapped Liz Fong-Jones to take point on COVID policy. (As a globe-trotting extrovert who goes to lots of events, she is our resident expert.) Having one czar in charge of policy turned out to be great; she didn’t have to ask permission from anyone or get consensus from a committee to update policies and make requests on the spot.

Leading into the event, our plans and preparations included:

  1. Adjacent outdoors space.
    We looked for a venue with extra capacity (plenty of space for social distancing and proper airflow) and a large outdoor area — covered, heated space for people to eat and socialize, even if it was rainy or cold.
  2. On site testing for everyone.
    Everyone was given a rapid nucleic test on checking in, and asked to take onesubsequently every 48 hours.

    • Testing was mandatory for everyone, including vendors, guests, and visiting dignitaries.
    • Positive first-line tests should confirmatory test with a PCR or second nucleic test and isolate until a negative comes back.
    • If anyone confirms positive, their instructions are to stay in their room until they test negative. Honeycomb pays for hotel and rebooking.
  3. Carbon dioxide monitors.
    Every conference room was supposed be set up with a mounted CO2 monitor where people can see it. (Carbon dioxide monitors are a good proxy measurement for air circulation and thus transmission risk.) We said, “if the number is above 650, masks will be required in the room”, but this didn’t quite play out (see “Retro” section).
  4. Social distancing.
    We had stickers for badges, color coded by the traffic light system. (This was for individuals to exercise control based on risk tolerance, not a universal rule):

    • Red — please keep your distance
    • Yellow — please ask before physical touch
    • Green — I’m comfortable with talking and physical touch
  5. Masks.
    Masks were available throughout the event, but not required. People were welcome to wear their own masks regardless, of course, and everyone kept one in their pocket, so they could put it on if anyone asked. (This happened a few times in my vicinity, and everybody cheerfully complied.)

    • We booked a shuttle to haul everyone into LA for dinner, and masks were required on the shuttle. People were also told to call an uber/lyft if they felt uncomfortable with the shuttle.
  6. Outdoor seating at restaurants.
    One of our most popular sessions was “small group dinners”, where we sent people out to LA restaurants in small cross-functional groups of 8 people (including at least one member of senior leadership per group). People could request an outdoor group.
  7. If a new variant or outbreak emerged…
    We wouldn’t be able to call the event off or reschedule without forfeiting the entire cost. Our contingency plan was to move the entire event outdoors to the pool area, if necessary.

Pacing ourselves: naps and snacks and breaks

We knew this trip was going to be intense and overwhelming for many if not most of us — even more so than most company offsites. Many people hadn’t traveled or spent time in large groups since the pandemic, all of us are used to working solo from home. Only a tenth of us (18 of 180) were working at Honeycomb in February of 2020, at our last offsite. So mental health and social overload were just as important to consider.

We told people, repeatedly:

Take care of yourselves. Do what you need to do to be fully present while you’re here, rather than here 100% of the time.

We scheduled only three sessions per day, plus lunch and dinner. We neither started very early or scheduled things very late. We left lots of padding in between sessions for snacks  and naps and breaks.

We set aside a “Recharge Room” with doors that closed, with nail polish, coloring books and markers, and board games. We set aside cubbies with signs marked “Introvert Corners — no talking please!”, and stocked them up with USB hubs and charging cables, so you could recharge your devices while recharging your soul (lol).

In retrospect

Over 86% of the company came (!!!) and we had a riotously good time — way, way more than I think any of us even expected or hoped to have. It was a memorable, sparkly reminder of the incomparable magic of what it’s like being together with people you care about.

We ran a survey afterwards. 50% of respondents said they felt safe, but their standards are low; 48% said they felt safe due to the testing, masking, and reminders; 8% said they felt safe because they spent as much time as possible outdoors. 10% (11 people, out of 109 responding) said they “did not feel very safe.”

However, I would note that 5 of those 11 people didn’t actually attend Swarm. And one person pointed out afterwards that you had to choose “I did not feel very safe” if you wanted to enter any feedback at all about safety, so it was possible if not probable that some people would have chosen another option if they could.

 

I think we did pretty well. However, things can always be better!

What worked well:

  1. Outdoor spaces for eating, temperate climate, extra-spacious rooms for circulatory purposes.
  2. The rapid-nucleic tests we used were made by Lucira. (Liz says: “The tests were far more reliable than we thought they’d be — the brand has a reputation for false positives but we saw none, and only 3 invalid tests out of several hundred performed.”)
  3. Testing. We were hyper-diligent about testing. We actually ran out of tests on Wednesday, despite provisioning two for each attendee and a bunch of extras, and had to emergency lift in more, which is a great sign.
  4. Taking a layered approach to COVID safety. Not relying on any single prevention method meant that if one didn’t work well, we still had a safety net.
  5. Having a knowledgeable COVID czar.
  6. The Recharge Room with nail polish, art supplies and games worked really well, although designating other areas as “Quiet Zones” was ineffective and unnecessary.

What we can improve on:

  1. The 30-60 minute breaks sprinkled throughout the day were well-intentioned, but ineffective. Everybody was fucking wiped by late afternoon. Next time, I would replace all those breaks with one solid “Nap Break” from 4-6pm.
    • I think it’s important to acknowledge that we were all going to be flattened no matter what though. We aren’t used to this! It is HARD to leave the party to take care of yourself!
  2. We should have a team of COVID safety marshals, in addition to the Code of Conduct team.
  3. Next time we will ask people to submit their COVID test results — even if it’s just dropping a pic in a slack channel. Trust but verify.
  4. The CO2 monitors were confusing. They were hard to find, and most of them got installed in the wrong place. Also, the 650 ppm rule would have essentially meant everyone had to wear a mask in any closed room, which was not the goal. Our error here was not in failing to stick to the 650 ppm rule, but in making a rule we couldn’t keep.
  5. Next year, we will be explicit about the fact that CO2 monitors are for informational purposes, so individuals can factor it into their own masking choices and/or decide whether to move outside.
  6. Next year we will also emphasize that it is okay to ask the people around you to mask up so you can participate.
    • If you aren’t comfortable asking those around you to put on a mask, you can DM a safety marshal who will ask on your behalf.
  7. One session in particular had external facilitators come in and roam around the (closed) rooms shouting stuff. They tested clear for COVID, but in retrospect I really wish we had asked them to mask up, as it generated a lot of anxiety. This was our biggest covid safety lapse, in my opinion.

Offsites are necessary. Human connection feeds your soul.

What I learned from this event is just how much we were all craving connection. I was a firm believer in how much in-person connection matters to start with, and it still blew me away just how special, how irreplaceably precious the experience was. There is no possible way we could have made the kind of connections, learned the same lessons, and formed the kind of bonds we did if we had tried to do this as a remote event. If anything, it just made me even more intent on creating an event safe enough that everyone can join.

On the last night we had a party, with karaoke and a dress theme: “Wear something you wouldn’t wear to work.” By that point, it enough intimacy had been established that people seemed to feel safe letting their inner freak flag fly a bit, and it was just fucking incredible getting a glimpse into everybody’s inner selves and stories.

It seemed like others felt the same way, too. In the anonymous post-event survey, 100% of attendees said it was valuable to them. Representative feedback:

“I didn’t know how much I needed this, and I can’t wait for next year.”

Reintroducing a practice of spending time together is not only possible, I feel like we were able to do it in ways that adapt to the realities of the world we live in today.

That matters. I know I’ve said this like five times now, but for a distributed company, gathering together in person isn’t a nice-to-have, it is an absolute necessity. After meeting all of my coworkers, I feel this more strongly than ever. 💜🐝

P.S. None of us got COVID in the wake of our offsite. A handful of us did, however, manage to catch a common cold. 🦠

How to Throw A Company Offsite In A “Post-COVID” World

Every Achievement Has A Denominator

mipsytipsy, , , 5 Comments

One of the classic failure modes of management is the empire-builder — the managers who measure their own status, rank or value by the number of teams and people “under” them.

Everyone knows you aren’t supposed to do this, but most of us secretly, sheepishly do it anyway to some extent. After all, it’s not untrue — the more teams and people that roll up to you, the wider your influence and the more impact you have on more people, by definition.

The other reason is, well, it’s what we’ve got. How else are we supposed to gauge our influence and impact, or our skill as a leader? We don’t really have any other language, metrics or metaphors readily available to us. 😖

Well… Here’s one:

✨Every achievement has a denominator✨

Organization size can be a liability

Let’s say you have 1,000 people in your org and you collectively achieve something remarkable. Good for you!

What if you achieved the same thing with 10,000 people, instead? What would that say about your leadership?

What if you achieved the same thing with 100 people?

Or even 10 people?

Lots of people take pride in their ability to manage large organizations. And with thousands of people in your org, you kinda better do something fucking great. But what if you instead took pride in your ability to deliver outsize results with a small denominator?

What if comp didn’t automatically bloat with the size of your org, but rather the impact of your work divided by the number of contributors — rewarding leaders for leaner teams, not larger ones?

Bigness itself is costly. There’s the cost of the engineers, managers, product and designers etc, of course. But the bigger it gets, the more coordination costs are incurred, which are the worst costs of all because they do not accrue to any user benefit — and often lead to lack of focus and product surface sprawl.

Constraints fuel creativity. Having “enough” engineers for a project is usually a terrible idea; you want to be constrained, you want to have to make hard decisions about where to spend your time and where to invest development cycles.

More often than not, scope is the enemy

As Ben Darfler wrote earlier this year about our approach to engineering levels at Honeycomb:

There are times when broad scope may be unavoidable, but at Honeycomb, we try to cultivate a healthy skepticism toward scope. More often than not, scope is the enemy. We would rather reward engineers who find clever ways to limit scope by decomposing problems in both time and size. We also want to reward engineers who work on the most important problems for the business, regardless of the size of the project. We don’t want to reward people for gaming out their work based on what will get them promoted.

The same is true for engineering managers, directors and VPs. We would rather reward them for getting things done with small, nimble teams, not for empire building and team sprawl. We want to reward them for working on the most important problems for the business, regardless of what size their teams are.

What was the denominator of the last big project you landed? Could you have done it with fewer people? How will you apply those learnings to the next big initiative?

Can we find more language and ways to talk about, or take pride in how efficiently we do big things? At the very least, perhaps we can start paying attention to the denominator of our achievements, and factor that into how we level and reward our leaders.

charity.

P.S. I did not invent this phrase, but I am unfortunately unable to credit the person I heard it from (a senior Googler). I simply think it’s brilliant, and so helpful.

Every Achievement Has A Denominator

The Hierarchy Is Bullshit (And Bad For Business)

mipsytipsy, , 16 Comments

My friend Molly has had an impressive career. She got a job as a software engineer after graduating from college, and after kicking ass for a year or so she was offered a promotion to management, which she accepted with relish. Molly was smart, driven, and fiercely ambitious, so she swiftly clambered up the ranks to hold director, VP, and other shiny leadership roles. It took two decades, an IPO and a vicious case of burnout before she allowed herself to admit how much she hated her work, and how desperately she envied (guess who??) the software engineers she worked alongside. Turns out, all she ever really wanted to do was write code every day. And now, to her dismay, it felt too late.

Why did it take Molly so long to realize what made her happy? I personally blame the fucking hierarchy.

The Hierarchy Lie

The “Big Lie” of hierarchy is that your organizational structure is a vertical tree from the CEO on down, where higher up is always better.

Of course any new grad is going to feel that way, on the heels of 15-20 years spent going through school year by year, grade by grade, measuring success via good grades and teacher approval. The early years of professional life are a similar blend of hard work, leveling up and basic skills acquisition. (They got Molly hopped on the leveling treadmill before she even had a chance to become a real adult, in other words. 😍)

But by the time you are fully baked as a senior contributor, maybe 7-8 years in, your relationship to levels and ladders should undergo a dramatic shift. At some point you have to learn to tune in to your own inner compass. What draws you in to your work? What fuels your growth and success?

Being an adult means not measuring yourself entirely on other people’s definition of success. Personal growth might come in the guise of a big promotion, but it also might look like a new job, a different role, a swing to management or back, becoming well-known as a subject matter expert, mentoring others, running an affinity group, picking up new skill sets, starting a company, trying your hand at consulting, speaking at conferences, taking a sabbatical, having a family, working part time, etc. No one gets to define that but you.

You have a thirty- or forty-year adult life and career in front of you. What the hell are you going to do with all that time and space??

Your career is not one mad sprint to the finish line

Literally nobody’s career looks like a straight line, going up, up up and to the right, from intern to CEO (to a coffin).

One of the most exhausting things about working at Facebook was the way engineering levels feltLiterally no one's career, ever. like a hamster wheel, where every single quarter you were expected to go go go go go, do more do more, scrape up ever more of your mortal soul to pour in more than you could last quarter — and the quarter before that, and before that, in ever-escalating intensity.

It was fucking exhausting, yo. Life does not work that way. Shit gets hilly.

The strategy for a fulfilling, lifelong career in tech is not to up the ante every interval. Nor is it to amass more and more power over others until you explode. Instead:

  1. Train yourself to love the feeling of constantly learning and pushing your boundaries. Feeling comfortable is the system blinking orange, and it should make you uneasy.
  2. Follow your nose into work that lights you up in the morning, work you can’t stop thinking about. If you’re bored, do something else.
  3. Say yes to opportunities!! Intensity is nothing to be afraid of. Instead of trying to cap your speed or your growth, learn to alternate it with recovery periods.
  4. If you aren’t sure what to do, make the choice that preserves or expands future optionality. Remember: Most startups fail. Will you be okay with your choices if (& when) this one does too?

Why do people climb the ladder? “Because it’s there.” And when they don’t have any other animating goals, the ladder fills a vacuum.

But if you never make the leap from externally-motivated to intrinsically-motivated, this will eventually becomes a serious risk factor for your career. Without an inner compass (and a renewable source of joy), you will struggle to locate and connect with the work that gives your life meaning. You will risk burnout, apathy and a serious lack of fucks given..

The times I have come closest to burnout or flaming out have never been when I was working the hardest, but when I cared the least. Or when I felt the least needed.📈📉💔

A disturbing number of companies would rather feel in control than unclench and perform better

But hey! Lack of inner drive isn’t the ONLY thing that drives people to climb the ladder. Plenty of companies fuck this up too, all on their lonesome. Let’s talk about more of the ways that companies mess up the workplace! Like by disempowering the people doing the work and giving all the power to managers, thereby forcing anyone who wants a say in their own job become one.

The way we talk about work is riddled with hierarchical, authoritarian phrases: “She was my superior”, “My boss made me do it”, “I got promoted into management”, and so on.

There are plenty of industries where line workers are still disempowered cogs and power structures are hierarchical and absolute (like flipping burgers at McDonalds, or factory line work). There are even software companies still trying to make it work in command-and-control mode, to whom engineers are interchangeable monkeys that ship story points and close JIRA tasks.

But if there’s one thing we know, it’s that for industries that are fueled by creativity and innovation, command-and-control leadership is poison. It stifles innovation, it saps initiative, it siphons away creativity and motivation and caring.

Studies also show that the more visible someone’s power is, the less likely anyone is to give them honest feedback.[2]

Companies that don’t learn this lesson are unlikely to win over the long run. Engineering is a deeply creative occupation, and authoritarian environments are toxic for creativity and people’s willingness to share information.

Hierarchy is just a data structure

The basic function of a hierarchy is to help us make sense of the world, simplify information, and make decisions. Hierarchy lets us break down enormous projects — like “let’s build a rocket!”, or “let’s invade the moon!” — into millions of bite size decisions and tasks, and this is how progress gets made.

A certain amount of authority is invested into the hierarchy model. If you are responsible for delivering a unit of work, the company needs to make sure you have enough resources and decision-making ability to do so. This is what we think of as the formal power structure [1], and there is nothing wrong with that. It’s what makes the system work.

The problem starts when we stop thinking of hierarchy as a neutral data structure — a utilitarian device for organizing groups and making decisions — and start projecting all kinds of social status and dominance onto it.

A sensitivity to social dominance is wired deep, deep into our little monkey brains. It’s what tells us we deserve more power, leverage, pride, influence, and autonomy — and simply have more value — than those below us. It’s what tells us those above us are better, stronger and more deserving than we are, and that we owe them our respect and deference.

It also tells us “if you lose status, YOU MIGHT DIE” 😱😱😱 which is why we may react to a perceived loss of status with a sting that seems astonishingly extreme and overwrought, even to ourselves, yet somehow impossible to shrug off.

hierarchies tend to get mixed up with social dominance

In general, it is better to pursue roles and growth based on the affirmative (what it is you want to learn, grow or do more of) than the negative (what you want to avoid, evade or stop doing). Your motivation systems don’t kick in to gear when you are feeling “lack of pain” — the system doesn’t work that way. They kick in when you get interested.

And if you are sick of doing something or being treated a certain way, chances are everyone else will hate it, too. Who wants to work at a company where all the shit rolls downhill?

Hierarchies have stuck around for one very good reason: because they work. Hierarchies are simple, intuitive, and allow large numbers to collaborate with low cognitive overhead. Unfortunately, most hierarchies become entwined with status and dominance markers, which can bring enormous downsides. At their worst, they can suck the literal life out of work, reducing us all to glum little cogs obeying orders.

We aren’t getting rid of hierarchy anytime soon. But we can use culture and ritual to gently untangle them from dominance, and we can choose to interpret formal power as a service function instead of a dictatorship. This frees people up to choose their work based on what makes them feel fulfilled, instead of their perceived status. (Also helpful? Flatter pay bands. 😛)

Good managers do not dictate and demand, they nurture, develop, and inspire. The most important roles in the company aren’t held by managers; they are all the little leaf nodes  busily building the product, supporting users, identifying markets, writing copy, etc. The people doing the work are why we exist as a company; all the rest is, with considerable due respect, overhead.

How to drain your hierarchy of social dominance

When it comes to hierarchy and team structure, there are the functional, organizational aspects (mostly good) and the social dominance parts (mostly bad). With that in mind, there are plenty of smaller things we can do as a team to remind people that we are equal colleagues, simply with different roles.

  • Be conscious of the language you use. Does it reinforce dominance and hierarchy? (Step one: stop calling management “a promotion”🥰)
  • De-emphasize trappings of power. The more you refer to someone’s formal power, the less likely anyone is to give them critical feedback or question them.
  • Push back against common but unhelpful practices, like “a manager should always make more money than the people who report to them.” Really? Why??
  • Are there opportunities for career advancement as an IC, or only as a manager? Everyone should have the ability to advance in their career.
  • Do your own dishes, everyone.
  • Practice visualizing the org chart upside down, where managers and execs support their teams from below rather than topping them from above. (I was going to write a whole post about this, then discovered other people have been doing that for the past decade. 🤣)

And then there is the big(ger) thing we can (and must!) do, in order to 1) make people go into management for the right reasons, 2) help senior IC roles remain attractive to highly skilled creative and technical contributors, and 3) encourage everybody to make career decisions based on curiosity, growth, and what’s best for the business, instead of turf and power grabs. Which is:

Practice transparency, from top to bottom

Share authority, decision-making and power

Technical contributors own technical decisions

Most people who go in to management don’t do it out of a burning desire to write performance reviews. They do it because they are fed the fuck up with being out of the loop, or not having a say in decisions over their own work. All they want is to be in the room where it happens, and management tends to be the only way you get an invite.

EVERY company says they believe in transparency, but hardly any of them are, by my count. Transparency doesn’t mean flooding people with every trivial detail, or freaking them out with constant fire drills. It does mean being actively forthcoming about important questions and matters which are happening or on the horizon…often before you are fully comfortable with it. Honestly, if you never feel any discomfort about your level of transparency, you probably aren’t transparent enough.

People do better work with more context! You’re equipping them with information to better understand the business problems and technical objectives, and thereby unleashing them and their creativity to help solve them. You’re also opening yourself up to questioning and sanity checks — which may feel uncomfortable, but 🌞sunlight is sanitizing🌞 — it is worth it.

Some practical tips for transparency

At Honeycomb, we present the full board deck after every board meeting in our all hands, and take questions. When we’re facing financial uncertainty, we say so, along with our working plan for dealing with it. We also do org-level updates in all hands, once per quarter per org. Each org presents a snapshot to the company of how they are doing, but we ask that no more than 2/3 of the presentation be about their successes and triumphs, and 1/3 of their material be about their failures and misses. Normalize talking about failure.

Being transparent isn’t about putting everyone on blast; it’s about cultivating a habit of awareness about what might be relevant to other people. It’s about building systems of feedback, updates and open questioning into your culture. This can be scary, so it’s also about training yourselves as a team to handle hard news without overreacting or shooting the messenger. If you always tell people what they want to hear, they’ll never trust you. You can’t trust someone’s ‘yes’ until you hear their ‘no’.

Transparency is always a balance between information and distraction, but I think these are healthy internal rules of thumb for management:

  1. If anyone has further questions or wants to know more details than what was shared, they are free to ask any manager or exec, who will willingly answer more fully, up to the boundaries of privacy or legal reasons. As employees, they have a right to know about the business they are part of. A right — not a privilege, which can be revoked on a whim.
  2. When making internal decisions about e.g. salary bands, individual exceptions to formal policy, etc, ask each other … if this decision were to leak, could we justify our reasoning with head held high? If you would feel ashamed, or if you really don’t want people to find out about it, it’s probably the wrong decision.

Some practical tips for distributing power

Power flows to managers by default, just like water flows downhill. Managers have to actively push back on this tendency by explicitly allocating powers and responsibilities to tech leads and engineers. Don’t hoard information, share context generously, and make sure you know when they would want to tap in to a discussion. Your job is not to “shield” them from the rest of the org; your job is to help them determine where they can add outsize value, and include them. Only if they trust you to loop them in will they feel free to go heads down and focus.

Wrap your senior ICs into planning and other leadership activities. Decisions about sociotechnical processes (code reviews, escalation points, SLI/SLOs, ownership etc) are usually better owned by staff+ engineers than anyone on the management track. Invite a couple of your seniormost engineers to join calibrations — they bring a valuable perspective to performance discussions that managers lack.

Demystify management. Blur the lines between people managers and engineers; delegate ownership and accountability for some important projects to ICs. Ask every engineer about their career interests, and if management is on the list, find opportunities for them to practice and improve at managerial skills — mentoring, interviewing, onboarding, etc.

Adults don’t like being told what to do

People do phenomenal work when they want to do it, when they are creatively and emotionally engaged at the level of optimum challenge, and when they know their work matters. That’s where you’ll find your state of flow. That is where you’ll do your best work, which is also the best way to get promoted and make durable advances in your career.

Not, ironically, by chasing levels and titles for their own sake. ☺️

People want to be challenged. They want you to ask them to step up and take responsibility for something hard. They want to be needed, and they want to have agency in the doing of it. Just like you do.

Oh yeah, back to Molly …

Molly, who I mentioned at the beginning, joined Honeycomb five years ago as a customer success exec. After realizing she wanted to go back to engineering, she switched to working our support desk to build up her technical chops while she practiced writing code on the side. She has now been working as a software engineer on the product team for over two years, and she is ✨rocking it.✨ It is NEVER too late. 🙌

<3 charity

p.s. Molly also says, “don’t waste time at bad companies, whether you’re climbing the ladder or not!” 🥂

 

[1] Formal power is only one kind of power, and in some ways it is the weakest, because it doesn’t belong to you. It belongs to the company and is only loaned out for you to wield on its behalf. (You don’t carry the innate ability to fire people along with you after you stop being an engineering manager, for example.) Formal powers are limited, enumerated, and functional. You don’t get to use them for any reason other than furthering the goals of the org, or else it is literally an abuse of power.

Formal power is fascinating in another way, too: which is that your formal power is seen as legitimate only if you ~basically always wield it in the ways everyone already expects you to. You can make a surprising call only so often; you can straight up overrule the wishes of your constituents extremely rarely. If you use your formal power to do things that people disagree with or don’t support, without taking the time to persuade them or create real consensus, you will squander your credibility and good faith unbelievably fast.

[2] I am not going to bother rustling up lots of links and citations, because I expect most of this falls into the voluminous category of “shit you already knew”. But if any of it sounds surprising to you, here are some classic reference works:

Flow, by Mihaly Csikszentmihalyi
Drive, by Dan Pink
The Culture Code: Secrets of Successful Groups, by Daniel Coyle
A Lapsed Anarchist’s Guide to Being a Better Leader, by Ari Weinzweig

[3] The scientific literature suggests that dominating instincts tend to emerge in more overtly hostile environments. Make of that what you will, I guess.

 

Some other writing I have done on this topic, or topics adjacent …

The Engineer/Manager Pendulum
The Pendulum or the Ladder
If Management Isn’t a Promotion, then Engineering isn’t a Demotion
Twin Anxieties of the Engineer Manager Pendulum
Things to Know About Engineering Levels
Advice for Engineering Managers who want to Climb the Ladder
On Engineers and Influence
Is There a Path Back from CTO to Engineer?

The Hierarchy Is Bullshit (And Bad For Business)

Questionable Advice: Is there a path back from CTO to engineer?

mipsytipsy, , 4 Comments

I received this question in the comments section of my piece on The Twin Anxieties of the Engineer/Manager Pendulum, and figured I might as well answer it. It definitely isn’t a question I’ve spent a lot of time thinking about or anything. 🥰

As a former CTO coming off a sabbatical and wanting to go back to engineering, it’s good to hear that this can be done. Having had coding, architecting, and scaling skills before getting pushed to more lead role and looking to get back to working after the sabbatical, what would the roadmap look like to achieve this? Is it still possible having been away for a few years? What would be a good role to target for re-entry: principal/staff engineer? architect? — Mark

Personally? If I were you, I would return to engineering as a regular old software engineer, writing and shipping code every day in the trenches with (this cannot be emphasized enough) a really, really good team.

Your rustiest skill sets are always going to be the most tactical ones — writing software, reviewing code, reproducing bugs, understanding a new production system.

As a former CTO, you have many other skill sets to pull from — management, strategy, architecture, coaching, staffing, fundraising, etc. These skills are valuable. But they don’t degrade the way hands-on development does. You’ll still remember how to write a performance review two (or twenty) years from now, but writing code is like speaking a language: you use it or lose it. And just like with a language, the best way to freshen up is full immersion.

It’s not just about refreshing your technical chops, it’s also about re-acclimating yourself to the rhythms of hours, days, and weeks spent in focus mode, building and creating.

Think back to the time you first moved from engineering into a management role. Do you remember how exhausting and intrusive it was at first, having meeting after meeting after meeting on your calendar? You had to context switch twenty times a day — you were context switching constantly. You had to walk into room after room after room full of people and their words and emotions. By the end of the day you would be drained dry (and the days felt so long).

As an engineer, you spent your days in stretches of deep focus and concentration, punctuated by the occasional meal, meeting or interruption. But as a manager, your life is nothing but interruptions (and time boxes, and time-boxed interruptions). It took time to for you adjust to manager life and learn where to seek out new dopamine hits. And it’s going to take time for you to adjust back.

How much time? About six months, at least for me. I don’t think it’s being overly dramatic to say that you have to allow enough time to become a different version of yourself. You can’t just change personas and entire ways of being like you change your clothing. The process is more like…a snake shedding its skin, or a caterpillar turning into a butterfly. Don’t rush the process.

And don’t just pick up where you left off as an engineer. This is a beautiful opportunity for you to enjoy the terrible frustration of beginner eyes. ☺️ Learn a new language, learn a new framework, learn a new way of packaging and deploying your code. Freshen up your toolchain. Try a new editor. Catch up on some new testing or validation ideas that have developed or gone mainstream since you were last in the coal mines. (Take modern observability for a test drive? 😉)

Shit moves fast. A lot will have changed.

If you try to pick up where you left off as an expert, it’s really going to suck as you stumble through the motions that used to feel effortless and automatic. But if you start with something new, the friction of learning will feel ordinary and predictable instead. And pretty soon you’ll feel the engine start to kick in: the accelerated learning curve you’ll remember from learning a new technical skill for the 50,000th time.

Because it’s not just about refreshing your technical skills and your daily cadence, either… it’s also about reconnecting with your curiosity, and your attachment to (and love for) technology.

And you better fucking love it, if you plan to inflict the world of agonies that is software development on yourself day after day. 😭 So you have to reconnect with that dopamine drip you get from learning things, fixing shit, and solving problems. And you have to reconnect with the emotional intensity of shipping code that will impact people’s lives — for better or for worse — and of being personally responsible for that code in production. Of knowing viscerally what it’s like to ship a diff that brings production down, or wakes up your coworker in the middle of the night, or corrupts user data.

So yes. It is absolutely possible to return to engineering after a few years away. And yeah, you could come back as a principal or staff engineer. Someone will definitely hire you. However, I suggest otherwise. I suggest you come back as a senior engineer, writing software every day, for a good 6-9 months.

Your grounding in the technical challenges and solution space will be much deeper and richer if you come back hands on than if you came in at a higher level, detached from the rhythms of daily development. Working closer to production and closer to users will give you the chance to rebuild the intense empathy and connectedness to your work and team that tends to seep away the higher you go up the food chain. You’ll be more confident in yourself as a technologist if you honor your need to relearn and rebuild. And you will earn much more respect from your fellow engineers this way. Engineers respect people who respect what they do.

It’s better than jumping straight into the role of a staff+ engineer and trying to refresh your tactical/technical skills on the side. And you’ll be an infinitely more effective staff+ engineer once you’ve done the refreshing.

But if it feels like a demotion, or it’s too hard to swallow, or if the politics of promotions at this company make you leery: compromise by getting yourself hired as a staff or principal engineer, while being clear with your hiring manager that you plan to spend the first 6+ months slinging diffs. They should be fine with it (delighted, really) since a) ANY staff+ hire is an investment for the long run, b) this is a great way to onboard any staff+ engineer, and c) I don’t believe anybody can actually have staff+ level impact during their first 6-12 months at a company, since so much of the job has to do with people, process, technical context, systems history, etc which accrues over time.

Have fun, and do report back! Tell us how it goes!

charity.

P.S.: You don’t say how long it’s been, but I’m operating under the assumption that it’s been 5-10 years since you last worked as an engineer.

P.P.S.: 🚨unsolicited opinion alert🚨 I would personally avoid any role that includes “Architect” in its title (except solutions architects). To me, “software architect” rings of “someone who can no longer write code or perform as a software engineer, who has probably been at the same company for so long that their skills and knowledge now consist entirely of minutiae about that particular company’s technology. likely to be useless and/or helpless at any other company.” I say this with all due apologies to my architect friends, every one of whom is technically dazzling, operationally up-to-date, and has beautiful hair.💆 🥰

 

 

Questionable Advice: Is there a path back from CTO to engineer?

Advice for Engineering Managers Who Want to Climb the Ladder

mipsytipsy, , , 13 Comments

We have been interviewing and hiring a pile of engineering directors at Honeycomb lately. In so doing, I’ve had some fascinating conversations with engineering managers who have been trying unsuccessfully to make the leap to director.

Here is a roundup of some of the ideas and advice I shared with them, and the original twitter thread that spawned this post.

What is an engineering director?

Given all the title inflation and general inconsistency out there, it seems worth describing what I have in mind when I say or hear “Engineering Director.”

In a traditional org chart, an engineering manager usually manages about 5-8 engineers, an engineering director manages 2-5 engineering managers, and a VP of engineering manages the directors. (At big companies, you may see managers and directors reporting to other managers and directors, and/or you may find a bunch of ‘title padding’ roles like Senior Manager, Senior Director etc.)

In smaller companies, it’s common to have a “Head of Engineering” (this is an appropriately weaselly title that commands just the right amount of respect while leaving plenty of space to hire additional people below or above them). Or all of engineering might roll up to a director or VP or CTO. It varies a lot.

When it comes to the work a director is expected to do, though, there’s a fair bit of consistency: we expect managers to manage ICs, and directors to manage managers.  Directors sit between the line managers and the strategic leadership roles. (More on this later.)

So if you’re an engineering manager, and you want to try being a director, the first thing you’ll want to understand is this: it is generally better to get there by being promoted than by getting offered a director title at a different company.

How to level up

Lots of engineers get tapped by their management to become managers, but not many become directors without a conversation and some intentional growth first. This means that for many of you, trying to become a director may be the first time you have ever consciously solicited a role outside the interview process. This can bring up feelings of awkwardness, even shamefulness or inappropriateness. You’ll just have to push through those.

If you ever want a job in upper leadership, you are going to have to learn how to shamelessly state your career goals. We want people in senior leadership who want to be there and are honing their skills in anticipation of an opportunity. Not “oops, I accidentally a VP.”

It is better to get promoted than hired up a level

There are a few reasons for this. It’s usually easier to get promoted than to get hired straight into a job you’ve never held before (at a org with high standards), and it also tends to be more sustainable/more likely to succeed if you get promoted in as well. Being a director is NOT just being a super-duper manager, it’s a different role and function entirely.

A lot of your ability to be successful as a director (or any kind of manager) comes from knowing the landscape, the product and the people, and having good relationships internally. When you are internally promoted, you already know the company and the people, so you get a leg up towards being successful. Whereas if you’ve just joined the company and are trying to learn the tech, the people, the relationships, and how the company works all at once, on top of trying to perform a new role for the first time.. well, that is a lot to take on at once.

There are exceptions, sure! Oodles of them[1]. But I would frankly look sideways at a place that wanted to hire me as a director if I haven’t been one, or hadn’t at least managed managers before. It’s at least a yellow flag. It tells me they are probably either a) very desperate or b) very sloppy with handing out titles.

If you want a promotion…

The obvious first step involves asking your manager, “what is the skill gap for me between the job I am doing right now and a director role?” Unlike in the movies, promotions don’t usually get surprise-dropped on people’s heads; people are usually cultivated for them. Registering your interest makes it more likely they will consider you, or help you develop skills in that direction as time moves on.

If you have a good manager who believes in you, and the opportunities exist at your company, that might even be all you have to do.(!)

And if so, lucky you. But as for the remaining ~80-90% of us (ha!) … well, we’ll need a bit more hustle.

Take inventory of your opportunities

Lots of companies aren’t large enough to need directors, or growing fast enough to create new opportunities. This can actually be the most challenging part of the equation, because there are generally a lot more managers who want to be directors than there are available openings.

If you do need to find a new job to reach your career goals, I would target fast-growing companies with at least 100 engineers. If you’re evaluating prospective employers based on your chance of advancement, consider the following::

  • Ask about their policies on internal vs external hires. Do they give preference to existing employees? How do they decide when to recruit vs grow from within?
  • Ask about the last time that someone was promoted into a similar role.
  • Tell the recruiter and hiring manager about your career goals. Don’t be shy. “My next career goal is to gain some experience managing managers” is fine. (That shouldn’t be the only reason you’re interested, of course.)
  • Size up the playing field. Is there oxygen at that level? Or are there a dozen other managers senior to you lined up for the same shot?

There are no sure bets. But you can do a lot to put yourself in the right place at the right time, signal your interest, and be prepared for the opportunity when it strikes.

a director is not a ‘super-senior manager’

A director is not just a manager on steroids: it is an entirely different job. It helps to have been a good manager before becoming a director, because many management skills will translate, but others will be entirely new to you. Expect this.

How being a good director is different from being a good manager

Let’s look at some of the ways that being a good engineering manager is different than being a good director.

  1. You can be a great EM, beloved by your team, without giving much thought to managing out or up. Directors cannot. If anything, it’s the opposite. You may get away with not coddling your EMs, but you must pull your weight for your peers and upper management.
  2. You can have a bit of a reputation for being stubborn or difficult as an EM, and that can be just fine. But having such a rep will probably sabotage your attempt at being promoted to director.
  3. You can be a powerful technical EM who sometimes jumps in to train engineers, be on call, or course correct technical and architectural decisions. This can even burnish your value and reputation as an EM. But this would all be a solid knock against you as a director.

Managers can get away with being opinionated and attached to technology, to some extent, while directors absolutely must balance lots of different stakeholders to achieve healthy business outcomes.

This difference of perspective is why managers will sometimes sniff about directors having sold out, or being “all about politics.”

(Blaming something on “politics” is usually a way of accidentally confessing that you don’t actually understand the constraints someone is operating under, IMO.)

A director’s job is running the business

Here’s the key fact: ✨directors run the business✨.

Managers should be focused on high-performing engineering teams. VPs should be focused on strategy and the longer term. Directors are the execution machines that knit technology with business objectives. (I like this piece, although the lede is a little buried. Key graf:)

managers, directors, VPs

Directors run the business. They are accountable for results. You can’t be bopping in and writing or reviewing code, or tossing off technical opinions. That’s not your job anymore.

Managing managers is a whole new skill set

The distance between managing engineers and managing managers is nearly as vast as the gulf between being an engineer and being a manager.

But it’s sneakier, because you don’t feel out of your depth as much as you did when you became a manager. 😁

As a manager, each of us instinctively draws on our own unique blend of strength and charisma — whatever it is that makes people look up to you and willing to accept your influence. Most of us can’t explain how we do it, because we run on instinct.

But as a director, you have to figure it out. Because you need to be able to debug it when the magic breaks down. You need to help your managers influence and lead using *their* unique strengths. What works for you won’t work for them. You have to learn how to unpack different leadership styles and support them in the way they need.

If you’re working towards a director role:

There are lots of areas where you can improve your director skills and increase your chances of being viewed as director material without any help whatsoever from your manager.

You ✨can not✨ be a blocker

Directors run the business … so you CANNOT be seen as a blocker. People must come to you of their own accord to get shit done and break through the blockers.

If they are going to other people for advice on how to break through YOU, you are not a good candidate for director. Figure out how to fix this before you do anything else.

Demonstrate impact beyond your team(s)

Another way to make yourself an attractive prospect for director is to work on systemic problems, driving impact at the org or company level. You could:

  • work to substantially increase the diversity of your teams or your candidate pipeline, and offer to work with recruiting and other managers to help them do the same (becoming BFFs with recruiting is often a canny move)
  • drive some cross-platform initiative to consolidate dozens of snowflake deploy processes and significantly reduce CI/CD build/deploy times, set an internal SLO for artifact build times, or successfully champion auto-deployment
  • champion an internal tools team with a mandate to increase developer productivity, and quantify the hell out of it
  • lead a revamp of the new hire onboarding process. Or add training and structure to the interview process and set an SLO of responding to every candidate within one week

I dunno — it all depends on what’s broken at your company. 🙃 Identify something causing widespread pain and frustration at the organizational level and fix it. 

Managing ‘up’ is not a ‘nice-to-have’…

If there’s a problem, make sure you are the one to bring it to your manager (and swiftly), along with “Here is the context, here’s where I went wrong, and this is what I’m planning to do about it.” No surprises.

At this point in your career, you should have mastered the art of not being a giant pain in the ass to your manager. Nobody wants a high-maintenance director. Do you reliably make problems go away, or do they boomerang back five times worse after you “fix” them?

…Neither is managing ‘out’

Managing “out” is important too. (Not “managing out”, which means terminating people from the company, but managing “out” as in horizontally, meaning your relationship with your peers.)

What do your peers think of you? Do you invest in those relationships? Do they see you as an ally and a source of wise counsel, or a source of chaos, gossip and instability, or a competitor with turf to protect? If you’re the manager that other managers seek out for a peer check, you might be a good candidate for director.

psst.. People are watching you

One of the most uncomfortable things to internalize if you climb the ladder is how much people will make snap judgments about you based on the tiniest fragments of information about you, and how those judgments may forever color the way they think of you or interact with you.

First impressions might be made by ten minutes together on the same zoom call…a few overheard fragments of people talking about you…even the expressions on your face as they pass you in the hallway. People will extrapolate a lot from a very little, and changing their impression of you later is hard work.

(Yes it’s frustrating, but you can’t really get upset about it, because you and I do it too. It’s part of being human. )

Because of that, you really do have to guard against being too cranky, too tired, or out of spoons. People WILL take it personally. It WILL come back to hurt you.

Remember, you don’t hear most feedback. If you visibly disagree with someone, assume 10x as many silently agree with them. If one person gives you a piece of hard feedback, assume 10x as many will never tell you. Be grateful. The more power you are perceived to have, the less feedback you will ever hear.

Pro tip

You can infer a surprising amount about how good a director candidate may be at their job, simply by listening closely to how they talk about their colleagues. Do they complain about being misunderstood or mistreated, do they minimize the difficulty or quality of others’ work, do they humblebrag, or do they take full responsibility for outcomes? And does their empathy fully extend to their peers in other departments, like sales and marketing?

Does it sound like they enjoy their work, and look forward to beginning it every day? Does it sound like they are all in the same little tugboat, all pulling in the same direction, or is there a baseline disconnect and lack of trust?

In conclusion…

Be approachable, be a drama dampener, project warmth. Control your calendar and carve out regular focus time. Guard your energy — never run your engine under 30%, and always leave something in the tank.

There are a lot more great responses and advice in the replies to my thread, btw. Go check them out if you’re interested.. and if you have something to say, contribute!.☺️

charity

Footnotes:

[1] Occasionally, it may work out to your benefit to jump into a new, higher title at a new company. This can happen when someone is already well qualified for the higher role, but is finding it difficult to get promoted (possibly due to insufficient opportunity or systemic biases). Just be aware that the job you were hired into is likely to be one where the titles are meaningless and/or the roles are chaotic. You may want to stay just long enough to get the title, then bounce to a healthier org.

Advice for Engineering Managers Who Want to Climb the Ladder

How can you tell if the company you’re interviewing with is rotten on the inside?

mipsytipsy, , , , 15 Comments

How can you tell the companies who are earnestly trying to improve apart from the ones who sound all polished and healthy from the outside, whilst rotting on the inside?

This seems to be on a lot of minds right now, what with the Great Resignation and all. There are no perfect companies, just like there are no perfect relationships; but there are many questions and techniques you can use to increase your confidence that a particular company is decent and self-aware, one whose quirks and foibles you are compatible with.

https://twitter.com/jetpack/status/1486212123525287938

Interviews are designed to make you feel like you are under inspection, like the interviewer holds all the power. This is an illusion. Your labor is valuable — it is vital — and you should be scrutinizing them every bit as closely as they you. In fact, here is Tip #1:

  • If they allow you plenty of time to converse with your interviewers throughout the process, great. If they tack on a cursory “any questions for us?” while wrapping up, they don’t think it matters what you think of them. Pull the ripcord.

Collect and practice good interview questions for you to ask potential employers. Write them down — your mind is likely to go blank under stress, and you don’t want to let them off the hook. There is a LOT of signal to be gained by probing down below the surface answers.

Backchanneling

  1. Whisper networks and backchannels are incredibly important. It can be especially valuable to talk to someone who has recently left the company: why did they leave? Would they go there again?
  2. Alternately, do you know anyone who has worked for or with their leadership, even if not at that company?
  3. If you know any women or under-represented minorities (URM) who work there, buy them lunch and ask for the unvarnished truth. That’s where you usually turn up the real dirt. 🥂

Diversity, equity and inclusion

Just because a company has a diverse workforce doesn’t necessarily mean it is a healthy place to work. (But it’s fair to give some points up front, because that doesn’t usually happen by accident.)

  1. Do they have a diverse leadership team? A diverse board?
  2. Is their company diverse overall, or are minorities concentrated in a few (lower-paying, high-turnover) departments?
  3. You might not want to write off all the companies that don’t meet points one and two, if for no other reason than it dramatically shrinks your available option pool. If they don’t have a particularly diverse team, and this is something that matters to you, that’s your cue to dig deeper:
    • Are they bothered by their lack of diversity? What’s the plan? Do they just feel generically sad about it, or have they set specific goals to improve by specific dates? What investments are they making?
    • Who works on DEI stuff currently? (Answers like “HR and recruiting”, or “we have a woman who’s really good at it” are bad answers.)
    • Who is accountable for making sure those goals are hit? (The only right answer is “our execs”. Having a “chief diversity officer” is an anti-pattern in my book.)
    • If the team is all guys, for example, ask if they’ve ever had any women on the team in the past. Did she/they leave? Do they know why?
  4. This is a GREAT one: “As a white man, I’d ask what they’ve done to find qualified women and minorities for the role I’m interviewing for.” (via David Daly) 🔥🔥

Company stuff

  1. What are their values? Do they feel bloodless and ripped from the pages of HBR, or are they unique, lived-in, and give you a glimpse of what the people there care about? Are they mentioned over the course of your interview?
  2. Ask tough questions about the business and try to ascertain whether they are hitting their quarterly goals, how much funding they have in the bank, what the growth curve looks like, what users really think about their product, and what the biggest obstacles to success are.
    • Companies that are floundering are going to be really stressful places to work, and even if the leadership is decent, they may find themselves backed into making some really tough decisions.
    • You want to work at a company on a strong growth trajectory for lots of reasons, but a big one is your own growth potential. You will learn the most the fastest at places that are growing fast, and have way more openings for promotions and leadership roles than a slower-growing company.
  3. Are people willing to speak freely about things they’ve tried that have failed, and things that don’t work well currently? Being self-aware and comfortable with visible failure are two of the most important self-correcting mechanisms a company can cultivate.
  4. EVERYBODY thinks they value transparency, so I wouldn’t even bother asking. Instead, ask for specific examples of leadership being forthcoming with bad news to the team, and team members delivering hard feedback or bad news to upper leadership. Transparency shouldn’t be something they’re especially proud of, so much as it is taken for granted. It’s in the air that you breathe.

Planning and the unplanned

  1. Ask about how decisions get made. A chestnut is, “how does work end up on my plate?” — meaning is there a business strategy (owned by whom?), a technical strategy, a product strategy, quarterly KPIs, customer requests, manager delegation, JIRA tickets…? (The most important part may be how similar the answers you get are. 🙃)
  2. How often does work get pre-empted and why? It’s a good thing if product development has to get put on ice once in a while so the team can focus on reliability and maintenance work. It’s a bad thing if they’re expected to stuff reliability work in the cracks around their product development, or if they’re incapable of sticking to a plan.
  3. What does “crunch time” look like? Nearly every company has one from time to time (it might even be a bad sign if it never happens), but this is when you find out your leadership’s true colors.
    • Do they praise people or call them out to thank them for pulling all nighters and other extremist behaviors? 🚨BZZT🚨
    • Is it voluntary? Are you trusted to set your own pace, your own limits, or are you pressured to do more? Are people expected to participate to the extent that they are able, and not expected to justify how hard or how much (so long as they communicate their capacity, of course)?
    • How long did it last, and how often does it happen, and why? It should be rare (1-2x/year at most), involve the whole company, and move the business forward meaningfully
    • Did they follow through by making sure people took time off afterwards to recover? Not just give permission, but actually make sure the human beings had a chance to refresh themselves? Did leaders set a good example by taking a breather themselves?

Believe it or not, crunch time done correctly can be an enormously exciting, intense, bonding time for a group of people who love what they do, culminating in a surge of collective triumph and celebration, followed by recovery time. If it was done correctly, and you ask about it afterwards, people will 💡light up💡.

Team stuff

  1. Unfortunately, culture can vary widely from function to function, even from manager to manager. Make sure you get a real interview slot with your actual manager — not just a screener or wrap-up call — and as much of the team as possible, too.
  2. Ask your potential manager about the last person they had to let go. Why? What was the process? What was the impact on the team? How did the person feel afterwards?
  3. Who is on call? How often do people get paged outside of hours, and how frequently do they work an incident? (Do managers track this?) Are you expected to keep shipping product during on call weeks, or devote your time to making the system better?
  4. If you had to ship a single line of code to production using the deploy pipeline, how long would that take? Remember, the lower the deploy interval, the happier and more productive you are likely to be as an engineer there. Under 15 minutes is great. Under an hour is tolerable. More than that, proceed with great caution.

The interview itself

  1. Was your interview well-organized and conducted in a timely fashion? Were you given detailed information about what to expect, and were your interviewers well-prepared, and conversational? Were the questions fair, open-ended, and relevant to the job in question?
  2. If they asked you to perform any kind of take-home labor of more than an hour or so, did they compensate you for your time?
  3. Did they get back to you swiftly at each step of the way to let you know where you stand and what comes next?
  4. Did you find the questions interesting and challenging? Do they have a clear idea of what success looks like for you in this role? Did you leave excited and buzzing with ideas about the work you could do together?
    • This is 👆 definitely more of a “how good is this job” question than “is this a shithole” question, but one of our honeycombers brought it up as an example of how a great interview can make you decide to leave a job , even one you’re perfectly happy with.
  5. The questions they ask you while interviewing you are the questions they ask everyone else. So…did they ask you about your views on diversity and team dynamics while interviewing you? Or is that not part of their filters, only their advertised persona?

Three more

  1. Do their employees seem to speak freely on twitter? If you are an agitator of sorts, are there others who agitate about similar issues — either with company support, or at least lack of censorship?
  2. How does the company respond to criticism and feedback? For that matter, how do they treat their competitors? Being competitive is fine, being mean is not.
  3. Get clear on your own expectations. What’s on your wishlist, and what’s make-or-break for you? If something is very important to you, consider telling the hiring manager up front. For example, “These are my expectations for how women are treated. How do you think your company matches up?” Their answers will speak volumes, and so will their comfort level with the question.

In closing

If you a join a new company, and two or three weeks in you’re just not feeling it, you’re wondering if you made a mistake — leave. You do not owe them a year of your life. Trust your instincts. Just leave it off your resume entirely and roll the dice again.

Employers are all too accustomed to feeling (and acting) like they hold all the power. They do not. Every tech company is a talent business, which rises and falls on the caliber of the people they can convince to stay. They aren’t doing you a favor by employing you; you are doing THEM a favor by lending them your creativity, labor, and a third of the hours in your day.

Do they deserve it? Will their success make the world a better place? If not, stop supporting them with your work and lend your muscle to a company that deserves it.

In the hottest job market of my lifetime, with millions of opportunities newly open to people who live literally anywhere, you owe it to yourself, your future self, and your family to take a good hard look at where you sit. 🍄 Are you happy? 🍄 Are you compensated well, is your time valued? 🍄 Are you still learning new things and improving your skills every day? 🍄 Is your company still on a growth trajectory? 🍄 Do you trust your leadership and your team, 🍄 do still you believe in the mission, and 🍄 do you think your labor contributes meaningfully to making the world a better place?

If not, consider joining the Great Resignation. I hear they have cookies.

Huge thanks to Amy Davis, Phillip Carter, Ian Smith, Sarah Voegeli, Kent Quirk, Liz Fong-Jones, Amanda Shapiro, Nick Rycar, Fred Hebert and David Daly, all of whom contributed to this post!

How can you tell if the company you’re interviewing with is rotten on the inside?

How “Engineering-Driven” Leads to “Engineering-Supremacy”

mipsytipsy, , , , , 11 Comments

Honeycomb has a reputation for being a very engineering-driven company. No surprise there, since it was founded by two engineers and our mission involves building an engineering product for other engineers.

We are never going to stop being engineering-driven in the sense that we are building for engineers and we always want engineers to have a seat at the table when it comes to what we build, and how, and why. But I am increasingly uncomfortable with the term “engineering-driven” and the asymmetry it implies.

We are less and less engineering-driven nowadays, for entirely good reasons — we want this to be just as much of a design-driven company and a product-driven company, and I would never want sales or marketing to feel like anything other than equal partners in our journey towards revolutionizing the way the world builds and runs code in production.

It is true that most honeycomb employees were engineers for the first few years, and our culture felt very engineer-centric. Other orgs were maybe comprised of a person or two, or had engineers trying to play them on TV, or just felt highly experimental.

But if there is one thing Christine and I were crystal clear on from the beginning, it’s this:

✨WE ARE HERE TO BUILD A BUSINESS✨

Not just the shiniest, most hardcore tech, not just the happiest, most diverse teams. These things matter to us — they matter a lot! But succeeding at business is what gives us the power to change the world in all of these other ways we care about changing it.

If business is booming and people are thirsty for more of whatever it is you’re serving, you pretty much get a blank check for radical experiments in sociotechnical transformation, be that libertarian or communitarian or anything in between.

If you don’t have the business to back it up, you get fuck-all.

Not only do you not get shit, you risk being pointed out as a Cautionary Tale of “what happens if $(thing you deeply care about) comes true.” Sit on THAT for a hot second. 😕

So yes, we cared. Which is not to say we knew how to build a great business. We most certainly did not. But both of us had been through too many startups (Linden Lab, Aardvark, Parse, etc) where the tech was amazing, the people were amazing, the product was amazing … and the business side just did not keep up. Which always leads to the same thing: heartbreak and devastation.

✨IF you can’t make it profitable✨
✨your destiny will inevitably be✨
✨taken out of your hands✨
✨and given to someone else.✨

So Christine and I have been repeating these twin facts back and forth to each other for over six years now:

  1. Honeycomb must succeed as a revenue-generating, eventually profitable business.
  2. We are not business experts. Therefore we have to make Honeycomb a place that explicitly values business expertise, that places it on the same level as engineering expertise.

We have worked hard to get better at understanding the business side (her more than me 🙃) but ultimately, we cannot be the domain experts in marketing or sales (or customer success, support, etc).

What we could do is demonstrate respect for those functions, bake that respect into our culture, and hire and support amazing business talent to run them with us.

On being “engineering-driven”

Self-described “engineering-driven” companies tend to fall into one of two traps. Either they alienate the business side by pinching their nose and holding business development at arm’s length (“aaahhh, i’m just an engineer! I have no interest in or capacity for participating in developing our marketing voice or sales pitch decks, get off me!”), or they act like engineering is a sort of super-skillset that makes you capable of doing everybody else’s job better than they possibly could. As though those other disciplines and skill sets aren’t every bit as deep and creative and challenging in their own right as developing software can be.

For the first few years we really did use engineers for all of those functions. We were trying to figure out how to build and explain and sell something new, which meant working out these things on the ground every day with our users. Engineer to engineer. What resonated? What clicked? What worked?

So we hired just a few engineers who were interested in how the business worked, and who were willing to work like Swiss Army knives across the org. We didn’t yet have a workable plan in place, which is what you need in order to bring domain experts on board, point them in a direction, and trust them to do what they do best in executing that plan.

Like I said, we didn’t know what to do or how to do it. But at least we knew that. Which kept us humble. And translated into a hard, fast rule which we set early on in our hiring process:

We DO NOT hire engineers who talk shit about sales and marketing.

If I was interviewing an engineer and they made any alienating sort of comment whatsoever about their counterparts on the business side, it was an automatic no. Easy out. We had a zero tolerance policy for talking down down about other functions, or joking, even for being unwilling to perform other business functions.

In retrospect, I think this is one of the best decisions we ever made.

Hiring engineers who respected other functions

We leaned hard into hiring engineers who asked curious questions about our business strategy and execution. We pursued engineers who talked about wanting to spend time directly with users, who were intrigued by the idea of writing marketing copy to help explain concepts to engineers, and who were ready, willing and able to go along on sales calls.

Once we finally found product-market fit, about 2-3 years ago, we stopped using engineers to play other roles and started hiring actual professionals in product, design, marketing, sales, customer success, etc to build and staff out their organizations. That was when we first started building the business for the longer term; until we found PMF, our event horizon was never more than 1-3 months ahead of “right now”.

(I’ll never forget going out to coffee with one of our earlier VPs of marketing, shortly after she was hired, and having her ask, in bemusement: “Why are all these engineers just sitting around in the #marketing channel? I’ve never had so many people giving opinions on my work!” 🙃)

Those early Swiss Army knife engineers have since stepped back, gratefully, into roles more centered around engineering. But that early knee-jerk reaction of ours established an important company norm that pays dividends to this day.

Every function of the business is equally challenging, creative, and worthy of respect. None of us are here to peacock; our skill sets serve the primary business goals. We all do our jobs better when we know more about how each other’s functions work.

These days, it’s not just about making sure we hire engineers who treat business counterparts like equals. It’s more about finding ways to stimulate the flow of information cross-functionally, creating a hunger for this information.

Caring about the big picture is a ✨learnable skill✨

You can try to hire people who care about the overall business outcomes, not just their own corner of reality, and we do select for this to  some degree — for all roles, not just engineers.

But you can also foster this curiosity and teach people to seek it out. Curiosity begets curiosity, and every single person at Honeycomb is doing something interesting. We all want to succeed and win together, and there’s something infectious and exciting about connecting all the dots that lead to success and reflecting that story back to the rest of the company.

For example,

    1. Every time we close a deal, a post gets written up and dropped into the announcements slack by the sales team. Not just who did we close and how much money did we make, but the full story of that customer’s interaction with honeycomb. How did they hear of us? Whose blog posts, training sessions, or office hours did they engage with? Did someone on the telemetry pull a record-fast turnaround on an integration they needed to get going? What pains did we solve effectively for them as a tool, and where were the rough edges that we can improve on in the future?

      The story is often half a page long or more, and tags a dozen or more people throughout all parts of the company, showing how everyone’s hard work added up and materially contributed to the final result.

    2. We have orgs take turn presenting in all hands — where they’re at, what they’ve built, and the impact of their contributions, week after week. Whether that’s the design team talking about how they’ve rolled out our new design system and how it is going to help everyone in the company experiment more and move more quickly, or it’s the people team showing how they’ve improved our recruiting, interviewing, and hiring processes to make people feel more seen, welcomed, and appreciated throughout the process.

      We expect people to be curious about the rest of the company. We expect honeybees to be interested in, excited about and celebratory of each other’s hard work. And it’s easy to be excited when you see people showing off work that they were excited to do.

    3. We have a weekly Friday “demo day” where people come and show off something they’ve built this week, rapid-fire. Whether it’s connecting to a mysql shell in the terminal to show off our newly consistent permissioning scheme, or product marketing showing off new work on the website. Everybody’s work counts. Everybody wants to see it.

    4. We have a #love channel in slack where you can drop in and tag someone when you’re feeling thankful for how much they just made your day better. We also have a “Gratefuls” section during all hands, where people speak up and give verbal props to coworkers who have really made a difference in their lives at work.

We have always attracted engineers who care about the business, not just the technology and the culture. As a result, we have consistently recruited and retained business leaders who are well above our weight class — our investors still sometimes marvel at the caliber of the business talent we have been able to attract. It is way above the norm for developer tools companies like ours.

“Engineering-driven” can be a mask for “engineering-supremacy”

Because the sad truth is that so many companies who pride themselves on being “engineering-driven” are actually what I would call more “engineering-supremacist”. Ask any top-tier sales or marketing leader out there about their experiences in the tech industry and you’ll hear a painful, rage-inducing list of times they were talked down to by technical founders, had their counsel blown off or overridden, had their plans scrapped and their budgets cut, and every other sort of disrespectful act you can imagine.

(I am aware that the opposite also exists; that there are companies and cultures out there that valorize and glorify sales or marketing while treating engineers like code monkeys and button pushers, but it’s less common around here. In neither direction is this okay.)

This isn’t good for business, and it isn’t good for people.

It is still true that engineering is the most mature and developed organization at the company, because it has been around the longest. But our other orgs are starting to catch up and figure out what it means to “be honeycomby” for them, on their own terms. How do our core values apply to the sales team, the developer evangelists, the marketing folks, the product people? We are starting to see this play out in real time, and it’s fascinating. It is better than forcing all teams to be “engineering-driven”.

Business success is what makes all things possible

We are known for the caliber of our engineering today. But none of that matters a whit if you never hear about us, or can’t buy us in a way that makes sense for you and your team, or if you can’t use the product, or if we don’t keep building the right things, the things you need to modernize your engineering teams and move into the future together.

When looking towards that future, I still want us to be known for our great engineering. But I also want us to be a magnet for great designers who trust that they can come here and be respected, for great product people who know they can come here and do the best work of their life. That won’t happen if we see ourselves as being “driven” by one third of the triad.

Supremacy destroys balance. Always.

And none of this, none of this works unless we have a surging, thriving business to keep the wind in our sails.

~charity

How “Engineering-Driven” Leads to “Engineering-Supremacy”

Questionable Advice: The Trap of The Premature Senior

mipsytipsy, , , , 14 Comments

I’ve been at my current job for three years, and I am suddenly, accidentally, the most senior engineer on the team. I spend my days handling things like bootcamps, mentoring, architecture, and helping other engineers carve off meaningful work. This has taken a huge toll on the kind of work I want to do as an IC. I still enjoy writing and shipping features, and I am not a manager, but now I feel like I spend my days conducting meetings, interviewing, and unblocking others constantly instead of writing code myself.

What should I do? How can I deal with this situation in an effective manner? How can I keep from getting burned out on zoom? How can I reclaim more of my time to write code for myself, without sacrificing my influence? Should I get a new job? I have thought about going out and getting a new job, but I really like having a say at a high level. Here I get looped into all of the most important decisions and meetings. If I get a new job, how can I avoid starting over at the bottom of the heap and just taking assignments from other people? P.S., this is my first job.

 

Get a new job.

Yes, you will reset your seniority and have to earn it all over again. Yes, it will be uncomfortable and your ego will be cranky over it. Yes, you will be at the bottom of the heap and take assignments from other people for a while. Yes, you should do it anyway.

What you are experiencing now is the alluring comfort of premature seniority. You’re the smartest kid in the room, you know every corner of the system inside and out, you win every argument and anticipate every objection and you are part of every decision and you feel so deeply, pleasingly needed by the people around you.

It’s a trap.

Get the fuck out of there.

There is a world of distance between being expert in this system and being an actual expert in your chosen craft. The second is seniority; the first is merely .. familiarity

Deep down I think you know this, and feel a gnawing insecurity over your position; why else would you have emailed me? You were right. Treasure that uneasy feeling in your gut, that discomfort in the face of supreme comfortable-ness. It will lead you to a long and prosperous career as an engineer if you learn to trust it.

Think of every job like an escalator — a 50-foot high escalator that takes about two years to ride to the top. But once you’ve summited, you stall out. You can either stay and wander on that floor, or you can step to the left and pick another escalator and ride it up another 50 feet. And another.

In my mind, someone becomes a real senior engineer after they’ve done this about three times. 2-3 teams, stacks, languages, and roles, over a 5-8 year period, and then they’re solidly baked. There are insights you can derive from having seen problems solved in a few different ways that you can’t with only a single point of reference.

You don’t become a senior engineer at the 50-foot ascent, no matter how thoroughly you know the landscape. You become a senior engineer somewhere well over 100 feet, with a couple of lane changes under your belt.

The act of learning a new language and/or stack is itself an important skill. Experiencing how different orgs ship code in vastly different ways is how you internalize that there’s no one blessed path, only different sets of tradeoffs, and how you learn to reason about those tradeoffs.

And it is good for us to start over with beginner eyes. It’s humbling, it’s clarifying, it’s a cleanse for the soul. If you get too attached to feeling senior, to feeling necessary, you will undervalue the virtues of fresh eyes and questioning, of influence without authority. It is good for you to practice uncertainty and influencing others without the cheat codes of deep familiarity.

Nobody wants to work with seniors who clutch their authority with a white knuckled grip. We want to work with those who wear it lightly, who remember what it was like in our shoes.

Ultimately, this is a strong argument for building our teams behind a Rawlsian veil of ignorance concerning our own place in the pecking order. Starting fresh yourself will help you build teams where it is not miserable to be a beginner, where beginners’ contributions are recognized, where even beginners do not simply “take orders”, as you said. Because literally nobody wants that, including the beginners you are working with on  your teams today.

After you have gotten a new job or two, and proven to yourself that you can level up again and master new stacks and technologies, that fretful inner voice questioning whether you deserve the respect you receive or not will calm down. You will have proven to yourself that your success wasn’t just a one-off, that you can be dropped into any situation, learn the local ropes and succeed. You will be a senior engineer.

Get the fuck out of there. Go. <3

 

 

 

Questionable Advice: The Trap of The Premature Senior

On Call Shouldn’t Suck: A Guide For Managers

mipsytipsy, , , , , , 7 Comments

There are few engineering topics that provoke as much heated commentary as oncall. Everybody has a strong opinion. So let me say straight up that there are few if any absolutes when it comes to doing this well; context is everything. What’s appropriate for a startup may not suit a larger team. Rules are made to be broken.

That said, I do have some feelings on the matter. Especially when it comes to the compact between engineering and management. Which is simply this:

It is engineering’s responsibility to be on call and own their code. It is management’s responsibility to make sure that on call does not suck. This is a handshake, it goes both ways, and if you do not hold up your end they should quit and leave you.

As for engineers who write code for 24×7 highly available services, it is a core part of their job is to support those services in production. (There are plenty of software jobs that do not involve building highly available services, for those who are offended by this.) Tossing it off to ops after tests pass is nothing but a thinly veiled form of engineering classism, and you can’t build high-performing systems by breaking up your feedback loops this way.

Someone needs to be responsible for your services in the off-hours. This cannot be an afterthought; it should play a prominent role in your hiring, team structure, and compensation decisions from the very start. These are decisions that define who you are and what you value as a team.

Some advice on how to organize your on call efforts, in no particular order.

  • It is easier to keep yourself from falling into an operational pit of doom than it is to claw your way out of one. Make good operational hygiene a priority from the start. Value good, clean, high-level abstractions that allow you to delegate large swaths of your infrastructure and operational burden to third parties who can do it better than you — serverless, AWS, *aaS, etc. Don’t fall into the trap of disrespecting operations engineering labor, it’s the only thing that can save you.
  • Invest in good release and deploy tooling. Make this part of your engineering roadmap, not something you find in the couch cushions. Get code into production within minutes after merging, and watch how many of your nightmares melt away or never happen.
  • Invest in good instrumentation and observability. Impress upon your engineers that their job is not done when tests pass; it is not done until they have watched users using their code in production. Promote an ownership mentality over the full software life cycle. This is how dev.to did it.
  • Construct your feedback loops thoughtfully. Try to alert the person who made the broken change directly. Never send an alert to someone who isn’t fully equipped and empowered to fix it.
  • When an engineer is on call, they are not responsible for normal project work — period. That time is sacred and devoted to fixing things, building tooling, and creating guard-rails to protect people from themselves. If nothing is on fire, the engineer can take the opportunity to fix whatever has been annoying them. Allow for plenty of agency and following one’s curiosity, wherever it may lead, and it will be a special treat.
  • Closely track how often your team gets alerted. Take ANY out-of-hours-alert seriously, and prioritize the work to fix it. Night time pages are heart attacks, not diabetes.
  • Consider joining the on call rotation yourself! If nothing else, generously pinch hit and be an eager and enthusiastic backup on the regular.
  • Reliability work and technical debt are not secondary to product work. Budget them into your roadmap, right alongside your features and fixes. Don’t plan so tightly that you have no flex for the unexpected. Don’t be afraid to push back on product and don’t neglect to sell it to your own bosses. People’s lives are in your hands; this is what you get paid to do.
  • Consider making after-hours on call fully-elective. Why not? What is keeping you from it? Fix those things. This is how Intercom did it.
  • Depending on your stage and available resources, consider compensating for it.This doesn’t have to be cash, it could be a Friday off the week after every on call rotation. The more established and funded a company you are, the more likely you should do this in order to surface the right incentives up the org chart.
  • Once you’ve dug yourself out of firefighting mode, invest in SLOs (Service Level Objectives). SLOs and observability are the mature way to get out of reactive mode and plan your engineering work based on tradeoffs and user impact.

I believe it is thoroughly possible to construct an on call rotation that is 100% opt-in, a badge of pride and accomplishment, something that brings meaning and mastery to people’s engineering roles and ties them emotionally to their users. I believe that being on call is something that you can genuinely look forward to.

But every single company is a unique complex sociotechnical snowflake. Flipping the script on whether on call is a burden or a blessing will require a unique solution, crafted to meet your specific needs and drawing on your specific history. It will require tinkering. It will take maintenance.

Above all: ✨RAISE YOUR STANDARDS✨ for what you expect from yourselves. Your greatest enemy is how easily you accept the status quo, and then make up excuses for why it is necessarily this way. You can do better. I know you can.

There is lots and lots of prior art out there when it comes to making on call work for you, and you should research it deeply. Watch some talks, read some pieces, talk to some people. But then you’ll have to strike out on your own and try something. Cargo-culting someone else’s solution is always the wrong answer.

Any asshole can write some code; owning and tending complex systems for the long run is the hard part. How you choose to shoulder this burden will be a deep reflection of your values and who you are as a team.

And if your on call experience is mandatory and severely life-impacting, and if you don’t take this dead seriously and fix it ASAP? I hope your team will leave you, and go find a place that truly values their time and sleep.

 

On Call Shouldn’t Suck: A Guide For Managers

The Official, Authorized List Of Legitimate Reasons For Deciding to Become a Manager

mipsytipsy, , , 17 Comments

“Why did you decide to become a manager?”

It’s a question that gets asked a lot, in job interviews, 1x1s, and plain old casual conversation. I ask this question a lot, and I am often frustrated (or bored) by the answers I hear back.

Most of them can be bucketed in one of three ways:

  1. The pious. “I just really, really love helping other people achieve their goals.”
  2. The pleasers. the ones who answer, then pause uncertainly: “Is that what you’re looking for?”
  3. The sheepish. “I probably shouldn’t say this, but..” (followed by something very close to real honesty)

People are rarely inclined to divulge the range and depth of their reasons for going into management. And why should they? We are constantly being lectured about what the RIGHT reasons for going into management are, with aspersions cast upon anyone who dares enter the profession for any reasons that are not completely selfless.

“I LOVE mentoring.” “I wanted to protect my team.” “I’m motivated by people problems.” “I just really love helping people grow.”

Okay.

I’m not saying that everybody who says these words is lying, but I would be surprised if it was the entire story. People make career moves for a complex mix of altruism and self-interest.

It’s socially acceptable to cop to the selfless reasons. But what about the rest? Like “I wanted more money”? “I wanted career progression and couldn’t get any as an IC”? What about “I couldn’t get a seat at the table as an engineer”, “I was tired of being left out of important decisions”, or “My reporting chain was opaque and kept fucking up, and I figured I couldn’t do any worse than those bozos”?

Now we’re talking.

Most people become managers to compensate for org fuckery.

In my experience, most engineers become managers primarily due to organizational dysfunction. When you become a manager you acquire certain institutional powers, and you can use those powers to change the thing that makes you miserable.

It’s a hack. A gnarly one. And like most hacks, it kinda works.

For example, say it pisses you off to be left out of decisions. So you become a manager, and then you can either a) use your power and access to push for including engineers in the decision-making process, or at very least b) you personally will no longer left out.

In a healthy org, I would argue that most of these reasons should not exist. You should not have to become a manager to have career progression, pay equity, access to information, to be included in the decision-making process, even to set company strategy (to an extent congruent with your level, impact, role, tenure, etc)..

Everybody can’t weigh in on everything, obviously, but technical leaders are the best people to make technical decisions, not managers. In healthy orgs, managers work to push those powers outwards to the people closest to the work rather than hoarding it for themselves.

Legitimate reasons for being interested in management.

If you claw away all the org fuckery that forces so many people who care deeply about their work and coworkers into management, there is only one honest reason left for why anyone should try management.

✨Because you feel like it.✨

Because you’re curious. Because there’s an opportunity, maybe, or it seems interesting. Because why not? It’s as good a reason as any. Why do you learn a new framework, a new language, why do you write about your work, why do you pick up any new skill or new role? Why do any of it?

We are not rational beings. First comes emotional urge (“I want that”), then comes rationalization (“because, uh, I love people?”). That’s just how our brains work. You don’t really have to defend or justify it any further.

In reality …

I have observed that many people (especially early-career) are semi-obsessed with getting in to management.

There are many reasons for this. In most places, it is still regarded as a promotion, not a support role / change of career. With high achievers, all you have to do is plunk a ladder next to them to make them want to climb it. Many people feel a lack of agency and lack of autonomy in their role, and they think becoming a manager will solve all their problems.

The swiftest cure for this delusion is  … actually becoming a manager.

Management is a role where you are granted certain institutional powers, at the expense of other powers, freedoms and benefits. Many people who try management figure out pretty quickly that it’s not for them. Formal powers are, in many ways, the weakest powers of them all.

Which is why I think anybody who is interested in management should get a shot at it. Let’s demystify the role, strip it of its mystique and glamour, and make it what it should be: a role of service to others not dominance over others; staffed by people who genuinely take joy in that people side of sociotechnical problem solving.

 

charity

 

 

The Official, Authorized List Of Legitimate Reasons For Deciding to Become a Manager

Trolley Problems as a Service

mipsytipsy, , , , 6 Comments

Consider:

  • Is it ethical to discriminate in whom you will sell to as a business?  What would you do if you found out that the work you do every day was being used to target and kill migrants at the border? 
  • Is it ethical or defensible to pay two people doing the same job different salaries if they live in different locations and have a different cost of living?  What if paying everyone the same rate means you are outcompeted by those who peg salaries to local rates, because they can vastly out-hire you?
  • You’re at the crowded hotel bar after a company-sponsored event, and one of your most valued customers begins loudly venting opinions about minorities in tech that you find alarming and abhorrent.  What responsibility do you have, if any?  How should you react?
  • If we were close to running out of money in the hypothetical future, should we do layoffs or offer pay cuts?

It’s not getting any simpler to live in this world, is it?  💔

Ethical problems are hard.  Even the ones that seem straightforward on the face of them get stickier the closer you look at them.  There are more stakeholders, more caveats, more cautionary tales, more unintended consequences than you can generally see at face value. It’s like fractal hardness, and anyone who thinks it’s easy is fooling themselves.

We’ve been running an experiment at Honeycomb for the past 6 months, where we talk through hypothetical ethical questions like these once a month. Sometimes they are ripped from the headlines, sometimes they are whatever I can invent the night before. I try to send them around in advance. The entire company is invited.**

Honeycomb is not a democracy, nor do I think that would be an effective way to run a company, any more than I think we should design our SDKs by committee or give everyone an equal vote on design mocks.

But I do think that we have a responsibility to act in the best interests of our stakeholders, to the best of our abilities, and to represent our employees. And that means we need to know where the team stands.

That’s one reason. Another is that people make the worst possible decisions when they’re taken off guard, when they are in an unfamiliar situation (and often panicking). Talking through a bunch of nightmare scenarios is a way for us to exercise these decision-making muscles while the stakes are low. We all get to experience what it’s like to hear a problem, have a kneejerk reaction .. then peeling back the onion to reveal layer after layer of dismaying complexities that muddy our snap certainties.

Honeycomb is a pretty transparent company; we believe that companies are created every day by the people who show up to labor together, so those people have a right to know most things. But it’s not always possible or ethically desirable to share all the gritty details that factor into a decision. My hope is that these practice runs help amplify employees’ voices, help them understand the way we approach big decisions, and help everyone make better decisions — and trust each other’s decisions — when things move fast and times get hard.

(Plus, these ethical puzzles are astonishingly fun to work through together. I highly recommend you borrow this idea and try it out at your own company.)

cheers, and please let me know if you do try it ☺️

charity

** We used to limit attendance to the first 6 people to show up, to try and keep the discussion more authentic and less performative. We recently relaxed this rule since it doesn’t seem to matter, peacocking hasn’t really been an issue.

Trolley Problems as a Service

Observability is a Many-Splendored Definition

mipsytipsy, , , , , 12 Comments

Last weekend, @swyx posted a great little primer to instrumentation titled “Observability Tools in JavaScript”.  A friend sent me the link and suggested that I might want to respond and clarify some things about observability, so I did, and we had a great conversation!  Here is a lightly edited transcript of my reply tweet storm.

First of all, confusion over terminology is understandable, because there are some big players out there actively trying to confuse you!  Big Monitoring is indeed actively trying to define observability down to “metrics, logs and traces”.  I guess they have been paying attention to the interest heating up around observability, and well… they have metrics, logs, and tracing tools to sell?  So they have hopped on the bandwagon with some undeniable zeal.

But metrics, logs and traces are just data types.  Which actually has nothing to do with observability.  Let me explain the difference, and why I think you should care about this.

“Observability? I do not think it means what you think it means.”

Observability is a borrowed term from mechanical engineering/control theory.  It means, paraphrasing: “can you understand what is happening inside the system — can you understand ANY internal state the system may get itself into, simply by asking questions from the outside?”  We can apply this concept to software in interesting ways, and we may end up using some data types, but that’s putting the cart before the horse.

It’s a bit like saying that “database replication means structs, longints and elegantly diagrammed English sentences.”  Er, no.. yes.. missing the point much?

This is such a reliable bait and switch that any time you hear someone talking about “metrics, logs and traces”, you can be pretty damn sure there’s no actual observability going on.  If there were, they’d be talking about that instead — it’s far more interesting!  If there isn’t, they fall back to talking about whatever legacy products they do have, and that typically means, you guessed it: metrics, logs and traces.

❌ Metrics

Metrics in particular are actually quite hostile to observability.  They are usually pre-aggregated, which means you are stuck with whatever questions you defined in advance, and even when they aren’t pre-aggregated they permanently discard the connective tissue of the request at write time, which destroys your ability to correlate issues across requests or track down any individual requests or drill down into a set of results — FOREVER.

Which doesn’t mean metrics aren’t useful!  They are useful for many things!  But they are useful for things like static dashboards, trend analysis over time, or monitoring that a dimension stays within defined thresholds.  Not observability.  (Liz would interrupt here and say that Google’s observability story involves metrics, and that is true — metrics with exemplars.  But this type of solution is not available outside Google as far as we know..)

❌ Logs

Ditto logs.  When I say “logs”, you think “unstructured strings, written out to disk haphazardly during execution, “many” log lines per request, probably contains 1-5 dimensions of useful data per log line, probably has a schema and some defined indexes for searching.”  Logs are at their best when you know exactly what to look for, then you can go and find it.

Again, these connotations and assumptions are the opposite of observability’s requirements, which deals with highly structured data only.  It is usually generated by instrumentation deep within the app, generally not buffered to local disk, issues a single event per request per service, is schemaless and indexless (or inferred schemas and autoindexed), and typically containing hundreds of dimensions per event.

❓ Traces

Traces?  Now we’re getting closer.  Tracing IS a big part of observability, but tracing just means visualizing events in order by time.  It certainly isn’t and shouldn’t be a standalone product, that just creates unnecessary friction and distance.  Hrmm … so what IS observability again, as applied to the software domain??

As a reminder, observability applied to software systems means having the ability to ask any question of your systems — understand any user’s behavior or subjective experience — without having to predict that question, behavior or experience in advance.

Observability is about unknown-unknowns.

At its core, observability is about these unknown-unknowns.

Plenty of tools are terrific at helping you ask the questions you could predict wanting to ask in advance.  That’s the easy part.  “What’s the error rate?”  “What is the 99th percentile latency for each service?”  “How many READ queries are taking longer than 30 seconds?”

  • Monitoring tools like DataDog do this — you predefine some checks, then set thresholds that mean ERROR/WARN/OK.
  • Logging tools like Splunk will slurp in any stream of log data, then let you index on questions you want to ask efficiently.
  • APM tools auto-instrument your code and generate lots of useful graphs and lists like “10 slowest endpoints”.

But if you *can’t* predict all the questions you’ll need to ask in advance, or if you *don’t* know what you’re looking for, then you’re in o11y territory.

  • This can happen for infrastructure reasons — microservices, containerization, polyglot storage strategies can result in a combinatorial explosion of components all talking to each other, such that you can’t usefully pre-generate graphs for every combination that can possibly degrade.
  • And it can happen — has already happened — to most of us for product reasons, as you’ll know if you’ve ever tried to figure out why a spike of errors was being caused by users on ios11 using a particular language pack but only in three countries, and only when the request hit the image export microservice running build_id 789782 if the user’s last name starts with “MC” and they then try to click on a particular button which then issues a db request using the wrong cache key for that shard.

Gathering the right data, then exploring the data.

Observability starts with gathering the data at the right level of abstraction, organized around the request path, such that you can slice and dice and group and  look for patterns and cross-correlations in the requests.

To do this, we need to stop firing off metrics and log lines willynilly and be more disciplined.  We need to issue one single arbitrarily-wide event per service per request, and it must contain the *full context* of that request. EVERYTHING you know about it, anything you did in it, all the parameters passed into it, etc.  Anything that might someday help you find and identify that request.

Then, when the request is poised to exit or error the service, you ship that blob off to your o11y store in one very wide structured event per request per service.

In order to deliver observability, your tool also needs to support high cardinality and high dimensionality.  Briefly, cardinality refers to the number of unique items in a set, and dimensionality means how many adjectives can describe your event.  If you want to read more, here is an overview of the space, and more technical requirements for observability

You REQUIRE the ability to chain and filter as many dimensions as you want with infinitely high cardinality for each one if you’re going to be able to ask arbitrary questions about your unknown unknowns.  This functionality is table stakes.  It is non negotiable.  And you cannot get it from any metrics or logs tool on the market today.

Why this matters.

Alright, this is getting pretty long. Let me tell you why I care so much, and why I want people like you specifically (referring to frontend engineers and folks earlier in their careers) to grok what’s at stake in the observability term wars.

We are way behind where we ought to be as an industry. We are shipping code we don’t understand, to systems we have never understood. Some poor sap is on call for this mess, and it’s killing them, which makes the software engineers averse to owning their own code in prod.  What a nightmare.

Meanwhile developers readily admit they waste >40% of their day doing bullshit that doesn’t move the business forward.  In large part this is because they are flying blind, just stabbing around in the dark.

We all just accept this.  We shrug and say well that’s just what it’s like, working on software is just a shit salad with a side of frustration, it’s just the way it is.

But it is fucking not.  It is un fucking necessary.  If you instrument your code, watch it deploy, then ask “is it doing what I expect, does anything else look weird” as a habit?  You can build a system that is both understandable and well-understood.  If you can see what you’re doing, and catch errors swiftly, it never has to become a shitty hairball in the first place.  That is a choice.

🌟 But observability in the original technical sense is a necessary prerequisite to this better world. 🌟

If you can’t break down by high cardinality dimensions like build ids, unique ids, requests, and function names and variables, if you cannot explore and swiftly skim through new questions on the fly, then you cannot inspect the intersection of (your code + production + users) with the specificity required to associate specific changes with specific behaviors.  You can’t look where you are going.

Observability as I define it is like taking off the blindfold and turning on the light before you take a swing at the pinata.  It is necessary, although not sufficient alone, to dramatically improve the way you build software.  Observability as they define it gets you to … exactly where you already are.  Which of these is a good use of a new technical term?

 

Do better.

And honestly, it’s the next generation who are best poised to learn the new ways and take advantage of them. Observability is far, far easier than the old ways and workarounds … but only if you don’t have decades of scar tissue and old habits to unlearn.

The less time you’ve spent using monitoring tools and ops workarounds, the easier it will be to embrace a new and better way of building and shipping well-crafted code.

Observability matters.  You should care about it.  And vendors need to stop trying to confuse people into buying the same old bullshit tools by smooshing them together and slapping on a new label.  Exactly how long do they expect to fool people for, anyway?

Observability is a Many-Splendored Definition

A Manager’s Bill of Responsibilities (and Rights)

mipsytipsy, , , 7 Comments

Over a year and a half ago, I wrote up a post about the rights and responsibilities due any engineer at Honeycomb.  At the time we were in the middle of a growth spurt, had just hired several new engineers, and I was in the process of turning over day-to-day engmeme2engineering management over to Emily.  Writing things down helped me codify what I actually cared about, and helped keep us true to our principles as we grew.

Tacked on to the end of the post was a list of manager responsibilities, almost as an afterthought. Many people protested, “don’t managers get any rights??” (and naturally I snapped “NO!  hahahahahha”)

I always intended to circle back and write a followup post with the rights and responsibilities for managers.  But it wasn’t til recently, as we are gearing up for another hiring spurt and have expanded our managerial ranks, that it really felt like its time had come.

The time has come, the time is now, as marvin k. mooney once said.  Added the bill of rights, and updated and expanded the list of responsibilities.  Thanks Emily Nakashima for co-writing it with me.

 

Manager’s Bill of Rights

  1. You shall receive honest, courageous, timely feedback about yourself and your team, from your reports, your peers, and your leaders.  (No one is exempt from feeding the hungry hungry feedback hippo!  NOO ONNEEEE!)  🦛🦛🦛🦛🦛🦛🦛
  2. Management will be treated with the same respect and importance as individual work.  reviewmeme
  3. You have the final say over hiring, firing, and leveling decisions for your team.  It is expected that you solicit feedback from your team and peers and drive consensus where possible.  But in the end, the say is yours.
  4. Management can be draining, difficult work, even at places that do it well.  You will get tactical, strategic, and emotional support from other managers.
  5. You cannot take care of others unless you first practice self-care.  You damn well better take vacations.  (Real ones.)
  6. You have the right to personal development, career progression, and professional support.  We will retain a leadership coach for you.
  7. You do not have to be a manager if you do not want to.  No one will ever pressure you.

Manager’s Responsibilities

  • Recruit and hire and train your team. Foster a sense of solidarity and “teaminess” as well as real emotional safety.
  • Cultivate an inclusive culture and redistribute opportunity.  Fuck a pedigree.  Resist monoculture.
  • Care for the people on your team. Support them in their career trajectory, personal goals, work/life balance, and inter- and intra-team dynamics.
  • Keep an eye out for people on other teams who aren’t getting the support they need, and work with your leadership and manager peers to fix the situation. catplays
  • Give feedback early and often. Receive feedback gracefully. Always say the hard things, but say them with love.
  • Move us relentlessly forward, staying alert for rabbit-holing and work that doesn’t contribute to our goals. Ensure redundancy/coverage of critical areas.
  • Own the planning process for your team, be accountable for the goals you set. Allocate resources by communicating priorities and requesting support. Add focus or urgency where needed.
  • Own your time and attention. Be accessible. Actively manage your calendar. Try not to make your emotions everyone else’s problems (but do lean on your own manager and your peers for support).
  • Make your own personal growth and self-care a priority. Model the values and traits we want employees to pattern themselves after.
  • Stay vulnerable.

(Easier said than done, huh?)

<3 charity

Screen Shot 2019-10-30 at 8.04.07 AM

A Manager’s Bill of Responsibilities (and Rights)