I’ve been at my current job for three years, and I am suddenly, accidentally, the most senior engineer on the team. I spend my days handling things like bootcamps, mentoring, architecture, and helping other engineers carve off meaningful work. This has taken a huge toll on the kind of work I want to do as an IC. I still enjoy writing and shipping features, and I am not a manager, but now I feel like I spend my days conducting meetings, interviewing, and unblocking others constantly instead of writing code myself.
What should I do? How can I deal with this situation in an effective manner? How can I keep from getting burned out on zoom? How can I reclaim more of my time to write code for myself, without sacrificing my influence? Should I get a new job? I have thought about going out and getting a new job, but I really like having a say at a high level. Here I get looped into all of the most important decisions and meetings. If I get a new job, how can I avoid starting over at the bottom of the heap and just taking assignments from other people? P.S., this is my first job.
Get a new job.
Yes, you will reset your seniority and have to earn it all over again. Yes, it will be uncomfortable and your ego will be cranky over it. Yes, you will be at the bottom of the heap and take assignments from other people for a while. Yes, you should do it anyway.
What you are experiencing now is the alluring comfort of premature seniority. You’re the smartest kid in the room, you know every corner of the system inside and out, you win every argument and anticipate every objection and you are part of every decision and you feel so deeply, pleasingly needed by the people around you.
It’s a trap.
Get the fuck out of there.
There is a world of distance between being expert in this system and being an actual expert in your chosen craft. The second is seniority; the first is merely .. familiarity
Deep down I think you know this, and feel a gnawing insecurity over your position; why else would you have emailed me? You were right. Treasure that uneasy feeling in your gut, that discomfort in the face of supreme comfortable-ness. It will lead you to a long and prosperous career as an engineer if you learn to trust it.
Think of every job like an escalator — a 50-foot high escalator that takes about two years to ride to the top. But once you’ve summited, you stall out. You can either stay and wander on that floor, or you can step to the left and pick another escalator and ride it up another 50 feet. And another.
In my mind, someone becomes a real senior engineer after they’ve done this about three times. 2-3 teams, stacks, languages, and roles, over a 5-8 year period, and then they’re solidly baked. There are insights you can derive from having seen problems solved in a few different ways that you can’t with only a single point of reference.
You don’t become a senior engineer at the 50-foot ascent, no matter how thoroughly you know the landscape. You become a senior engineer somewhere well over 100 feet, with a couple of lane changes under your belt.
The act of learning a new language and/or stack is itself an important skill. Experiencing how different orgs ship code in vastly different ways is how you internalize that there’s no one blessed path, only different sets of tradeoffs, and how you learn to reason about those tradeoffs.
And it is good for us to start over with beginner eyes. It’s humbling, it’s clarifying, it’s a cleanse for the soul. If you get too attached to feeling senior, to feeling necessary, you will undervalue the virtues of fresh eyes and questioning, of influence without authority. It is good for you to practice uncertainty and influencing others without the cheat codes of deep familiarity.
Nobody wants to work with seniors who clutch their authority with a white knuckled grip. We want to work with those who wear it lightly, who remember what it was like in our shoes.
Ultimately, this is a strong argument for building our teams behind a Rawlsian veil of ignorance concerning our own place in the pecking order. Starting fresh yourself will help you build teams where it is not miserable to be a beginner, where beginners’ contributions are recognized, where even beginners do not simply “take orders”, as you said. Because literally nobody wants that, including the beginners you are working with on your teams today.
After you have gotten a new job or two, and proven to yourself that you can level up again and master new stacks and technologies, that fretful inner voice questioning whether you deserve the respect you receive or not will calm down. You will have proven to yourself that your success wasn’t just a one-off, that you can be dropped into any situation, learn the local ropes and succeed. You will be a senior engineer.
There are few engineering topics that provoke as much heated commentary as oncall. Everybody has a strong opinion. So let me say straight up that there are few if any absolutes when it comes to doing this well; context is everything. What’s appropriate for a startup may not suit a larger team. Rules are made to be broken.
That said, I do have some feelings on the matter. Especially when it comes to the compact between engineering and management. Which is simply this:
It is engineering’s responsibility to be on call and own their code. It is management’s responsibility to make sure that on call does not suck. This is a handshake, it goes both ways, and if you do not hold up your end they should quit and leave you.
As for engineers who write code for 24×7 highly available services, it is a core part of their job is to support those services in production. (There are plenty of software jobs that do not involve building highly available services, for those who are offended by this.) Tossing it off to ops after tests pass is nothing but a thinly veiled form of engineering classism, and you can’t build high-performing systems by breaking up your feedback loops this way.
Someone needs to be responsible for your services in the off-hours. This cannot be an afterthought; it should play a prominent role in your hiring, team structure, and compensation decisions from the very start. These are decisions that define who you are and what you value as a team.
Some advice on how to organize your on call efforts, in no particular order.
It is easier to keep yourself from falling into an operational pit of doom than it is to claw your way out of one. Make good operational hygiene a priority from the start. Value good, clean, high-level abstractions that allow you to delegate large swaths of your infrastructure and operational burden to third parties who can do it better than you — serverless, AWS, *aaS, etc. Don’t fall into the trap of disrespecting operations engineering labor, it’s the only thing that can save you.
Invest in good release and deploy tooling. Make this part of your engineering roadmap, not something you find in the couch cushions. Get code into production within minutes after merging, and watch how many of your nightmares melt away or never happen.
Invest in good instrumentation and observability. Impress upon your engineers that their job is not done when tests pass; it is not done until they have watched users using their code in production. Promote an ownership mentality over the full software life cycle. This is how dev.to did it.
Construct your feedback loops thoughtfully. Try to alert the person who made the broken change directly. Never send an alert to someone who isn’t fully equipped and empowered to fix it.
When an engineer is on call, they are not responsible for normal project work — period. That time is sacred and devoted to fixing things, building tooling, and creating guard-rails to protect people from themselves. If nothing is on fire, the engineer can take the opportunity to fix whatever has been annoying them. Allow for plenty of agency and following one’s curiosity, wherever it may lead, and it will be a special treat.
Closely track how often your team gets alerted. Take ANY out-of-hours-alert seriously, and prioritize the work to fix it. Night time pages are heart attacks, not diabetes.
Consider joining the on call rotation yourself! If nothing else, generously pinch hit and be an eager and enthusiastic backup on the regular.
Reliability work and technical debt are not secondary to product work. Budget them into your roadmap, right alongside your features and fixes. Don’t plan so tightly that you have no flex for the unexpected. Don’t be afraid to push back on product and don’t neglect to sell it to your own bosses. People’s lives are in your hands; this is what you get paid to do.
Consider making after-hours on call fully-elective. Why not? What is keeping you from it? Fix those things. This is how Intercom did it.
Depending on your stage and available resources, consider compensating for it. This doesn’t have to be cash, it could be a Friday off the week after every on call rotation. The more established and funded a company you are, the more likely you should do this in order to surface the right incentives up the org chart.
Once you’ve dug yourself out of firefighting mode, invest in SLOs (Service Level Objectives). SLOs and observability are the mature way to get out of reactive mode and plan your engineering work based on tradeoffs and user impact.
I believe it is thoroughly possible to construct an on call rotation that is 100% opt-in, a badge of pride and accomplishment, something that brings meaning and mastery to people’s engineering roles and ties them emotionally to their users. I believe that being on call is something that you can genuinely look forward to.
But every single company is a unique complex sociotechnical snowflake. Flipping the script on whether on call is a burden or a blessing will require a unique solution, crafted to meet your specific needs and drawing on your specific history. It will require tinkering. It will take maintenance.
Above all: ✨RAISE YOUR STANDARDS✨ for what you expect from yourselves. Your greatest enemy is how easily you accept the status quo, and then make up excuses for why it is necessarily this way. You can do better. I know you can.
treat every alarm like a heart attack. _fix_ the motherfucker.
i do not care if this causes product development to screech to a halt. amortize it over a slightly longer period of time and it will more than pay for itself. https://t.co/JSck2u86ff
There is lots and lots of prior art out there when it comes to making on call work for you, and you should research it deeply. Watch some talks, read some pieces, talk to some people. But then you’ll have to strike out on your own and try something. Cargo-culting someone else’s solution is always the wrong answer.
Any asshole can write some code; owning and tending complex systems for the long run is the hard part. How you choose to shoulder this burden will be a deep reflection of your values and who you are as a team.
And if your on call experience is mandatory and severely life-impacting, and if you don’t take this dead seriously and fix it ASAP? I hope your team will leave you, and go find a place that truly values their time and sleep.
It’s a question that gets asked a lot, in job interviews, 1x1s, and plain old casual conversation. I ask this question a lot, and I am often frustrated (or bored) by the answers I hear back.
Most of them can be bucketed in one of three ways:
The pious. “I just really, really love helping other people achieve their goals.”
The pleasers. the ones who answer, then pause uncertainly: “Is that what you’re looking for?”
The sheepish. “I probably shouldn’t say this, but..” (followed by something very close to real honesty)
People are rarely inclined to divulge the range and depth of their reasons for going into management. And why should they? We are constantly being lectured about what the RIGHT reasons for going into management are, with aspersions cast upon anyone who dares enter the profession for any reasons that are not completely selfless.
“I LOVE mentoring.” “I wanted to protect my team.” “I’m motivated by people problems.” “I just really love helping people grow.”
I’m not saying that everybody who says these words is lying, but I would be surprised if it was the entire story. People make career moves for a complex mix of altruism and self-interest.
It’s socially acceptable to cop to the selfless reasons. But what about the rest? Like “I wanted more money”? “I wanted career progression and couldn’t get any as an IC”? What about “I couldn’t get a seat at the table as an engineer”, “I was tired of being left out of important decisions”, or “My reporting chain was opaque and kept fucking up, and I figured I couldn’t do any worse than those bozos”?
Now we’re talking.
Most people become managers to compensate for org fuckery.
In my experience, most engineers become managers primarily due to organizational dysfunction. When you become a manager you acquire certain institutional powers, and you can use those powers to change the thing that makes you miserable.
It’s a hack. A gnarly one. And like most hacks, it kinda works.
For example, say it pisses you off to be left out of decisions. So you become a manager, and then you can either a) use your power and access to push for including engineers in the decision-making process, or at very least b) you personally will no longer left out.
In a healthy org, I would argue that most of these reasons should not exist. You should not have to become a manager to have career progression, pay equity, access to information, to be included in the decision-making process, even to set company strategy (to an extent congruent with your level, impact, role, tenure, etc)..
Everybody can’t weigh in on everything, obviously, but technical leaders are the best people to make technical decisions, not managers. In healthy orgs, managers work to push those powers outwards to the people closest to the work rather than hoarding it for themselves.
Legitimate reasons for being interested in management.
If you claw away all the org fuckery that forces so many people who care deeply about their work and coworkers into management, there is only one honest reason left for why anyone should try management.
✨Because you feel like it.✨
Because you’re curious. Because there’s an opportunity, maybe, or it seems interesting. Because why not? It’s as good a reason as any. Why do you learn a new framework, a new language, why do you write about your work, why do you pick up any new skill or new role? Why do any of it?
We are not rational beings. First comes emotional urge (“I want that”), then comes rationalization (“because, uh, I love people?”). That’s just how our brains work. You don’t really have to defend or justify it any further.
In reality …
I have observed that many people (especially early-career) are semi-obsessed with getting in to management.
There are many reasons for this. In most places, it is still regarded as a promotion, not a support role / change of career. With high achievers, all you have to do is plunk a ladder next to them to make them want to climb it. Many people feel a lack of agency and lack of autonomy in their role, and they think becoming a manager will solve all their problems.
The swiftest cure for this delusion is … actually becoming a manager.
Management is a role where you are granted certain institutional powers, at the expense of other powers, freedoms and benefits. Many people who try management figure out pretty quickly that it’s not for them. Formal powers are, in many ways, the weakest powers of them all.
Which is why I think anybody who is interested in management should get a shot at it. Let’s demystify the role, strip it of its mystique and glamour, and make it what it should be: a role of service to others not dominance over others; staffed by people who genuinely take joy in that people side of sociotechnical problem solving.
Is it ethical to discriminate in whom you will sell to as a business? What would you do if you found out that the work you do every day was being used to target and kill migrants at the border?
Is it ethical or defensible to pay two people doing the same job different salaries if they live in different locations and have a different cost of living? What if paying everyone the same rate means you are outcompeted by those who peg salaries to local rates, because they can vastly out-hire you?
You’re at the crowded hotel bar after a company-sponsored event, and one of your most valued customers begins loudly venting opinions about minorities in tech that you find alarming and abhorrent. What responsibility do you have, if any? How should you react?
If we were close to running out of money in the hypothetical future, should we do layoffs or offer pay cuts?
It’s not getting any simpler to live in this world, is it? 💔
Ethical problems are hard. Even the ones that seem straightforward on the face of them get stickier the closer you look at them. There are more stakeholders, more caveats, more cautionary tales, more unintended consequences than you can generally see at face value. It’s like fractal hardness, and anyone who thinks it’s easy is fooling themselves.
We’ve been running an experiment at Honeycomb for the past 6 months, where we talk through hypothetical ethical questions like these once a month. Sometimes they are ripped from the headlines, sometimes they are whatever I can invent the night before. I try to send them around in advance. The entire company is invited.**
Honeycomb is not a democracy, nor do I think that would be an effective way to run a company, any more than I think we should design our SDKs by committee or give everyone an equal vote on design mocks.
But I do think that we have a responsibility to act in the best interests of our stakeholders, to the best of our abilities, and to represent our employees. And that means we need to know where the team stands.
That’s one reason. Another is that people make the worst possible decisions when they’re taken off guard, when they are in an unfamiliar situation (and often panicking). Talking through a bunch of nightmare scenarios is a way for us to exercise these decision-making muscles while the stakes are low. We all get to experience what it’s like to hear a problem, have a kneejerk reaction .. then peeling back the onion to reveal layer after layer of dismaying complexities that muddy our snap certainties.
Honeycomb is a pretty transparent company; we believe that companies are created every day by the people who show up to labor together, so those people have a right to know most things. But it’s not always possible or ethically desirable to share all the gritty details that factor into a decision. My hope is that these practice runs help amplify employees’ voices, help them understand the way we approach big decisions, and help everyone make better decisions — and trust each other’s decisions — when things move fast and times get hard.
(Plus, these ethical puzzles are astonishingly fun to work through together. I highly recommend you borrow this idea and try it out at your own company.)
cheers, and please let me know if you do try it ☺️
** We used to limit attendance to the first 6 people to show up, to try and keep the discussion more authentic and less performative. We recently relaxed this rule since it doesn’t seem to matter, peacocking hasn’t really been an issue.
First of all, confusion over terminology is understandable, because there are some big players out there actively trying to confuse you! Big Monitoring is indeed actively trying to define observability down to “metrics, logs and traces”. I guess they have been paying attention to the interest heating up around observability, and well… they have metrics, logs, and tracing tools to sell? So they have hopped on the bandwagon with some undeniable zeal.
But metrics, logs and traces are just data types. Which actually has nothing to do with observability. Let me explain the difference, and why I think you should care about this.
“Observability? I do not think it means what you think it means.”
Observability is a borrowed term from mechanical engineering/control theory. It means, paraphrasing: “can you understand what is happening inside the system — can you understand ANY internal state the system may get itself into, simply by asking questions from the outside?” We can apply this concept to software in interesting ways, and we may end up using some data types, but that’s putting the cart before the horse.
It’s a bit like saying that “database replication means structs, longints and elegantly diagrammed English sentences.” Er, no.. yes.. missing the point much?
This is such a reliable bait and switch that any time you hear someone talking about “metrics, logs and traces”, you can be pretty damn sure there’s no actual observability going on. If there were, they’d be talking about that instead — it’s far more interesting! If there isn’t, they fall back to talking about whatever legacy products they do have, and that typically means, you guessed it: metrics, logs and traces.
Metrics in particular are actually quite hostile to observability. They are usually pre-aggregated, which means you are stuck with whatever questions you defined in advance, and even when they aren’t pre-aggregated they permanently discard the connective tissue of the request at write time, which destroys your ability to correlate issues across requests or track down any individual requests or drill down into a set of results — FOREVER.
Which doesn’t mean metrics aren’t useful! They are useful for many things! But they are useful for things like static dashboards, trend analysis over time, or monitoring that a dimension stays within defined thresholds. Not observability. (Liz would interrupt here and say that Google’s observability story involves metrics, and that is true — metrics with exemplars. But this type of solution is not available outside Google as far as we know..)
Ditto logs. When I say “logs”, you think “unstructured strings, written out to disk haphazardly during execution, “many” log lines per request, probably contains 1-5 dimensions of useful data per log line, probably has a schema and some defined indexes for searching.” Logs are at their best when you know exactly what to look for, then you can go and find it.
Again, these connotations and assumptions are the opposite of observability’s requirements, which deals with highly structured data only. It is usually generated by instrumentation deep within the app, generally not buffered to local disk, issues a single event per request per service, is schemaless and indexless (or inferred schemas and autoindexed), and typically containing hundreds of dimensions per event.
Traces? Now we’re getting closer. Tracing IS a big part of observability, but tracing just means visualizing events in order by time. It certainly isn’t and shouldn’t be a standalone product, that just creates unnecessary friction and distance. Hrmm … so what IS observability again, as applied to the software domain??
As a reminder, observability applied to software systems means having the ability to ask any question of your systems — understand any user’s behavior or subjective experience — without having to predict that question, behavior or experience in advance.
Observability is about unknown-unknowns.
At its core, observability is about these unknown-unknowns.
Plenty of tools are terrific at helping you ask the questions you could predict wanting to ask in advance. That’s the easy part. “What’s the error rate?” “What is the 99th percentile latency for each service?” “How many READ queries are taking longer than 30 seconds?”
Monitoring tools like DataDog do this — you predefine some checks, then set thresholds that mean ERROR/WARN/OK.
Logging tools like Splunk will slurp in any stream of log data, then let you index on questions you want to ask efficiently.
APM tools auto-instrument your code and generate lots of useful graphs and lists like “10 slowest endpoints”.
But if you *can’t* predict all the questions you’ll need to ask in advance, or if you *don’t* know what you’re looking for, then you’re in o11y territory.
This can happen for infrastructure reasons — microservices, containerization, polyglot storage strategies can result in a combinatorial explosion of components all talking to each other, such that you can’t usefully pre-generate graphs for every combination that can possibly degrade.
And it can happen — has already happened — to most of us for product reasons, as you’ll know if you’ve ever tried to figure out why a spike of errors was being caused by users on ios11 using a particular language pack but only in three countries, and only when the request hit the image export microservice running build_id 789782 if the user’s last name starts with “MC” and they then try to click on a particular button which then issues a db request using the wrong cache key for that shard.
Gathering the right data, then exploring the data.
Observability starts with gathering the data at the right level of abstraction, organized around the request path, such that you can slice and dice and group and look for patterns and cross-correlations in the requests.
To do this, we need to stop firing off metrics and log lines willynilly and be more disciplined. We need to issue one single arbitrarily-wide event per service per request, and it must contain the *full context* of that request. EVERYTHING you know about it, anything you did in it, all the parameters passed into it, etc. Anything that might someday help you find and identify that request.
Then, when the request is poised to exit or error the service, you ship that blob off to your o11y store in one very wide structured event per request per service.
In order to deliver observability, your tool also needs to support high cardinality and high dimensionality. Briefly, cardinality refers to the number of unique items in a set, and dimensionality means how many adjectives can describe your event. If you want to read more, here is an overview of the space, and more technical requirements for observability
You REQUIRE the ability to chain and filter as many dimensions as you want with infinitely high cardinality for each one if you’re going to be able to ask arbitrary questions about your unknown unknowns. This functionality is table stakes. It is non negotiable. And you cannot get it from any metrics or logs tool on the market today.
Why this matters.
Alright, this is getting pretty long. Let me tell you why I care so much, and why I want people like you specifically (referring to frontend engineers and folks earlier in their careers) to grok what’s at stake in the observability term wars.
We are way behind where we ought to be as an industry. We are shipping code we don’t understand, to systems we have never understood. Some poor sap is on call for this mess, and it’s killing them, which makes the software engineers averse to owning their own code in prod. What a nightmare.
Meanwhile developers readily admit they waste >40% of their day doing bullshit that doesn’t move the business forward. In large part this is because they are flying blind, just stabbing around in the dark.
We all just accept this. We shrug and say well that’s just what it’s like, working on software is just a shit salad with a side of frustration, it’s just the way it is.
But it is fucking not. It is un fucking necessary. If you instrument your code, watch it deploy, then ask “is it doing what I expect, does anything else look weird” as a habit? You can build a system that is both understandable and well-understood. If you can see what you’re doing, and catch errors swiftly, it never has to become a shitty hairball in the first place. That is a choice.
🌟 Butobservability in the original technical sense is a necessary prerequisite to this better world. 🌟
If you can’t break down by high cardinality dimensions like build ids, unique ids, requests, and function names and variables, if you cannot explore and swiftly skim through new questions on the fly, then you cannot inspect the intersection of (your code + production + users) with the specificity required to associate specific changes with specific behaviors. You can’t look where you are going.
Observability as I define it is like taking off the blindfold and turning on the light before you take a swing at the pinata. It is necessary, although not sufficient alone, to dramatically improve the way you build software. Observability as they define it gets you to … exactly where you already are. Which of these is a good use of a new technical term?
And honestly, it’s the next generation who are best poised to learn the new ways and take advantage of them. Observability is far, far easier than the old ways and workarounds … but only if you don’t have decades of scar tissue and old habits to unlearn.
The less time you’ve spent using monitoring tools and ops workarounds, the easier it will be to embrace a new and better way of building and shipping well-crafted code.
Observability matters. You should care about it. And vendors need to stop trying to confuse people into buying the same old bullshit tools by smooshing them together and slapping on a new label. Exactly how long do they expect to fool people for, anyway?
Over a year and a half ago, I wrote up a post about the rights and responsibilities due any engineer at Honeycomb. At the time we were in the middle of a growth spurt, had just hired several new engineers, and I was in the process of turning over day-to-day engineering management over to Emily. Writing things down helped me codify what I actually cared about, and helped keep us true to our principles as we grew.
Tacked on to the end of the post was a list of manager responsibilities, almost as an afterthought. Many people protested, “don’t managers get any rights??” (and naturally I snapped “NO! hahahahahha”)
I always intended to circle back and write a followup post with the rights and responsibilities for managers. But it wasn’t til recently, as we are gearing up for another hiring spurt and have expanded our managerial ranks, that it really felt like its time had come.
The time has come, the time is now, as marvin k. mooney once said. Added the bill of rights, and updated and expanded the list of responsibilities. Thanks Emily Nakashima for co-writing it with me.
Manager’s Bill of Rights
You shall receive honest, courageous, timely feedback about yourself and your team, from your reports, your peers, and your leaders. (No one is exempt from feeding the hungry hungry feedback hippo! NOO ONNEEEE!) 🦛🦛🦛🦛🦛🦛🦛
Management will be treated with the same respect and importance as individual work.
You have the final say over hiring, firing, and leveling decisions for your team. It is expected that you solicit feedback from your team and peers and drive consensus where possible. But in the end, the say is yours.
Management can be draining, difficult work, even at places that do it well. You will get tactical, strategic, and emotional support from other managers.
You cannot take care of others unless you first practice self-care. You damn well better take vacations. (Real ones.)
You have the right to personal development, career progression, and professional support. We will retain a leadership coach for you.
You do not have to be a manager if you do not want to. No one will ever pressure you.
Recruit and hire and train your team. Foster a sense of solidarity and “teaminess” as well as real emotional safety.
Cultivate an inclusive culture and redistribute opportunity. Fuck a pedigree. Resist monoculture.
Care for the people on your team. Support them in their career trajectory, personal goals, work/life balance, and inter- and intra-team dynamics.
Keep an eye out for people on other teams who aren’t getting the support they need, and work with your leadership and manager peers to fix the situation.
Give feedback early and often. Receive feedback gracefully. Always say the hard things, but say them with love.
Move us relentlessly forward, staying alert for rabbit-holing and work that doesn’t contribute to our goals. Ensure redundancy/coverage of critical areas.
Own the planning process for your team, be accountable for the goals you set. Allocate resources by communicating priorities and requesting support. Add focus or urgency where needed.
Own your time and attention. Be accessible. Actively manage your calendar. Try not to make your emotions everyone else’s problems (but do lean on your own manager and your peers for support).
Make your own personal growth and self-care a priority. Model the values and traits we want employees to pattern themselves after.
(With 🙏 to Joe Beda, whose brilliant idea for a blog post this was. Thanks for letting me borrow it!)
Interviewing is hard and it sucks.
In theory, it really shouldn’t be. You’re a highly paid professional and your skills are in high demand. This ought to be a meeting between equals to mutually explore what a longer-term relationship might look like. Why take the outcome personally? There are at least as many reasons for you to decide not to join a company as for the company to decide not to hire you, right?
In reality, of course, all the situational cues and incentives line up to make you feel like the whole thing is a referendum on whether or not you personally are Good Enough (smart enough, senior enough, skilled enough, cool enough) to join their fancy club.
People stay at shitty jobs far, far longer than they ought to, just because interviews can be so genuinely crushing to your spirit and sense of self. Even when they aren’t the worst, it can leave a lasting sting when they decline to hire you.
But there is an important asymmetry here. By not hiring someone, I very rarely mean it as a rejection of that person. (Not unless they were, like, mean to the office manager, or directed all their technical questions to the male interviewers.) On the contrary, I generally hold the people we decline to hire — or have had to let go! — in extremely high opinion.
So if someone interviews at Honeycomb, I do not want them to walk away feeling stung, hurt, or bad about themselves. I would like them to walk away feeling good about themselves and our interactions, even if one or both of us are disappointed by the outcome. I want them to feel the same way about themselves as I feel about them, especially since there’s a high likelihood that I may want to work with them in the future.
So here are the real, honest-to-god most common reasons why I don’t hire someone.
If you’ve worked at a Google or Facebook before, you may have a certain mental model of how hiring works. You ask the candidate a bunch of questions, and if they do well enough, you hire them. This could not be more different from early stage startup hiring, which is defined in every way by scarcity.
I only have a few precious slots to fill this year, and every single one of them is tied to one or more key company initiatives or goals, without which we may fail as a company. Emily and I spend hours obsessively discussing what the profile we are looking for is, what the smallest possible set of key strengths and skills that this hire must have, inter-team and intra-team dynamics and what elements are missing or need to be bolstered from the team as it stands. And at the end of the day, there are not nearly as many slots to fill as there are awesome people we’d like to hire. Not even close. Having to choose between several differently wonderful people can be *excruciating*.
No, not that kind. (Yes, we care about cultivating a diverse team and support that goal through our recruiting and hiring processes, but it’s not a factor in our hiring decisions.) I mean your level, stage in your career, educational background, professional background, trajectory, areas of focus and strengths. We are trying to build radical new tools for sociotechnical systems; tools that are friendly, intuitive, and accessible to every engineer (and engineering-adjacent profession) in the world.
How well do you think we’re going to do at our goal if the people building it are all ex-Facebook, ex-MIT senior engineers? If everyone has the exact same reference points and professional training, we will all have the same blind spots. Even if our team looks like a fucking Benetton ad.
3. We are assembling a team, not hiring individuals.
We spend at least as much time hashing out what the subtle needs of the team are right now as talking about the individual candidate. Maybe what we need is a senior candidate who loves mentoring with her whole heart, or a language polyglot who can help unify the look and feel of our integrations across ten different languages and platforms. Or maybe we have plenty of accomplished mentors, but the team is really lacking someone with expertise in query profiling and db tuning, and we expect this to be a big source of pain in the coming year. Maybe we realize we have nobody on the team who is interested in management, and we are definitely going to need someone to grow into or be hired on as a manager a year or two from now.
There is no value judgment or hierarchy attached to any of these skills or particulars. We simply need what we need, and you are who you are.
4. I am not confident that we can make you successful in this role at this time.
We rarely turn people down for purely technical reasons, because technical skills can be learned. But there can be some combination of your skills, past experience, geographical location, time zone, experience with working remotely, etc — that just gives us pause. If we cast forward a year, do we think you are going to be joyfully humming along and enjoying yourself, working more-or-less independently and collaboratively? If we can’t convince ourselves this is true, for whatever reasons, we are unlikely to hire you. (But we would love to talk with you again someday.)
5. The team needs someone operating at a different level.
Don’t assume this always means “you aren’t senior enough”. We have had to turn down people at least as often for being too senior as not senior enough. An organization can only absorb so many principal and senior engineers; there just isn’t enough high-level strategic work to go around. I believe happy, healthy teams are comprised of a range of levels — you need more junior folks asking naive questions that give senior folks the opportunity to explain themselves and catch their dumb mistakes. You need there to be at least one sweet child who is just so completely stoked to build their very first login page.
A team staffed with nothing but extremely senior developers will be a dysfunctional, bored and contentious team where no one is really growing up or being challenged as they should.
6. We don’t have the kind of work you need or want.
The first time we tried hiring junior developers, we ran into this problem hardcore. We simply didn’t have enough entry-level work for them to do. Everything was frustratingly complex and hard for them, so they weren’t able to operate independently, and we couldn’t spare an engineer to pair with them full time.
This also manifests in other ways. Like, lots of SREs and data engineers would LOVE to work at honeycomb. But we don’t have enough ops engineering work or data problems to keep them busy full time. (Well — that’s not precisely true. They could probably keep busy. But it wouldn’t be aligned with our core needs as a business, which makes them premature optimizations we cannot afford.)
7. Communication skills.
We select highly for communication skills. The core of our technical interview involves improving and extending a piece of code, then bringing it in the next day to discuss it with your peers. We believe that if you can explain what you did and why, you can definitely do the work, and the reverse is not necessarily true. We also believe that communication skills are at the foundation of a team’s ability to learn from its mistakes and improve as a unit. We value high-performing teams, therefore we select for those skills.
There are many excellent engineers who are not good communicators, or who do not value communication the way we do, and while we may respect you very much, it’s not a great fit for our team.
8. You don’t actually want to work at a startup.
“I really want to work at a startup. Also the things that are really important to me are: work/life balance, predictability, high salary, gold benefits, stability, working from 10 to 5 on the dot, knowing what i’ll be working on for the next month, not having things change unexpectedly, never being on call, never needing to think or care about work out of hours …”
To be clear, it is not a red flag if you care about work/life balance. We care about that too — who the hell doesn’t? But startups are inherently more chaotic and unpredictable, and roles are more fluid and dynamic, and I want to make sure your expectations are aligned with reality.
9. You just want to work for women.
I hate it when I’m interviewing someone and I ask why they’re interested in Honeycomb, and they enthusiastically say “Because it was founded by women!”, and I wait for the rest of it, but that’s all there is. That’s it? Nothing interests you about the problem, the competitive space, the people, the customers … nothing?? It’s fine if the leadership team is what first caught your eye. But it’s kind of insulting to just stop there. Just imagine if somebody asked you out on a date “because you’re a woman”. Low. Fucking. Bar.
10. I truly want you to be happy.
I have no interest in making a hard sell to people who are dubious about Honeycomb. I don’t want to hire people who can capably do the job, but whose hearts are really elsewhere doing other things, or who barely tolerate going to work every day. I want to join with people who see their labor as an extension of themselves, who see work as an important part of their life’s project. I only want you to work here if it’s what’s best for you.
11. I’m not perfect.
We have made the wrong decision before, and will do so again. >_<
As a candidate, it is tempting to feel like you will get the job if you are awesome enough, therefore if you do not get the job it must be because you were insufficiently awesome. But that is not how hiring works — not for highly constrained startups, anyway.
If we brought you in for an interview, we already think you’re awesome. Period. Now we’re just trying to figure out if you narrowly intersect the skill sets we are lacking that we need to succeed this year.
If you could be a fly on the wall, listening to us talk about you, the phrase you would hear over and over is not “how good are they?”, but “what will they need to be successful? can we provide the support they need?” We know this is as much of a referendum on us as it is on you. And we are not perfect.
Seven years ago I was working on backend infra for mobile apps at Parse, resenting MongoDB and its accursed single write lock per replica with all my dirty, blackened soul. That’s when Miles Ward asked me to give a customer testimonial for MongoDB at AWS reinvent.
It was my first time EVER speaking in public, and I had never been more terrified. I have always been a writer, not a talker, and I was pathologically afraid of speaking in public, or even having groups of people look at me. I scripted every word, memorized my lines, even printed it all out just in case my laptop didn’t work. I had nightmares every night. For three months I woke up every night in a cold sweat, shaking.
And I bombed, completely and utterly. The laptop DIDN’T work, my limbs and tongue froze, I was shaking so badly I could hardly read my printout, and after I rushed through the last sentences I turned and stumbled robotically off the stage, fully unaware that people were raising their hands and asking questions. I even tripped over the microphone cord in my haste to escape the stage.
Afterwards I burned with unpleasantries — fear, anger, humiliation, rage at being so bad at anything. It was excruciating. For the next two years I sought out every opportunity I could get to talk at a meetup, conference, anything. I got a prescription for propranolol to help manage the physical symptoms of panic. I gave 17 more talks that year, spending most nights and weekends working on them or rehearsing, and 21 the year after that. I hated every second of it.
I hated it, but I burned up my fear and aversion as fuel. Until around 18 months later, when I realized that I no longer had nightmares and had forgotten to pack my meds for a conference. I brute forced my way through to the other side, and public speaking became just an ordinary skill or a tool like any other.
I was on a podcast last week where the topic was career journeys. They asked me what piece of career advice I would like to give to people. I promptly said that following your bliss is nice, but I think it’s important to learn to lean into pain.
“Pain is nature’s teacher,” I said. Feedback loops train us every day, mostly unconsciously. We feel aversion for pain, and we enjoy dopamine hits, and out of those and other brain chemicals our habits are made. All it takes is a little tolerance for discomfort and a some conscious tweaking of those feedback loops, and you can train yourself to achieve big things without even really trying.
But then I hesitated. Yes, leaning in to pain has done well for me in my career. But that is not the whole story, it leaves off some important truths. It has also hurt me and held me back.
Misery is not a virtue. Pain is awful. That’s why it’s so powerful and primal. It’s a pre-conscious mechanism, an acute response that kicks in long before your conscious mind. Even just the suggestion of pain (or memory of past trauma) will train you to twist and contort around to avoid it.
When you are in pain, your horizons shrink. Your vision narrows, you curl inward. You have to expend enormous amounts of energy just moving forward through the day inch by inch.
Everything is hard when you’re in pain. Your creative brain shuts down. Basic life functions become impossible tests. You have to spend so much time compensating for your reduced capacity that learning new things is nearly impossible. You can’t pick up on subtle signals when your nerves are screaming in agony. And you grow numb over time, as they die off from sheer exhaustion.
I am no longer the CEO of honeycomb.
I never wanted to be CEO; I always fiercely wanted a technical role. But it was a matter of company survival, and I did my best. I wasn’t a great CEO, although we did pretty well at the things I am good at or care about. But I couldn’t expand past them.
I hated every second of it. I cried every single day for the first year and a half. I tried to will myself into loving a role I couldn’t stand, tried to brute force my way to success like I always do. It didn’t get better. My ability to be present and curious and expansive withered. I got numb.
Turns out not every problem can be powered through on a high pain tolerance. The collateral damage starts to rack up. Sometimes the only way to succeed is to redefine success.
Pain is a terrific teacher, but pain is an acute response. Chronic pain will hijack your reward pathways, your perspective, your relationships, and every other productive system and leave them stunted.
Leaning in to pain can be powerful if you have the agency and ability to change it, or practice it to mastery, or even just adapt your own emotional responses to it. If you don’t or you can’t, leaning in to pain will kill you. Having the wisdom to know the difference is everything. Or so I’m learning.
From here on out I’ll be in the CTO seat. I don’t know what that even means yet, but I guess we’ll find out. Stay tuned. <3
You’ve convinced the security team and other stakeholders, you’ve gotten the integration running, you’re getting promising results from dev-test or staging environments… now it’s time to move from proof-of-concept to full implementation.Depending on your situation this might be a transition from staging to production, or it might mean increasing a feature flipper flag from 5% to 100%, or it might mean increasing coverage of an integration from one API endpoint to cover your entire developer footprint.
Taking into account Murphy’s Law, we expect that some things will go wrong during the rollout.Perhaps during coverage, a developer realizes that the schema designed to handle the app’s event mechanism can’t represent a scenario, requiring a redesign or a hacky solution.Or perhaps the metrics dashboard shows elevated error rates from the API frontend, and while there’s no smoking gun, the ops oncall decides to rollback the integration Just In Case it’s causing the incident.
This gives us another chance to practice empathy — while it’s easy, wearing the champion hat, to dismiss any issues found by looking for someone to blame, ultimately this poisons trust within your organization and will hamper success.It’s more effective, in the long run (and often even in the short run), to find common ground with your peers in other disciplines and teams, and work through to solutions that satisfy everybody.
Keeping the lights on
In all likelihood as integration succeeds, the team will rapidly develop experts and expertise, as well as idiomatic ways to use the product.Let the experts surprise you; folks you might not expect can step up when given a chance.Expertise flourishes when given guidance and goals; as the team becomes comfortable with the integration, explicitly recognize a leader or point person for each vendor relationship.Having one person explicitly responsible for a relationship lets them pay attention to those vendor emails, updates, and avoid the tragedy of the “but I thought *you* were” commons.This Integration Lead is also a center of knowledge transfer for your organization — they won’t know everything or help every user come up to speed, but they can help empower the local power users in each team to ramp up their teams on the integration.
As comfort grows you will start to consider ways to change your usage, for example growing into new kinds of data.This is a good time to revisit that security checklist — does the change increase PII exposure to your vendor?Would the new data lead to additional requirements such as per-field encryption?Don’t let these security concerns block you from gaining valuable insight using the new tool, but do take the chance to talk it over with your security experts as appropriate.
Throughout this organic growth, the Integration Lead remains core to managing your changing profile of usage of the vendor they shepherd; as new categories of data are added to the integration, the Lead has responsibility to ensure that the vendor relationship and risk profile are well matched to the needs that the new usage (and presumably, business value) is placing on the relationship.
Documenting the Intergation Lead role and responsibilities is critical. The team should know when to check in, and writing it down helps it happen.When new code has a security implication, or a new use case potentially amplifies the cost of an integration, bringing the domain expert in will avoid unhappy surprises.Knowing how to find out who to bring in, and when to bring them in, will keep your team getting the right eyes on their changes.
Security threats and other challenges change over time, too.Collaborating with your security team so that they know what systems are in use helps your team take note of new information that is relevant to your business. A simple example is noting when your vendors publish a breach announcement, but more complex examples happen too — your vendor transitions cloud providers from AWS to Azure and the security team gets an alert about unexpected data flows from your production cluster; with transparency and trust such events become part of a routine process rather than an emergency.
It’s all operational
Monitoring and alerting is a fact of operations life, and this has to include vendor integrations (even when the vendor integration is a monitoring product.)All of your operations best practices are needed here — keep your alerts clean and actionable so that you don’t develop pager fatigue, and monitor performance of the integration so that you don’t get blindsided by a creeping latency monster in your APIs.
Authentication and authorization are changing as the threat landscape evolves and industry moves from SMS verification codes to U2F/WebAuthn.Does your vendor support your SSO integration?If they can’t support the same SSO that you use everywhere else and can’t add it — or worse, look confused when you mention SSO — that’s probably a sign you should consider a different vendor.
A beautiful sunset
Have a plan beforehand for what needs to be done should you stop using the service.Got any mobile apps that depend on APIs that will go away or start returning permission errors?Be sure to test these scenarios ahead of time.
What happens at contract termination to data stored on the service?Do you need to explicitly delete data when ceasing use?
Do you need to remove integrations from your systems before ending the commercial relationship, or can the technical shutdown and business shutdown run in parallel?
In all likelihood these are contingency plans that will never be needed, and they don’t need to be fully fleshed out to start, but a little bit of forethought can avoid unpleasant surprises.
Year after year
Industry best practice and common sense dictate that you should revisit the security questionnaire annually (if not more frequently). Use this chance to take stock of the last year and check in — are you getting value from the service?What has changed in your business needs and the competitive landscape?
It’s entirely possible that a new year brings new challenges, which could make your current vendor even more valuable (time to negotiate a better contract rate!) or could mean you’d do better with a competing service.Has the vendor gone through any major changes?They might have new offerings that suit your needs well, or they may have pivoted away from the features you need.
Check in with your friends on the security team as well; standards evolve, and last year’s sufficient solution might not be good enough for new requirements.
Andy thinks out loud about security, society, and the problems with computers on Twitter.
❤️ Thanks so much reading, folks. Please feel free to drop any complaints, comments, or additional tips to us in the comments, or direct them to me on twitter.
All this pain will someday be worth it. 🙏❤️ charity + friends
“Get Aligned With Security”
by Lilly Ryan
If your team has decided on a third-party service to help you gather data and debug product issues, how do you convince an often overeager internal security team to help you adopt it?
When this service is something that provides a pathway for developers to access production data, as analytics tools often do, making the case for access to that data can screech to a halt at the mention of the word “production”. Progressing past that point will take time, empathy, and consideration.
I have been on both sides of the “adopting a new service” fence: as a developer hoping to introduce something new and useful to our stack, and now as a security professional who spends her days trying to bust holes in other people’s setups. I understand both sides of the sometimes-conflicting needs to both ship software and to keep systems safe.
This guide has advice to help you solve the immediate problem of choosing and deploying a third-party service with the approval of your security team. But it also has advice for how to strengthen the working relationship between your security and development teams over the longer term. No two companies are the same, so please adapt these ideas to fit your circumstances.
Understanding the security mindset
The biggest problems in technology are never really about technology, but about people. Seeing your security team as people and understanding where they are coming from will help you to establish empathy with them so that both of you want to help each other get what you want, not block each other.
First, understand where your security team is coming from. Development teams need to build features, improve the product, understand and ship good code. Security teams need to make sure you don’t end up on the cover of the NYT for data breaches, that your business isn’t halted by ransomware, and that you’re not building your product on a vulnerable stack.
This can be an unfamiliar frame of mind for developers. Software development tends to attract positive-minded people who love creating things and are excited about the possibilities of new technology. Software security tends to attract negative thinkers who are skilled at finding all the flaws in a system. These are very different mentalities, and the people who occupy them tend to have very different assumptions, vocabularies, and worldviews.
But if you and your security team can’t share the same worldview, it will be hard to trust each other and come to agreement. This is where practicing empathy can be helpful.
Before approaching your security team with your request to approve a new vendor, you may want to run some practice exercises for putting yourselves in their shoes and forcing yourselves to deliberately cultivate a negative thinking mindset to experience how they may react — not just in terms of the objective risk to the business, or the compliance headaches it might cause, but also what arguments might resonate with them and what emotional reactions they might have.
My favourite exercise for getting teams to think negatively is what I call the Land Astronaut approach.
The “Land Astronaut” Game
Imagine you are an astronaut on the International Space Station. Literally everything you do in space has death as a highly possible outcome. So astronauts spend a lot of time analysing, re-enacting, and optimizing their reactions to events, until it becomes muscle memory. By expecting and training for failure, astronauts use negative thinking to anticipate and mitigate flaws before they happen. It makes their chances of survival greater and their people ready for any crisis.
Your project may not be as high-stakes as a space mission, and your feet will most likely remain on the ground for the duration of your work, but you can bet your security team is regularly indulging in worst-case astronaut-type thinking. You and your team should try it, too.
Pick a service for you and your team to game out. Schedule an hour, book a room with a whiteboard, put on your Land Astronaut helmets. Then tell your team to spend half an hour brainstorming about all the terrible things that can happen to that service, or to the rest of your stack when that service is introduced. Negative thoughts only!
Start brainstorming together. Start out by being as outlandish as possible (what happens if their data centre is suddenly overrun by a stampede of elephants?). Eventually you will find that you’ll tire of the extreme worst case scenarios and come to consider more realistic outcomes — some of which which you may not have thought of outside of the structure of the activity.
After half an hour, or whenever you feel like you’re all done brainstorming, take off your Land Astronaut helmets, sift out the most plausible of the worst case scenarios, and try to come up with answers or strategies that will help you counteract them. Which risks are plausible enough that you should mitigate them? Which are you prepared to gamble on never happening? How will this risk calculus change as your company grows and takes on more exposure?
Doing this with your team will allow you all to practice the negative thinking mindset together and get a feel for how your colleagues in the security team might approach this request. (While this may seem similar to threat modelling exercises you might have done in the past, the focus here is on learning to adopt a security mindset and gaining empathy for this thought process, rather than running through a technical checklist of common areas of concern.)
While you still have your helmets within reach, use your negative thinking mindset to fill out the spreadsheet from the first piece in this series. This will help you anticipate most of the reasonable objections security might raise, and may help you include useful detail the security team might not have known to ask for.
Once you have prepared your list of answers to George’s worksheet and held a team Land Astronaut session together, you will have come most of the way to getting on board with the way your security team thinks.
Preparing for compromise
You’ve considered your options carefully, you’ve learned how to harness negative thinking to your advantage, and you’re ready to talk to your colleagues in security – but sometimes, even with all of these tools at your disposal, you may not walk away with all of the things you are hoping for.
Being willing to compromise and anticipating some of those compromises before you approach the security team will help you negotiate more successfully.
While your Land Astronaut helmets are still within reach, consider using your negative thinking mindset game to identify areas where you may be asked to compromise. If you’re asking for production access to this new service for observability and debugging purposes, think about what kinds of objections may be raised about this and how you might counter them or accommodate them. Consider continuing the activity with half of the team remaining in the Land Astronaut role while the other half advocates from a positive thinking standpoint. This dynamic will get you having conversations about compromise early on, so that when the security team inevitably raises eyebrows, you are ready with answers.
Be prepared to consider compromises you had not anticipated, and enter into discussions with the security team with as open a mind as possible. Remember the team is balancing priorities of not only your team, but other business and development teams as well. If you and your security colleagues are doing the hard work to meet each other halfway then you are more likely to arrive at a solution that satisfies both parties.
Working together for the long term
While the previous strategies we’ve covered focus on short-term outcomes, in this continuous-deployment, shift-left world we now live in, the best way to convince your security team of the benefits of a third-party service – or any other decision – is to have them along from day one, as part of the team.
Roles and teams are increasingly fluid and boundary-crossing, yet security remains one of the roles least likely to be considered for inclusion on a software development team. Even in 2019, the task of ensuring that your product and stack are secure and well-defended is often left until the end of the development cycle. This contributes a great deal to the combative atmosphere that is common.
Bringing security people into the development process much earlier builds rapport and prevents these adversarial, territorial dynamics. Consider working together to build Disaster Recovery plans and coordinating for shared production ownership.
If your organisation isn’t ready for that kind of structural shift, there are other ways to work together more closely with your security colleagues.
Try having members of your team spend a week or two embedded with the security team. You may even consider a rolling exchange – a developer for a security team member – so that developers build the security mindset, and the security team is able to understand the problems your team is facing (and why you are looking at introducing this new service).
At the very least, you should make regular time to meet with the security team, get to know them as people, and avoid springing things on them late in the project when change is hardest.
Riding off together into the sunset…?
If you’ve taken the time to get to know your security team and how they think, you’ll hopefully be able to get what you want from them – or perhaps you’ll understand why their objections were valid, and come up with a better solution that works well for both of you.
Investing in a strong relationship between your development and security teams will rarely lead to the apocalypse. Instead, you’ll end up with a better product, probably some new work friends, and maybe an exciting idea for a boundary-crossing new career in tech.
But this story isn’t over! Once you get the green light from security, you’ll need to think about how to roll your new service out safely, maintain it, and consider its full lifespan within your company. Which leads us to part three of this series, on rolling it out and maintaining it … both your integration and your relationship with the security team.
Lilly Ryan is a pen tester, Python wrangler, and recovering historian from Melbourne. She writes and speaks internationally about ethical software, social identities after death, teamwork, and the telegraph. More recently she has researched the domestic use of arsenic in Victorian England, attempted urban camouflage, reverse engineered APIs, wielded the Oxford comma, and baked a really good lemon shortbread.