I found your blog (from Hacker News, I think) and your “Questionable Advice: The Trap of the Premature Senior” spoke to me, but I was wondering, do you have any followup advice on handling compensation in this situation? Should one be open to making less with this type of move, or is that not actually an issue?
You should ABSOLUTELY be open to making less. Consider it an investment in your long term career. Think of the extra money your company is paying you now as a hostage premium on top of your real market value. Staying isn’t what’s good for you, and that makes it hazard pay.
Your career is the single most valuable asset you have — it’s a multimillion dollar appreciating asset, and you should curate and guide it with an eye toward longevity. The first decade of your career is way, way too early to start optimizing for salary over experience.
This question was very relevant to me right now, as I recently spoke to a recruiter about a position that’s more in line with my career goals but pays less, and her immediate reaction was “don’t go backward on compensation.” But I can see the trade-off value in losing some comp to gain “better” experience.
Yeah, I think that’s terrible advice. 🙂 In general, if the compensation is fair and respectful, I think that is the absolute *worst* reason to make a job decision.
Salary is not a one-way escalator that you hop on after school, and gracefully exit at the peak upon retiring. It’s possible your recruiter’s advice was based on the assumption that your employers are likely to ask for your current salary and base their offers off of it. That used to be common practice, but is less and less so because of the fairness issues involved. Comp should be based on the value of your labor to the company, not your past comp, and in many states it is illegal to ask about your salary.
You may choose to take lower salaries at various times in your career — to work at a nonprofit or gov job, to learn new skills, in exchange for more flexibility or vacation time or titles or stock options, or as a result of moving to a different city or country.
I am not a financial advisor, but I am a big believer in retaining optionality. If the opportunity of your dreams came along with a starting salary of 150k, but every penny of your 170k salary is already committed, that’s a big loss of opportunity for you. This is a strong argument for trying to live well within your means, save religiously, and always have fuck you money in the bank. If you’re young and you get a salary bump, consider automatically diverting the raise into savings. Avoiding lifestyle inflation is the most painless way to save.
We have the unfathomable luxury of being incredibly well compensated for what we do. What is that luxury worth if we don’t use it to liberate ourselves, to facilitate happiness and fulfillment?
“How do I feel worthwhile as a manager when my people are doing all the implementing?”
— An Engineering Manager
Hey, real quick: how long have you been managing? If it’s less than two years, honey, the answer is “you don’t.” Your feelings about your performance don’t mean much in a new role. If you think you’re crushing it, you probably aren’t. But hey, if you think you’re screwing it up royally, you probably aren’t that either. ☺️
It took years for you to develop reliable instincts as an engineer, right? Then you switched careers and went right back to beginnerhood. That rarely feels good. So just don’t worry about it. Try not to obsess over how well you’re doing or not doing. Just engage your beginner brain, set phasers to “curiosity!” and actively pursue every learning opportunity for a year or two. Your judgment will improve. Give it time.
But experienced managers still struggle with this too. So if that’s you: let’s talk.
job satisfaction feels different for managers
First, let’s be clear: job satisfaction as a manager, should you find it, will feel very different than it did as an engineer. As an engineer, you get that very tactile sense of merging code, solving puzzles and incrementally pushing the business forward. It has a rhythm and a powerful drip, drip, drip of dopamine, and as a manager you will never ever feel that. Sorry! Some people eventually make peace with this, but many never do. No shame in that.
This is partly a function of time and proximity. Manager successes and failures play out over a much longer period of time than the successes and failures of writing and debugging code, and you can only indirectly trace your impact. It can be hard to draw a straight line from cause to effect. Some of your greatest successes may resonate and compound for years to come, yet the person might not remember, may never even have known how you contributed to their triumph. (Hell, you might not either.)
It is also related to your changing relationship with public credit and attribution. It is extremely poor form for managers to go around taking credit for things, so hopefully your org has a sturdy culture of celebrating the people doing the work and not their manager. But if you are used to receiving that stream of praise and recognition, it can be disorienting and deeply demoralizing when it dries up.
Most managers are unreliable narrators
There seems to be precisely one acceptable answer to the question of what motivates managers: loftily waxing on and on about how they get ALL their joy and fulfillment from empowering others and watching other people succeed without ever personally building anything tangible or receiving ANY of the credit. I call bullshit. (This bugs the ever-loving crap out of me.)
It reminds me of the self-abnegating monologues women are supposed to give about how amazing motherhood is, as they’re covered in vomit and haven’t slept in a week.
There is nothing wrong with wanting credit for your work, and affirmation and validation, and there is nothing inherently noble about not wanting those things. Whatever motivates you, motivates you. What matters is that you are self-aware about your needs, generous with credit, and conscious of who you lean on to get those needs met. Anyway, lots of people who become managers find themselves suddenly adrift and lacking reliable indicators about their job performance.
Part of becoming an effective manager over time is learning to recognize your own contributions and derive your own inner sense of worth. Nobody wants a needy manager. So here’s where I’d start: by locating your impact in the Really Big Stuff, the small personal moments, and any sort of crisis.
1 💜 The Really Big Stuff.
Are your users happy, and your business growing? Are you setting ambitious strategic goals and hitting them? Are your DORA metrics excellent? Are people happy to join your team and report to you? Are they awesome? Great, then you deserve some portion of the credit for that.
Most big shit is unfortunately only truly visible over much longer timelines, **but!** the longer you are a manager the more sensitive your feelers will become, the more they will pick up on subtle hints that betray deeper concerns. The sideways glance that suggests lack of trust, the offhand comment with an edge that sticks with you — those are fleeting clues which you may then delicately and expertly probe and use to disarm bad situations before they deteriorate or detonate.
(And sure, it can be really lovely to watch someone succeed when you know you had a small part to play. Congrats, you earned your salary. I just find it a little creepy and culty to act like this is what every manager must live for. There is nothing whatsoever wrong with you if living vicariously thru your reports’ successes doesn’t do it for you.)
2 💙 Random little moments.
On the flip side are the small and precious moments: did you just make someone feel supported in taking their mental health days? Did you pick up on someone’s anxiety and take a moment to check in them? Did they leave with a smile? Did you amplify someone’s voice, or help them work through a problem, or argue someone into receiving a well-deserved raise? Wielding your manager powers for good can be so easy and so gratifying.
(Seriously — give yourself a little pat on the back. This is the closest you’re going to get to a compliment most weeks. And nobody else is going to do it for you. 🤣)
3 💚 In a crisis.
Every manager will eventually encounter a crisis, and those are the moments that reveal the most about how well you have done your job. Do you have the credibility to speak for your team? Does your manager reach out for your support? Do your peers take you seriously or confide in your? Will people vouch for you? You’ll find out!
Not to put it too harshly, but in a clutch situation, are you a source of calm or are you often on the list of “situations to be managed”? Do you consistently tamp down drama and lower the stakes and the volume, or do you react in ways that amplify and escalate emotionally-charged situations? Do your feelings become other people’s problems? This reflects your ability to regulate your own feelings and emotional impulses under stress, and most of us quite overestimate our own power to self-regulate.
Go to therapy. Practice that mindfulness shit. Find what works for you, but pay attention to the energy you are contributing to any situation.
“Does it even matter if I come to work or not?”
A friend of mine was recently lamenting that it didn’t feel like it mattered if he came to work or missed all his 1x1s or not. What even was the point of showing up, as a manager?
In a way, he’s right. It shouldn’t matter if you’re out for a day, or a week. No single 1×1 should make or break something major, or you were already on terribly thin ice.
It is impossible to predict what the next crisis will be. All you can do in the meantime is keep your sociotechnical systems humming along and steadily work to improve them. Build good relationships and deepen the web of trust around you. Optimize your systems so that your people can spend as much of their time as possible on satisfying, high impact work. Make sure nobody is running ragged or being taken advantage of. Ensure redundancy and resiliency across all social and technical domains. Never stop learning. Keep your troops shiny.
Managerial value accrues over time. You can’t show up in the middle of a crisis and start fixing trust issues, any more than you can be a good coach who only shows up on game days. Train yourself to rejoice when things go smoothly in your absence (and really mean it).
TLDR
In the end, your worth as a manager is seen in the trail you leave behind. The teams that got buy-in to achieve continuous delivery. The coworkers who fondly remember working together and recruit each other, or follow you from job to job for years. The way they saw you advocate for them. Set the bar high enough that their future managers will be compared to you. 🙃
If you are a good manager, you will rack up moments over the years that mean the world to you, heartwarming and vulnerable moments when people share the impact you’ve had on them. Treasure those like rare, unpredictable treats that they are, but don’t confuse them with fuel. It will never be enough to run on.
(And hey — if you’re just starting out, and all this sounds impossibly long-term? — never underestimate the value of just being fucking kind and generous and a pleasure to interact with. The job isn’t a popularity contest — the day will come when your effectiveness means the right people hating you. But that day is not today. And it is hard to be a good manager unless people genuinely enjoy talking to you and affirmatively want to help you meet your goals.)
I’ve been at my current job for three years, and I am suddenly, accidentally, the most senior engineer on the team. I spend my days handling things like bootcamps, mentoring, architecture, and helping other engineers carve off meaningful work. This has taken a huge toll on the kind of work I want to do as an IC. I still enjoy writing and shipping features, and I am not a manager, but now I feel like I spend my days conducting meetings, interviewing, and unblocking others constantly instead of writing code myself.
What should I do? How can I deal with this situation in an effective manner? How can I keep from getting burned out on zoom? How can I reclaim more of my time to write code for myself, without sacrificing my influence? Should I get a new job? I have thought about going out and getting a new job, but I really like having a say at a high level. Here I get looped into all of the most important decisions and meetings. If I get a new job, how can I avoid starting over at the bottom of the heap and just taking assignments from other people? P.S., this is my first job.
Get a new job.
Yes, you will reset your seniority and have to earn it all over again. Yes, it will be uncomfortable and your ego will be cranky over it. Yes, you will be at the bottom of the heap and take assignments from other people for a while. Yes, you should do it anyway.
What you are experiencing now is the alluring comfort of premature seniority. You’re the smartest kid in the room, you know every corner of the system inside and out, you win every argument and anticipate every objection and you are part of every decision and you feel so deeply, pleasingly needed by the people around you.
It’s a trap.
Get the fuck out of there.
There is a world of distance between being expert in this system and being an actual expert in your chosen craft. The second is seniority; the first is merely .. familiarity
Deep down I think you know this, and feel a gnawing insecurity over your position; why else would you have emailed me? You were right. Treasure that uneasy feeling in your gut, that discomfort in the face of supreme comfortable-ness. It will lead you to a long and prosperous career as an engineer if you learn to trust it.
Think of every job like an escalator — a 50-foot high escalator that takes about two years to ride to the top. But once you’ve summited, you stall out. You can either stay and wander on that floor, or you can step to the left and pick another escalator and ride it up another 50 feet. And another.
In my mind, someone becomes a real senior engineer after they’ve done this about three times. 2-3 teams, stacks, languages, and roles, over a 5-8 year period, and then they’re solidly baked. There are insights you can derive from having seen problems solved in a few different ways that you can’t with only a single point of reference.
You don’t become a senior engineer at the 50-foot ascent, no matter how thoroughly you know the landscape. You become a senior engineer somewhere well over 100 feet, with a couple of lane changes under your belt.
The act of learning a new language and/or stack is itself an important skill. Experiencing how different orgs ship code in vastly different ways is how you internalize that there’s no one blessed path, only different sets of tradeoffs, and how you learn to reason about those tradeoffs.
And it is good for us to start over with beginner eyes. It’s humbling, it’s clarifying, it’s a cleanse for the soul. If you get too attached to feeling senior, to feeling necessary, you will undervalue the virtues of fresh eyes and questioning, of influence without authority. It is good for you to practice uncertainty and influencing others without the cheat codes of deep familiarity.
Nobody wants to work with seniors who clutch their authority with a white knuckled grip. We want to work with those who wear it lightly, who remember what it was like in our shoes.
Ultimately, this is a strong argument for building our teams behind a Rawlsian veil of ignorance concerning our own place in the pecking order. Starting fresh yourself will help you build teams where it is not miserable to be a beginner, where beginners’ contributions are recognized, where even beginners do not simply “take orders”, as you said. Because literally nobody wants that, including the beginners you are working with on your teams today.
After you have gotten a new job or two, and proven to yourself that you can level up again and master new stacks and technologies, that fretful inner voice questioning whether you deserve the respect you receive or not will calm down. You will have proven to yourself that your success wasn’t just a one-off, that you can be dropped into any situation, learn the local ropes and succeed. You will be a senior engineer.
My company has recently begun pushing for us to build and staff out what I can only describe as “command centers”. They’re picturing graphs, dashboards…people sitting around watching their monitors all day just to find out which apps or teams are having issues. With your experience in monitoring and observability, and your opinions on teams supporting their own applications…do you think this sounds like a bad idea? What are things to watch out for, or some ways this might all go sideways?
— Anonymous
Jesus motherfucking Christ on a stick. Is it 1995 where you work? That’s the only way I can try and read this plan like it makes sense.
It’s a giant waste of money and no, it won’t work. This path leads into a death spiral where alarms are going off constantly (yet somehow never actually catching the real problems), people getting burned out, and anyone competent will either a) leave or b) refuse to be on call. Sideways enough for you yet?
Snark aside, there are two foundational flaws with this plan.
1) watching graphs is pointless. You can automate that shit, remember? ✨Computers!✨ Furthermore, this whole monitoring-based approach will only ever help you find the known unknowns, the problems you already know to look for. But most of your actual problems will be unknown unknowns, the ones you don’t know about yet.
2) those people watching the graphs… When something goes wrong, what exactly can they do about it? The answer, unfortunately, is “not much”. The only people who can swiftly diagnose and fix complex systems issues are the people who build and maintain those systems, and those people are busy building and maintaining, not watching graphs.
That extra human layer is worse than useless; it is actively harmful. By insulating developers from the consequences of their actions, you are concealing from them the information they need to understand the consequences of their actions. You are interfering with the most basic of feedback loops and causing it to malfunction.
The best time to find a bug is as soon as possible after writing it, while it’s all fresh in your head. If you let it fester for days, weeks, or months, it will be exponentially more challenging to find and solve. And the best people to find those bugs are the people who wrote them
Helpful? Hope so. Good luck. And if they implement this anyway — leave. You deserve to work for a company that won’t waste your fucking time.
Dan Golant asked a great question today: “Any advice/reading on how to establish a team’s critical path?”
I repeated back: “establish a critical path?” and he clarified:
Yea, like, you talk about buttoning up your “critical path”, making sure it’s well-monitored etc. I think that the right first step to really improving Observability is establishing what business processes *must* happen, what our “critical paths” are. I’m trying to figure out whether there are particularly good questions to ask that can help us document what these paths are for my team/group in Eng.
“Critical path” is one of those phrases that I think I probably use a lot. Possibly because the very first real job I ever had was when I took a break from college and worked at criticalpath.net (“we handle the world’s email”) — and by “work” I mean, “lived in SF for a year when I was 18 and went to a lot of raves and did a lot of drugs with people way cooler than me”. Then I went back to college, the dotcom boom crashed, and the CP CFO and CEO actually went to jail for cooking the books, becoming the only tech execs I am aware of who actually went to jail.
Where was I.
Right, critical path. What I said to Dan is this: “What makes you money?”
Like, if you could only deploy three end-to-end checks that would perform entire operations on your site and ensure they work at all times, what would they be? what would they do? “Submit a payment” is a super common one; another is new user signups.
The idea here is to draw up a list of the things that are absolutely worth waking someone up to fix immediately, night or day, rain or shine. That list should be as compact and well-defined as possible. This allows you to be explicit about the fact that anything else can wait til morning, or some other less-demanding service level agreement.
And typically the right place to start on this list is by asking yourselves: “what makes us money?” as a proxy for the real questions, which are: “what actions allow us to survive as a business? What do our customers care the absolute most about? What makes us us?” That’s your critical path.
Someone will usually seize this opportunity to argue that absolutely any deterioration in service is worth paging someone immediately to fix it, day or night. They are wrong, but it’s good to flush these assumptions out and have this argument kindly out in the open.
(Also, this is really a question about service level objectives. So if you’re asking yourself about the critical path, you should probably consider buying Alex Hidalgo’s book on SLOs, and you may want to look into the Honeycomb SLO product, the only one in the industry that actually implements SLOs as the Google SRE book defines them (thanks Liz!) and lets you jump straight from “what are our customers experiencing?” to “WHY are they experiencing it”, without bouncing awkwardly from aggregate metrics to logs and back and just … hoping … the spikes line up according to your visual approximations.)
I work as an engineering manager for a company whose non-technology leadership insists there has to be a way to measure the individual productivity of a software engineer. I have the opposite belief. I don’t believe you can measure the productivity of “professional” careers, or thought workers (ex: how do measure productivity of a doctor, lawyer, or chemist?).
For software engineering in particular, I feel that metrics can be gamed, don’t tell the whole story, or in some cases, are completely arbitrary. Do you measure individual developer productivity? If so, what do you measure, and why do you feel it’s valuable? If you don’t and share similar feelings as mine, how would you recommend I justify that position to non-technology leadership?
Thanks for your time.
Anonymous Engineering Manager
Dear Anon,
Once upon a time I had a job as a sysadmin, 100% remote, where all work was tracked using RT tasks. I soon realized that the owner didn’t have a lot of independent technical judgment, and his main barometer for the caliber of our contributions was the number of tasks we closed each day.
I became a ticket-closing machine. I’d snap up the quick and easy tasks within seconds. I’d pattern match and close in bulk when I found a solution for a group of tasks. I dove deep into the list of stale tickets looking for ones I could close as “did not respond” or “waiting for response”, especially once I realized there was no penalty for closing the same ticket over and over.
My boss worshiped me. I was bored as fuck. Sigh.
I guess what I’m trying to say is, I am fully in your camp. I don’t think you can measure the “productivity” of a creative professional by assigning metrics to their behaviors or process markers, and I think that attempting to derive or inflict such metrics can inflict a lot of damage.
In fact, I would say that to the extent you can reduce a job to a set of metrics, that job can be automated away. Metrics are for easy problems — discrete, self-contained, well-understood problems. The more challenging and novel a problem, the less reliable these metrics will be.
Your execs should fucking well know this: how would THEY like to be evaluated based on, like, how many emails they send in a day? Do they believe that would be good for the business? Or would they object that they are tasked with the holistic success of the org, and that their roles are too complex to reduce to a set of metrics without context?
This actually makes my blood boil. It is condescending as fuck for leadership to treat engineers like task-crunching interchangeable cogs. It reveals a deep misunderstanding of how sociotechnical systems are developed and sustained (plus authoritarian tendencies, and usually a big dollop of personal insecurity).
But what is the alternative?
In my experience, the “right” answer, i.e. the best way to run consistently high-performing teams, involves some combination of the following:
Outcome-based management that practices focusing on impact, plus
Team level health metrics, combined with
Engineering ladder and regular lightweight reviews, and
Managers who are well calibrated across the org, and encouraged to interrogate their own biases openly & with curiosity.
The right way to look at performance is at the team level. Individual engineers don’t own or maintain code; teams do. The team is the irreducible unit of ownership. So you need to incentivize people to think about work and spending their time cooperatively, optimizing for what is best for the team.
Some of the hardest and most impactful engineering work will be all but invisible on any set of individual metrics. You want people to trust that their manager will have their backs and value their contributions appropriately at review time, if they simply act in the team’s best interest. You do not want them to waste time gaming the metrics or courting personal political favor.
This is one of the reasons that managers need to be technical — so they can cultivate their own independent judgment, instead of basing reviews on hearsay. Because some resources (i.e. your budget for individual bonuses) are unfortunately zero-sum, and you are always going to rely on the good judgment of your engineering leaders when it comes to evaluating the relative impact of individual contributions.
“I would say that Joe’s contribution this quarter had greater impact than Jane’s. But is that really true? Jane did a LOT of mentoring and other “glue” work, which tends to be under-acknowledged as leadership work, so I just want to make sure I am evaluating this fairly … Does anyone else have a perspective on this? What might I be missing?” — a manager keeping themselves honest in calibrations
I do think every team should be tracking the 4 DORA metrics — time elapsed between merge and deploy, frequency of deploy, time to recover from outages, duration of outages — as well as how often someone is paged outside of business hours. These track pretty closely to engineering productivity and efficiency.
But leadership should do its best to be outcome oriented. The harder the problem, the more senior the contributor, the less business anyone has dictating the details of how or why. Make your agreements, then focus on impact.
This is harder on managers, for sure — it’s easier to count the hours someone spends at their desk or how many lines of code they commit than to develop a nuanced understanding of the quality and timbre of an engineer’s contributions to the product, team and the company over time. It is easier to micromanage the details than to negotiate a mutual understanding of what actually matters, commit to doing your part … and then step away, trusting them to fill in the gaps.
But we should expect this; it’s worth it. It is in those gaps where we feel trusted to act that we find joy and autonomy in our labor, where we do our best work as skilled artisans.
Last night I was talking with Mark Ferlatte about the advice we have given our respective companies in this pandemic era. He shared with me this link, on how to salvage a disastrous day. It’s a good link: you should read it.
My favorite part: “Your feelings will follow your actions. Just do it.”
The hardest part for me is, “Book-end your day. Don’t push it into the midnight hours.” Ugh. I really, really struggle with this because my brain takes a long long time to settle in and get started on a task to the point where I feel like I’m on a roll with it, and once I’m on a roll I do not want to stop until I’m done. Because god knows how long it will be — days? weeks?? — until I can catch this wave again, feel inspired again. But it’s true, if I stay up all night working I’m just setting myself up for a fuzzy, blundery tomorrow.
The advice we gave Honeycombers was differently shaped, though similar in spirit. I’ve had a few people ask me to share it, so here it is.
We formally request …
First, we would like to point out that what you are all being asked to do right now is impossible. Parenting, homeschooling, working, caregiving, correcting misinformed neighbors, being an engaged citizen … it is fifteen people’s worth of work. It is literally impossible.
But hey, it has always been impossible. We have never been able to do everything we want to do — there isn’t enough time. There was never enough time! We succeed as a company not by doing everything on our list, but by saying no to the right things; by NOT-doing enough most things so we can focus on the few things we have identified that matter most. That was true before COVID, it’s just truer now.
So: let’s all focus hard on our top priority. Shed as much of the other stuff as you have to. Shed more. Ask your manager for help figuring out what to shed, until you are down to an amount you can probably manage.
And speaking of focus:
You aren’t operating at full capacity. We all get that right now: none of us are. And nobody expects you to. So please spend zero energy on performing like you’re doing work, or acting extra-responsive, or keeping up a front like things are normal and you’re doing fine. That performance costs you precious energy, while doing nothing to get us closer to our goals.
What we need from you is not performance or busy-busy-ness but your engaged creative self — your active, curious mind engaging with our top problem. I would rather have 30 minutes of your creative energy applied to our biggest problem today than five hours of your distracted split-brain, juggling, trying to keep up with chat and seem like you’re as available per usual today.
So when you’re figuring out your schedule, please optimize for that — focused time on our biggest problem — and then communicate your availability to your team. If you’re a parent and you can only really work three days a week, calendar that. (If you’re not a parent, remember that you too are allowed to feel overwhelmed and underwater. Just because some have it even harder, doesn’t invalidate what you’re going through.)
In Summary,
Take care of yourself
Take care of your loved ones
Say no to as much as you possibly can
Focus on impact
No performative normalcy
Remember: this is temporary 🖤
We are incredibly fortunate — to be here, to have these resources, to have each other. It’s okay to have bad days; this is why we have teams, to carry each other through the hardest spots. Do your best. Everything is going to be okay, more or less.
Welcome to the second installment of my advice column! Last time we talked about the emotional impact of going back to engineering after a stint in management. If you have a question you’d like to ask, please email me or DM it to me on twitter.
Hi Charity! I hope it’s ok to just ask you this…
I’m trying to get our company more aware of observability and I’m finding it difficult to convince people to look more into it. We currently don’t have the kind of systems that would require it much – but we will in future and I want us to be ahead of the game.
If you have any tips about how to explain this to developers (who are aware that quality is important but don’t always advocate for it / do it as much as I’d prefer), or have concrete examples of “here’s a situation that we needed observability to solve – and here’s how we solved it”, I’d be super grateful.
If this is too much to ask, let me know too 🙂
I’ve been talking to Abby Bangser a lot recently – and I’m “classifying” observability as “exploring in production” in my mental map – if you have philosophical thoughts on that, I’d also love to hear them 🙂
Yay, what a GREAT note! I feel like I get asked some subset or variation of these questions several times a week, and I am delighted for the opportunity to both write up a response for you and post it for others to read. I bet there are orders of magnitude more people out there with the same questions who *don’t* ask, so I really appreciate those who do. <3
I want to talk about the nuts and bolts of pitching to engineering teams and shepherding technical decisions like this, and I promise I will offer you some links to examples and other materials. But first I want to examine some of the assumptions in your note, because they elegantly illuminate a couple of common myths and misconceptions.
Myth #1: you don’t need observability til you have problems of scale
First of all, there’s this misconception that observability is something you only need when you have really super duper hard problems, or that it’s only justified when you have microservices and large distributed systems or crazy scaling problems. No, no no nononono.
There may come a point where you are ABSOLUTELY FUCKED if you don’t have observability, but it is ALWAYS better to develop with it. It is never not better to be able to see what the fuck you are doing! The image in my head is of a hiker with one of those little headlamps on that lets them see where they’re putting their feet down. Most teams are out there shipping opaque, poorly understood code blindly — shipping it out to systems which are themselves crap snowballs of opaque, poorly understood code. This is costly, dangerous, and extremely wasteful of engineering time.
Ever seen an engineering team of 200, and struggled to understand how the product could possibly need more than one or two teams of engineers? They’re all fighting with the crap snowball.
Developing software with observability is better at ANY scale. It’s better for monoliths, it’s better for tiny one-person teams, it’s better for pre-production services, it’s better for literally everyone always. The sooner and earlier you adopt it, the more compounding value you will reap over time, and the more of your engineers’ time will be devoted to forward progress and creating value.
Myth #2: observability is harder and more technically advancedthan monitoring
Actually, it’s the opposite — it’s much easier. If you sat a new grad down and asked them to instrument their code and debug a small problem, it would be fairly straightforward with observability. Observability speaks the native language of variables, functions and API endpoints, the mental model maps cleanly to the request path, and you can straightforwardly ask any question you can come up with. (A key tenet of observability is that it gives an engineer the ability to ask any question, without having had to anticipate it in advance.)
With metrics and logging libraries, on the other hand, it’s far more complicated.you have to make a bunch of awkward decisions about where to emit various types of statistics, and it is terrifyingly easy to make poor choices (with terminal performance implications for your code and/or the remote data source). When asking questions, you are locked in to asking only the questions that you chose to ask a long time ago. You spend a lot of time translating the relationships between code and lowlevel systems resources, and since you can’t break down by users/apps you are blocked from asking the most straightforward and useful questions entirely!
Doing it the old way Is. Fucking. Hard. Doing it the newer way is actually much easier, save for the fact that it is, well, newer — and thus harder to google examples for copy-pasta. But if you’re saturated in decades of old school ops tooling, you may have some unlearning to do before observability seems obvious to you.
Myth #3: observability is a purely technical solution
To be clear, you can just add an observability tool to your stack and go on about your business — same old things, same old way, but now with high cardinality!
You can, but you shouldn’t.
These are sociotechnical systems and they are best improved with sociotechnical solutions. Tools are an absolutely necessary and inextricable part of it. But so are on call rotations and the fundamental virtuous feedback loop of you build it, you run it. So are code reviews, monitoring checks, alerts, escalations, and a blameless culture. So are managers who allocate enough time away from the product roadmap to truly fix deep technical rifts and explosions, even when it’s inconvenient, so the engineers aren’t in constant monkeypatch mode.
I believe that observability is a prerequisite for any major effort to have saner systems, simply because it’s so powerful being able to see the impact of what you’ve done. In the hands of a creative, dedicated team, simply wearing a headlamp can be transformational.
Observability is your five senses for production.
You’re right on the money when you ask if it’s about exploring production, but you could also use words that are even more basic, like “understanding” or “inspecting”. Observability is to software systems as a debugger is to software code. It shines a light on the black box. It allows you to move much faster, with more confidence, and catch bugs much sooner in the lifecycle — before users have even noticed. It rewards you for writing code that is easy to illuminate and understand in production.
So why isn’t everyone already doing it? Well, making the leap isn’t frictionless. There’s a minimal amount of instrumentation to learn (easier than people expect, but it’s nonzero) and then you need to learn to see your code through the lens of your own instrumentation. You might need to refactor your use of older tools, such as metrics libraries, monitoring checks and log lines. You’ll need to learn another query interface and how it behaves on your systems. You might find yourself amending your code review and deploy processes a bit.
Nothing too terrible, but it’s all new. We hate changing our tool kits until absolutely fucking necessary. Back at Parse/Facebook, I actually clung to my sed/awk/shell wizardry until I was professionally shamed into learning new ways when others began debugging shit faster than I could. (I was used to being the debugger of last resort, so this really pissed me off.) So I super get it! So let’s talk about how to get your team aligned and hungry for change.
Okay okay okay already, how do I get my team on board?
If we were on the phone right now, I would be peppering you with a bunch of questions about your organization. Who owns production? Who is on call? Who runs the software that devs write? What is your deploy process, and how often does it get updated, and by who? Does it have an owner? What are the personalities of your senior folks, who made the decisions to invest in the current tools (and what are they), what motivates them, who are your most persuasive internal voices? Etc. Every team is different. <3
There’s a virtuous feedback loop you need to hook up and kickstart and tweak here, where the people with the original intent in their heads (software engineers) are also informed and motivated, i.e. empowered to make the changes and personally impacted when things are broken. I recommend starting by putting your software engineers on call for production (if you haven’t). This has a way of convincing even the toughest cases that they have a strong personal interest in quality and understandability.
Pay attention to your feedback loop and the alignment of incentives, and make sure your teams are given enough time to actually fix the broken things, and motivation usually isn’t a problem. (If it is, then perhaps another feedback loop is lacking: your engineers feeling sufficiently aligned with your users and their pain. But that’s another post.)
Technical ownership over technical outcomes
I appreciate that you want your team to own the technical decisions. I believe very strongly that this is the right way to go. But it doesn’t mean you can’t have influence or impact, and particularly in times like this.
It is literally your job to have your head up, scanning the horizon for opportunities and relevant threats. It’s their job to be heads down, focusing on creating and delivering excellent work. So it is absolutely appropriate for you to flag something like observability as both an opportunity and a potential threat, if ignored.
If I were in your situation and wanted my team to check out some technical concept, I might send around a great talk or two and ask folks to watch it, and then maybe schedule a lunchtime discussion. Or I might invite a tech luminary in to talk with the team, give a presentation and answer their questions. Or schedule a hack week to apply the concept to a current top problem, or something else of that nature.
But if I really wanted them to take it fucking seriously, I would put my thumb on the scale. I would find myself a champion, load them up with context, and give them ample time and space to skill up, prototype, and eventually present to the team a set of recommendations. (And I would stay in close contact with them throughout that period, to make sure they didn’t veer too far off course or lose sight of my goals.)
Get a champion.
Ideally you want to turn the person who is most invested in the old way of doing things — the person who owns the ELK cluster, say, or who was responsible for selecting the previous monitoring toolkit, or the goto person for ops questions — from your greatest obstacle into your proxy warrior. This only works if you know that person is open-minded and secure enough to give it a fair shot & publicly change course, has sufficiently good technical judgment to evaluate and project into the future, and has the necessary clout with their peers. If they don’t, or if they’re too afraid to buck consensus: pick someone else.
Give them context.
Take them for a long walk. Pour your heart and soul out to them. Tell them what you’ve learned, what you’ve heard, what you hope it can do for you, what you fear will happen if you don’t. It’s okay to get personal and to admit your uncertainties. The more context they have, the better the chance they will come out with an outcome you are happy with. Get them worried about the same things that worry you, get them excited about the same possibilities that excite you. Give them a sense of the stakes.
And don’t forget to tell them why you are picking them — because they are listened to by their peers, because they are already expert in the problem area, because you trust their technical judgment and their ability to evaluate new things — all the reasons for picking them will translate well into the best kind of flattery — the true kind.
Give them a deadline.
A week or two should be plenty. Most likely, the decision is not going to be unilaterally theirs (this also gives you a bit of wiggle room should they come back going “ah no ELK is great forever and ever”), but their recommendations should carry serious weight with the team and technical leadership. Make it clear what sort of outcome you would be very pleased with (e.g. a trial period for a new service) and what reasons you would find compelling for declining to pursue the project (i.e. your tech is unsupported, cost prohibitive, etc). Ideally they should use this time to get real production data into the services they are testing out, so they can actually experience and weigh the benefits, not just read the marketing copy.
As a rule of thumb, I always assume that managers can’t convince engineers to do things: only other engineers can. But what you can do instead is set up an engineer to be your champion. And then just sit quietly in the corner, nodding, with an interested look on your face.
The nuclear option
if you <3 prod, prod will <3 you back
You have one final option. If there is no appropriate champion to be found, or insufficient time, or if you have sufficient trust with the team that you judge it the right thing to do: you can simply order them to do something your way. This can feel squicky. It’s not a good habit to get into. It usually results in things being done a bit slower, more reluctantly, more half-assedly. And you sacrifice some of your power every time you lean on your authority to get your team to do something.
But it’s just as bad for a leader to take it off the table entirely.
Sometimes you will see things they can’t. If you cannot wield your power when circumstances call for it, then you don’t fucking have real power — you have unilaterally disarmed yourself, to the detriment of your org. You can get away with this maybe twice a year, tops.
But here’s the thing: if you order something to be done, and it turns out in the end that you were right? You earn back all the power you expended on it plus interest. If you were right, unquestionably right in the eyes of the team, they will respect you more for having laid down the law and made sure they did the right thing.
One of my stretch goals for 2019 was to start writing an advice column. I get a lot of questions about everything under the sun: observability, databases, career advice, management problems, what the best stack is for a startup, how to hire and interview, etc. And while I enjoy this, having a high opinion of my own opinions and all, it doesn’t scale as well as writing essays. I do have a (rather all-consuming) day job.
So I’d like to share some of the (edited and lightly anonymized) questions I get asked and some of the answers I have given. With permission, of course. And so, with great appreciation to my anonymous correspondent for letting me publish this, here is one.
Hi Charity,
I’ve been in tech for 25 years. I don’t have a degree, but I worked my way up from menial jobs to engineering, and since then I have worked on some of the biggest sites in the world. I have been offered a management role many times, but every time I refused. Until about two years ago, when I said “fuck it, I’m almost 40; why not try.”
I took the job with boundless enthusiasm and motivation, because the team was honestly a mess. We were building everything on-prem, and ops was constantly bullying developers over their supposed incompetence. I had gone to conferences, listened to podcasts, and read enough blog posts that my head was full of “DevOps/CloudNative/ServiceOriented//You-build-it-you-run-it/ServantLeaders” idealism. I knew I couldn’t make it any worse, and thought maybe, just maybe I could even make it better.
Soon after I took the job, though, there were company-wide layoffs. It was not done well, and morale was low and sour. People started leaving for happier pastures. But I stayed. It was an interesting challenge, and I threw my heart and soul into it.
For two years I have stayed and grinded it out: recruiting (oh that is so hard), hiring, and then starting a migration to a cloud provider, and with the help of more and more people on the new team, slowly shifted the mindset of the whole engineering group to embrace devops best practices. Now service teams own their code in production and are on-call for them, migrate themselves to the cloud with my team supporting them and building tools for them. It is almost unrecognizable compared to where we were when I began managing.
A beautiful story isn’t it? I hope you’re still reading. 🙂
Now I have to say that with my schedule full of 1:1s, budgeting, hiring, firing, publishing papers of mission statements and OKRs, shaping the teams, wielding influence, I realized that I enjoyed none of the above. I read your 17 reasons not to be a manager, and I check so many boxes. It is a pain in the ass to constantly listen to people’s egos, talk to them and keep everybody aligned (which obviously never happens). And of course I am being crushed between top-down on-the-spot business decisions and bottom-up frustration of poorly executed engineering work under deadlines. I am also destroyed by the mistrust and power games I am witnessing (or involved in, sometimes). while I long for collaboration and trust. And of course when things go well my team gets all the praise, and when things go wrong I take all the blame. I honestly don’t know how one can survive without the energy provided by praise and a sense of achievement.
All of the above makes me miss being an IC (Individual Contributor), where I could work for 8 hours straight without talking to anyone, build stuff, say what I wanted when I wanted, switch jobs if I wasn’t happy, and basically be a little shit like the ones you mention in your article.
Now you may say it’s obvious: I should find a new IC job in a healthier company. You even wrote about it. Going back to IC after two years of management is actually a good move.
But when I think about doing it, I get stuck. I don’t know if I would be able to do it again, or if I could still enjoy it. I’ve seen too many things, I’ve tasted what it’s like to be (sometimes) in control, and I did have a big impact on the company’s direction over time. I like that. If I went back to being an IC, I would feel small and meaningless, like just another cog in the machine. And of course, being 40-ish, I will compete with all those 20-something smartasses who were born with kubernetes.
Thank you for reading. Could you give me your thoughts on this? In any case, it was good to get it off my chest.
Cheers,
Cog?
Dear Cog?,
Holy shitballs! What an amazing story! That is an incredible achievement in just two years, let alone as a rookie manager. You deserve huge props for having the vision, the courage, and the tenacity to drive such a massive change through.
Of COURSE you’re feeling bored and restless. You didn’t set out on a glorious quest for a life of updating mission statements and OKRs, balancing budgets, tending to people’s egos and fluffing their feelings, tweaking job descriptions, endless 1x1s and meetings meetings meetings, and the rest of the corporate middle manager’s portfolio. You wanted something much bigger. You wanted to change the world. And you did!
But now you’ve done it. What’s next?
First of all, YOUR COMPANY SUCKS. You don’t once mention your leadership — where are they in all this? If you had a good manager, they would be encouraging you and eagerly lining up a new and bigger role to keep you challenged and engaged at work. They are not, so they don’t deserve you. Fuck em. Please leave.
Another thing I am hearing from you is, you harbor no secret desire to climb the managerial ranks at this time. You don’t love the daily rhythms of management (believe it or not, some do); you crave novelty and mastery and advancement. It sounds like you are willing to endure being a manager, so long as that is useful or required in order to tackle bigger and harder problems. Nothing wrong with that! But when the music stops, it’s time to move on. Nobody should be saddled with a manager whose heart isn’t in the work.
You’re at the two year mark. This is a pivotal moment, because it’s the beginning of the end of the time when you can easily slip back into technical work. It will get harder and harder over the next 2-3 years, and at some point you will no longer have the option.
Picking up another technical role is the most strategic option, the one that maximizes your future opportunities as a technical leader. But you do not seem excited by this option; instead you feel many complex and uncomfortable things. It feels like going backwards. It feels like losing ground. It feels like ceding status and power.
“Management isn’t a promotion, it’s a career change.”
But if management is not a promotion, then going back to an engineering role should not feel like a demotion! What the fuck?!
It’s one thing to say that. Whether it’s true or not is another question entirely, a question of policy and org dynamics. The fact is that in most places, most of the power does go to the managers, and management IS a promotion. Power flows naturally away from engineers and towards managers unless the org actively and vigorously pushes back on this tendency by explicitly allocating certain powers and responsibilities to other roles.
I’m betting your org doesn’t do this. So yeah, going back to being an IC WILL be a step down in terms of your power and influence and ability to set the agenda. That’s going to feel crappy, no question. We humans hate that.
Three points.
You cannot go back to doing exactly what you did before, for the very simple reason that you are not the same person. You are going to be attuned to power dynamics and ways of influencing that you never were before — and remember, leadership is primarily exercised through influence, not explicit authority.Senior ICs who have been managers are supremely powerful beings, who tend to wield outsize influence. Smart managers will lean on them extensively for everything from shadow management and mentorship to advice, strategy, etc. (Dumb managers don’t. So find a smart manager who isn’t threatened by your experience.)
You’re a short-timer here, remember? Your company sucks. You’re just renewing your technical skills and pulling a paycheck while finding a company that will treat you better, that is more aligned with your values.
Lastly (and most importantly), I have a question. Why did you need to become a manager in order to drive sweeping technical change over the past two years? WHY couldn’t you have done it as a senior IC? Shouldn’t technical people be responsible for technical decisions, and people managers responsible for people decisions? Could this be your next challenge, or part of it? Could you go back to being an engineer, equipped with your shiny new powers of influence and mystical aura of recent management experience, and use it to organize the other senior ICs to assert their rightful ownership over technical decisions? Could you use your newfound clout with leadership and upper management to convince them that this will help them recruit and retain better talent, and is a better way to run a technical org — for everyone?
I believe this is a better way, but I have only ever seen these changes happen when agitated for and demanded by the senior ICs. If the senior ICs don’t assert their leadership, managers are unlikely to give it to them. If managers try, but senior ICs don’t inhabit their power, eventually the managers just shrug and go back to making all the decisions. That is why ultimately this is a change that must be driven and owned — at a minimum co-owned — by the senior individual contributors.
I hope you can push back against that fear of being small and meaningless as an individual contributor. The fact that it very often is this way, especially in strongly hierarchical organizations, does not mean that it has to be this way; and in healthy organizations it is not this way. Command-and-control systems are not conducive to creative flourishing. We have to fight the baggage of the authoritarian structures we inherited in order to make better ones.
Organizations are created afresh each and every day — not created for us, but by us. Help create the organization you want to work at, where senior people are respected equally and have domains of ownership whether they manage people or technology. If your current gig won’t value that labor, find one that will..
They exist. And they want to hire you.
Lots of companies are DYING to hire this kind of senior IC, someone who is still hands on yet feels responsibility for the team as a whole, who knows the business side, who knows how to mentor and craft a culture and can herd cats when nec
There are companies that know how to use ICs at the strategic level, even executive level. There are bosses who will see you not as a threat, but as a *huge asset* they can entrust with monumental work.
As a senior contributor who moves fluidly between roles, you are especially well-equipped to help shape a sociotechnical organization. Could you make it your mission to model the kind of relationship you want to see between management and ICs, whichever side you happen to be on? We need more people figuring out how to build organizations where management is not a promotion, just a change of career, and where going back and forth carries no baggage about promotions and demotions. Help us.
And when you figure it out, please don’t keep it to yourself. Expand your influence and share your findings by writing your experiences in blog posts, in articles, in talks. Tell stories. Show people people how much better it is this way. Be so magnificently effective and mysteriously influential as a senior IC that all the baby engineers you work with want to grow up to be just like you.
Hope this helps.
charity
P.S. — Oh and stop fretting about “competing” with the 20-somethings kuberneteheads, you dork. You have been learning shit your whole career and you’ll learn this shit too. The tech is the easy part. The tech will always be the easy part. 🙂
Some people eventually make peace with this, but many never do. No shame in that.
Computers!✨ Furthermore, this whole monitoring-based approach will only ever help you find the known unknowns, the problems you already know to look for. But most of your actual problems will be 
All of the above makes me miss being an IC (Individual Contributor), where I could work for 8 hours straight without talking to anyone, build stuff, say what I wanted when I wanted, switch jobs if I wasn’t happy, and basically be a little shit like the ones you mention in your article.
t’s one thing to say that. Whether it’s true or not is another question entirely, a question of policy and org dynamics. The fact is that in most places, most of the power does go to the managers, and management IS a promotion. Power flows naturally away from engineers and towards managers unless the org actively and vigorously pushes back on this tendency by explicitly allocating certain powers and responsibilities to other roles.
