Dan Golant asked a great question today: “Any advice/reading on how to establish a team’s critical path?”
I repeated back: “establish a critical path?” and he clarified:
Yea, like, you talk about buttoning up your “critical path”, making sure it’s well-monitored etc. I think that the right first step to really improving Observability is establishing what business processes *must* happen, what our “critical paths” are. I’m trying to figure out whether there are particularly good questions to ask that can help us document what these paths are for my team/group in Eng.
“Critical path” is one of those phrases that I think I probably use a lot. Possibly because the very first real job I ever had was when I took a break from college and worked at criticalpath.net (“we handle the world’s email”) — and by “work” I mean, “lived in SF for a year when I was 18 and went to a lot of raves and did a lot of drugs with people way cooler than me”. Then I went back to college, the dotcom boom crashed, and the CP CFO and CEO actually went to jail for cooking the books, becoming the only tech execs I am aware of who actually went to jail.
Where was I.
Right, critical path. What I said to Dan is this: “What makes you money?”
Like, if you could only deploy three end-to-end checks that would perform entire operations on your site and ensure they work at all times, what would they be? what would they do? “Submit a payment” is a super common one; another is new user signups.
The idea here is to draw up a list of the things that are absolutely worth waking someone up to fix immediately, night or day, rain or shine. That list should be as compact and well-defined as possible. This allows you to be explicit about the fact that anything else can wait til morning, or some other less-demanding service level agreement.
And typically the right place to start on this list is by asking yourselves: “what makes us money?” as a proxy for the real questions, which are: “what actions allow us to survive as a business? What do our customers care the absolute most about? What makes us us?” That’s your critical path.
Someone will usually seize this opportunity to argue that absolutely any deterioration in service is worth paging someone immediately to fix it, day or night. They are wrong, but it’s good to flush these assumptions out and have this argument kindly out in the open.
(Also, this is really a question about service level objectives. So if you’re asking yourself about the critical path, you should probably consider buying Alex Hidalgo’s book on SLOs, and you may want to look into the Honeycomb SLO product, the only one in the industry that actually implements SLOs as the Google SRE book defines them (thanks Liz!) and lets you jump straight from “what are our customers experiencing?” to “WHY are they experiencing it”, without bouncing awkwardly from aggregate metrics to logs and back and just … hoping … the spikes line up according to your visual approximations.)
I just read this piece, which is basically a very long subtweet about my Friday deploy threads. Go on and read it: I’ll wait.
Here’s the thing. After getting over some of the personal gibes (smug optimism? literally no one has ever accused me of being an optimist, kind sir), you may be expecting me to issue a vigorous rebuttal. But I shan’t. Because we are actually in violent agreement, almost entirely.
I have repeatedly stressed the following points:
I want to make engineers’ lives better, by giving them more uninterrupted weekends and nights of sleep. This is the goal that underpins everything I do.
Anyone who ships code should develop and exercise good engineering judgment about when to deploy, every day of the week
Every team has to make their own determination about which policies and norms are right given their circumstances and risk tolerance
A policy of “no Friday deploys” may be reasonable for now but should be seen as a smell, a sign that your deploys are risky. It is also likely to make things WORSE for you, not better, by causing you to adopt other risky practices (e.g. elongating the interval between merge and deploy, batching changes up in a single deploy)
This has been the most frustrating thing about this conversation: that a) I am not in fact the absolutist y’all are arguing against, and b) MY number one priority is engineers and their work/life balance. Which makes this particularly aggravating:
Lastly there is some strange argument that choosing not to deploy on Friday “Shouldn’t be a source of glee and pride”. That one I haven’t figured out yet, because I have always had a lot of glee and pride in being extremely (overly?) protective of the work/life balance of the engineers who either work for me, or with me. I don’t expect that to change.
Hold up. Did you catch that clever little logic switcheroo? You defined “not deploying on Friday” as being a priori synonymous with “protecting the work/life balance of engineers”. This is how I know you haven’t actually grasped my point, and are arguing against a straw man. My entire point is that the behaviors and practices associated with blocking Friday deploys are in fact hurting your engineers.
I, too, take a lot of glee and pride in being extremely, massively, yes even OVERLY protective of the work/life balance of the engineers who either work for me, or with me.
AND THAT IS WHY WE DEPLOY ON FRIDAYS.
Because it is BETTER for them. Because it is part of a deploy ecosystem which results in them being woken up less and having fewer weekends interrupted overall than if I had blocked deploys on Fridays.
It’s not about Fridays. It’s about having a healthy ecosystem and feedback loop where you trust your deploys, where deploys aren’t a big deal, and they never cause engineers to have to work outside working hours. And part of how you get there is by not artificially blocking off a big bunch of the week and not deploying during that time, because that breaks up your virtuous feedback loop and causes your deploys to be much more likely to fail in terrible ways.
The other thing that annoys me is when people say, primly, “you can’t guarantee any deploy is safe, but you can guarantee people have plans for the weekend.”
Know what else you can guarantee? That people would like to sleep through the fucking night, even on weeknights.
When I hear people say this all I hear is that they don’t care enough to invest the time to actually fix their shit so it won’t wake people up or interrupt their off time, seven days a week. Enough with the virtue signaling already.
You cannot have it both ways, where you block off a bunch of undeployable time AND you have robust, resilient, swift deploys. Somehow I keep not getting this core point across to a substantial number of very intelligent people. So let me try a different way.
Let’s try telling a story.
A tale of two startups
Here are two case studies.
Company X is a three-year-old startup. It is a large, fast-growing multi-tenant platform on a large distributed system with spiky traffic, lots of user-submitted data, and a very green database. Company X deploys the API about once per day, and does a global deploy of all services every Tuesday. Deploys often involve some firefighting and a rollback or two, and Tuesdays often involve deploying and reverting all day (sigh).
Pager volume at Company X isn’t the worst, but usually involves getting woken up a couple times a week, and there are deploy-related alerts after maybe a third of deploys, which then need to be triaged to figure out whose diff was the cause.
Company Z is a three-year-old startup. It is a large, fast-growing multi-tenant platform on a large distributed system with spiky traffic, lots of user-submitted data, and a very green house-built distributed storage engine. Company Z automatically triggers a deploy within 30 minutes of a merge to master, for all services impacted by that merge. Developers at company Z practice observability-driven deployment, where they instrument all changes, ask “how will I know if this change doesn’t work?” during code review, and have a muscle memory habit of checking to see if their changes are working as intended or not after they merge to master.
Deploys rarely result in the pager going off at Company Z; most problems are caught visually by the engineer and reverted or fixed before any paging alert can fire. Pager volume consists of roughly one alert per week outside of working hours, and no one is woken up more than a couple times per year.
Same damn problem, better damn solutions.
If it wasn’t extremely obvious, these companies are my last two jobs, Parse (company X, from 2012-2016) and Honeycomb (company Z, from 2016-present).
They have a LOT in common. Both are services for developers, both are platforms, both are running highly elastic microservices written in golang, both get lots of spiky traffic and store lots of user-defined data in a young, homebrewed columnar storage engine. They were even built by some of the same people (I built infra for both, and they share four more of the same developers).
At Parse, deploys were run by ops engineers because of how common it was for there to be some firefighting involved. We discouraged people from deploying on Fridays, we locked deploys around holidays and big launches. At Honeycomb, none of these things are true. In fact, we literally can’t remember a time when it was hard to debug a deploy-related change.
What’s the difference between Company X and Company Z?
So: what’s the difference? Why are the two companies so dramatically different in the riskiness of their deploys, and the amount of human toil it takes to keep them up?
I’ve thought about this a lot. It comes down to three main things.
Single merge per deploy
I think that I’ve been reluctant to hammer this home as much as I ought to, because I’m exquisitely sensitive about sounding like an obnoxious vendor trying to sell you things. 😛 (Which has absolutely been detrimental to my argument.)
When I say observability, I mean in the precise technical definition as I laid out in this piece: with high cardinality, arbitrarily wide structured events, etc. Metrics and other generic telemetry will not give you the ability to do the necessary things, e.g. break down by build id in combination with all your other dimensions to see the world through the lens of your instrumentation. Here, for example, are all the deploys for a particular service last Friday:
Each shaded area is the duration of an individual deploy: you can see the counters for each build id, as the new versions replace the old ones,
2. Observability-driven development.
This is cultural as well as technical. By this I mean instrumenting a couple steps ahead of yourself as you are developing and shipping code. I mean making a cultural practice of asking each other “how will you know if this is broken?” during code review. I mean always going and looking at your service through the lens of your instrumentation after every diff you ship. Like muscle memory.
3. Single merge per deploy.
The number one thing you can do to make your deploys intelligible, other than observability and instrumentation, is this: deploy one changeset at a time, as swiftly as possible after it is merged to master. NEVER glom multiple changesets into a single deploy — that’s how you get into a state where you aren’t sure which change is at fault, or who to escalate to, or if it’s an intersection of multiple changes, or if you should just start bisecting blindly to try and isolate the source of the problem. THIS is what turns deploys into long, painful marathons.
And NEVER wait hours or days to deploy after the change is merged. As a developer, you know full well how this goes. After you merge to master one of two things will happen. Either:
you promptly pull up a window to watch your changes roll out, checking on your instrumentation to see if it’s doing what you intended it to or if anything looks weird, OR
you close the project and open a new one.
When you switch to a new project, your brain starts rapidly evicting all the rich context about what you had intended to do and and overwriting it with all the new details about the new project.
Whereas if you shipped that changeset right after merging, then you can WATCH it roll out. And 80-90% of all problems can be, should be caught right here, before your users ever notice — before alerts can fire off and page you. If you have the ability to break down by build id, zoom in on any errors that happen to arise, see exactly which dimensions all the errors have in common and how they differ from the healthy requests, see exactly what the context is for any erroring requests.
Healthy feedback loops == healthy systems.
That tight, short feedback loop of build/ship/observe is the beating heart of a healthy, observable distributed system that can be run and maintained by human beings, without it sucking your life force or ruining your sleep schedule or will to live.
Most engineers have never worked on a system like this. Most engineers have no idea what a yawning chasm exists between a healthy, tractable system and where they are now. Most engineers have no idea what a difference observability can make. Most engineers are far more familiar with spending 40-50% of their week fumbling around in the dark, trying to figure out where in the system is the problem they are trying to fix, and what kind of context do they need to reproduce.
Most engineers are dealing with systems where they blindly shipped bugs with no observability, and reports about those bugs started to trickle in over the next hours, days, weeks, months, or years. Most engineers are dealing with systems that are obfuscated and obscure, systems which are tangled heaps of bugs and poorly understood behavior for years compounding upon years on end.
That’s why it doesn’t seem like such a big deal to you break up that tight, short feedback loop. That’s why it doesn’t fill you with horror to think of merging on Friday morning and deploying on Monday. That’s why it doesn’t appall you to clump together all the changes that happen to get merged between Friday and Monday and push them out in a single deploy.
It just doesn’t seem that much worse than what you normally deal with. You think this raging trash fire is, unfortunately … normal.
How realistic is this, though, really?
Maybe you’re rolling your eyes at me now. “Sure, Charity, that’s nice for you, on your brand new shiny system. Ours has years of technical debt, It’s unrealistic to hold us to the same standard.”
Yeah, I know. It is much harder to dig yourself out of a hole than it is to not create a hole in the first place. No doubt about that.
Harder, yes. But not impossible.
I have done it.
Parse in 2013 was a trash fire. It woke us up every night, we spent a lot of time stabbing around in the dark after every deploy. But after we got acquired by Facebook, after we started shipping some data sets into Scuba, after (in retrospect, I can say) we had event-level observability for our systems, we were able to start paying down that debt and fixing our deploy systems.
We started hooking up that virtuous feedback loop, step by step.
We reworked our CI/CD system so that it built a new artifact after every single merge.
We put developers at the steering wheel so they could push their own changes out.
We got better at instrumentation, and we made a habit of going to look at it during or after each deploy.
We hooked up the pager so it would alert the person who merged the last diff, if an alert was generated within an hour after that service was deployed.
We started finding bugs quicker, faster, and paying down the tech debt we had amassed from shipping code without observability/visibility for many years.
Developers got in the habit of shipping their own changes, and watching them as they rolled out, and finding/fixing their bugs immediately.
It took some time. But after a year of this, our formerly flaky, obscure, mysterious, massively multi-tenant service that was going down every day and wreaking havoc on our sleep schedules was tamed. Deploys were swift and drama-free. We stopped blocking deploys on Fridays, holidays, or any other days, because we realized our systems were more stable when we always shipped consistently and quickly.
Allow me to repeat. Our systems were more stable when we always shipped right after the changes were merged. Our systems were less stable when we carved out times to pause deployments. This was not common wisdom at the time, so it surprised me; yet I found it to be true over and over and over again.
This is literally why I started Honeycomb.
When I was leaving Facebook, I suddenly realized that this meant going back to the Dark Ages in terms of tooling. I had become so accustomed to having the Parse+scuba tooling and being able to iteratively explore and ask any question without having to predict it in advance. I couldn’t fathom giving it up.
The idea of going back to a world without observability, a world where one deployed and then stared anxiously at dashboards — it was unthinkable. It was like I was being asked to give up my five senses for production — like I was going to be blind, deaf, dumb, without taste or touch.
Look, I agree with nearly everything in the author’s piece. I could have written that piece myself five years ago.
But since then, I’ve learned that systems can be better. They MUST be better. Our systems are getting so rapidly more complex, they are outstripping our ability to understand and manage them using the past generation of tools. If we don’t change our ways, it will chew up another generation of engineering lives, sleep schedules, relationships.
Observability isn’t the whole story. But it’s certainly where it starts. If you can’t see where you’re going, you can’t go very far.
Get you some observability.
And then raise your standards for how systems should feel, and how much of your human life they should consume. Do better.
Because I couldn’t agree with that other post more: it really is all about people and their real lives.
Listen, if you can swing a four day work week, more power to you (most of us can’t). Any day you aren’t merging code to master, you have no need to deploy either. It’s not about Fridays; it’s about the swift, virtuous feedback loop.
And nobody should be shamed for what they need to do to survive, given the state of their systems today.
But things aren’t gonna get better unless you see clearly how you are contributing to your present pain. And congratulating ourselves for blocking Friday deploys is like congratulating ourselves for swatting ourselves in the face with the flyswatter. It’s a gross hack.
Maybe you had a good reason. Sure. But I’m telling you, if you truly do care about people and their work/life balance: we can do a lot better.
I made a vow this year to post one blog post a month, then I didn’t post anything at all from May to September. I have some catching up to do. 😑 I’ve also been meaning to transcribe some of the twitter rants that I end up linking back to into blog posts, so if there’s anything you especially want me to write about, tell me now while I’m in repentance mode.
This is one request I happened to make a note of because I can’t believe I haven’t already written it up! I’ve been saying the same thing over and over in talks and on twitter for years, but apparently never a blog post.
The question is: what is the proper role of alerting in the modern era of distributed systems? Has it changed? What are the updated best practices for alerting?
@mipsytipsy I've seen your thoughts on dashboards vs searching but haven't seen many thoughts from you on Alerting. Let me know if I've missed a blog somewhere on that! 🙂
It’s a great question. I want to wax philosophically about some stuff, but first let me briefly outline the way to modernize your alerting best practices:
implement SLOs and/or end-to-end checks that traverse key code paths and correlate to user-impacting events
create a secondary channel (tasks, ticketing system, whatever) for “things that on call should look at soon, but are not impacting users yet” which does not page anyone, but which on call is expected to look at (at least) first thing in the morning, last thing in the evening, and midday
move as many paging alerts as possible to the secondary channel, by engineering your services to auto-remediate or run in degraded mode until they can be patched up
wake people up only for SLOs and health checks that correlate to user-impacting events
Or, in an even shorter formulation: delete all your paging alerts, then page only on e2e alerts that mean users are in pain. Rely on debugging tools for debugging, and paging only when users are in pain.
To understand why I advocate deleting all your paging alerts, and when it’s safe to delete them, first we need to understand why have we accumulated so many crappy paging alerts over the years.
Monoliths, LAMP stacks, and death by pagebomb
Here, let’s crib a couple of slides from one of my talks on observability. Here are the characteristics of older monolithic LAMP-stack style systems, and best practices for running them:
The sad truth is, that when all you have is time series aggregates and traditional monitoring dashboards, you aren’t really debugging with science so much as you are relying on your gut and a handful of dashboards, using intuition and scraps of data to try and reconstruct an impossibly complex system state.
This works ok, as long as you have a relatively limited set of failure scenarios that happen over and over again. You can just pattern match from past failures to current data, and most of the time your intuition can bridge the gap correctly. Every time there’s an outage, you post mortem the incident, figure out what happened, build a dashboard “to help us find the problem immediately next time”, create a detailed runbook for how to respond to it, and (often) configure a paging alert to detect that scenario.
Over time you build up a rich library of these responses. So most of the time when you get paged you get a cluster of pages that actually serves to help you debug what’s happening. For example, at Parse, if the error graph had a particular shape I immediately knew it was a redis outage. Or, if I got paged about a high % of app servers all timing out in a short period of time, I could be almost certain the problem was due to mysql connections. And so forth.
Things fall apart; the pagebomb cannot stand
However, this model falls apart fast with distributed systems. There are just too many failures. Failure is constant, continuous, eternal. Failure stops being interesting. It has to stop being interesting, or you will die.
Instead of a limited set of recurring error conditions, you have an infinitely long list of things that almost never happen …. except that one time they do. If you invest your time into runbooks and monitoring checks, it’s wasted time if that edge case never happens again.
Frankly, any time you get paged about a distributed system, it should be a genuinely new failure that requires your full creative attention. You shouldn’t just be checking your phone, going “oh THAT again”, and flipping through a runbook. Every time you get paged it should be genuinely new and interesting.
Oh damn this talk looks baller. 😍 "Failure is important, but it is no longer interesting" — @this_hits_home… Netflix once again shining the light on where the rest of us need to get to over the next 3-5 years. 🙌🏅🎬 https://t.co/OY40Y0BTSa
And thus you should actually have drastically fewer paging alerts than you used to.
A better way: observability and SLOs.
Instead of paging alerts for every specific failure scenario, the technically correct answer is to define your SLOs (service level objectives) and page only on those, i.e. when you are going to run out of budget ahead of schedule. But most people aren’t yet operating at this level of sophistication. (SLOs sound easy, but are unbelievably challenging to do well; many great teams have tried and failed. This is why we have built an SLO feature into Honeycomb that does the heavy lifting for you. Currently alpha testing with users.)
If you haven’t yet caught the SLO religion, the alternate answer is that “you should only page on high level end-to-end alerts, the ones which traverse the code paths that make you money and correspond to user pain”. Alert on the three golden signals: request rate, latency, and errors, and make sure to traverse every shard and/or storage type in your critical path.
That’s it. Don’t alert on the state of individual storage instances, or replication, or anything that isn’t user-visible.
(To be clear: by “alert” I mean “paging humans at any time of day or night”. You might reasonably choose to page people during normal work hours, but during sleepy hours most errors should be routed to a non-paging address. Only wake people up for actual user-visible problems.)
Here’s the thing. The reason we had all those paging alerts was because we depended on them to understand our systems.
Once you make the shift to observability, once you have rich instrumentation and the ability to swiftly zoom in from high level “there might be a problem” to identifying specifically what the errors have in common, or the source of the problem — you no longer need to lean on that scattershot bunch of pagebombs to understand your systems. You should be able to confidently ask any question of your systems, understand any system state — even if you have never encountered it before.
With observability, you debug by systematically following the trail of crumbs back to their source, whatever that is. Those paging alerts were a crutch, and now you don’t need them anymore.
Everyone is on call && on call doesn’t suck.
I often talk about how modern systems require software ownership. The person who is writing the software, who has the original intent in their head, needs to shepherd that code out into production and watch real users use it. You can’t chop that up into multiple roles, dev and ops. You just can’t. Software engineers working on highly available systems need to be on call for their code.
But the flip side of this responsibility belongs to management. If you’re asking everyone to be on call, it is your sworn duty to make sure that on call does not suck. People shouldn’t have to plan their lives around being on call. People shouldn’t have to expect to be woken up on a regular basis. Every paging alert out of hours should be as serious as a heart attack, and this means allocating real engineering resources to keeping tech debt down and noise levels low.
And the way you get there is first invest in observability, then delete all your paging alerts and start over from scratch.
I hadn’t seen anyone say something like this in quite a while. I remember saying things like this myself as recently as, oh, 2016, but I thought the zeitgeist had moved on to continuous delivery.
Which is not to say that Friday freezes don’t happen anymore, or even that they shouldn’t; I just thought that this was no longer seen as a badge of responsibility and honor, rather a source of mild embarrassment. (Much like the fact that you still don’t automatedly restore your db backups and verify them every night. Do you.)
So I responded with an equally hyperbolic and indefensible claim:
If you're scared of pushing to production on Fridays, I recommend reassigning all your developer cycles off of feature development and onto your CI/CD process and observability tooling for as long as it takes to ✨fix that✨.
Now obviously, OBVIOUSLY, reassigning all your developer cycles is probably a terrible idea. You don’t get 100x parallel efficiency if you put 100 developers on a single problem. So I thought it was clear that this said somewhat tongue in cheek, serious-but-not-really. I was wrong there too.
So let me explain.
There’s nothing morally “wrong” with Friday freezes. But it is a costly and cumbersome bandage for a problem that you would be better served to address directly. And if your stated goal is to protect people’s off hours, this strategy is likely to sabotage that goal and cause them to waste far more time and get woken up much more often, and it stunts your engineers’ technical development on top of that.
Fear is the mind-killer.
Fear of deploys is the ultimate technical debt. How much time does your company waste, between engineers:
waiting until it is “safe” to deploy,
batching up changes into bigger changes that are decidedly unsafe to deploy,
debugging broken deploys that had many changes batched into them,
waiting nervously to get paged after a deploy goes out,
figuring out if now is a good time to deploy or not,
cleaning up terrible deploy-related catastrophuckes
Anxiety related to deploys is the single largest source of technical debt in many, many orgs. Technical debt, lest we forget, is not the same as “bad code”. Tech debt hurts your people.
Saying “don’t push to production” is a code smell. Hearing it once a month at unpredictable intervals is concerning. Hearing it EVERY WEEK for an ENTIRE DAY OF THE WEEK should be a heartstopper alarm. If you’ve been living under this policy you may be numb to its horror, but just because you’re used to hearing it doesn’t make it any less noxious.
If you’re used to hearing it and saying it on a weekly basis, you are afraid of your deploys and you should fix that.
It’s a smell. If you can’t deploy at 6pm on a Friday it means you don’t understand or don’t trust your systems or process. That might be ok if you openly acknowledge it, but if your not talking about it, that’s dangerous
If you are a software company, shipping code is your heartbeat. Shipping code should be as reliable and sturdy and fast and unremarkable as possible, because this is the drumbeat by which value gets delivered to your org.
Deploys are the heartbeat of your company.
Every time your production pipeline stops, it is a heart attack. It should not be ok to go around nonchalantly telling people to halt the lifeblood of their systems based on something as pedestrian as the day of the week.
Why are you afraid to push to prod? Usually it boils down to one or more factors:
your deploys frequently break, and require manual intervention just to get to a good state
your test coverage is not good, your monitoring checks are not good, so you rely on users to report problems back to you and this trickles in over days
recovering from deploys gone bad can regularly cause everything to grind to a halt for hours or days while you recover, so you don’t want to even embark on a deploy without 24 hours of work day ahead of you
your deploys are painfully slow, and take hours to run tests and go live.
These are pretty darn good reasons. If this is the state you are in, I totally get why you don’t want to deploy on Fridays. So what are you doing to actively fix those states? How long do you think these emergency controls will be in effect?
The answers of “nothing” and “forever” are unacceptable. These are eminently fixable problems, and the amount of drag they create on your engineering team and ability to execute are the equivalent of five-alarm fires.
Fix. That. Take some cycles off product and fix your fucking deploy pipeline.
It is difficult to combine "you shouldn't do this because it is a symptom of a systemic problem, you should instead address the systemic problem" with "yes actually you should do it for as long as the systemic problem exists, because it is adaptive".
If you’ve been paying attention to the DORA report or Accelerate, you know that the way you address the problem of flaky deploys is NOT by slowing down or adding roadblocks and friction, but by shipping more QUICKLY.
Science says: ship fast, ship often.
Deploy on every commit. Smaller, coherent changesets transform into debuggable, understandable deploys. If we’ve learned anything from recent research, it’s that velocity of deploys and lowered error rates are not in tension with each other, they actually reinforce each other. When one gets better, the other does too.
So by slowing down or batching up or pausing your deploys, you are materially contributing to the worsening of your own overall state.
If you block devs from merging on Fridays, then you are sacrificing a fifth of your velocity and overall output. That’s a lot of fucking output.
If you do not block merges on Fridays, and only block deploys, you are queueing up a bunch of changes to all get shipped days later, long after the engineers wrote the code and have forgotten half of the context. Any problems you encounter will be MUCH harder to debug on Monday in a muddled blob of changes than they would have been just shipping crisply, one at a time on Friday. Is it worth sacrificing your entire Monday? Monday-Tuesday? Monday-Tuesday-Wednesday?
The worst, most PTSD-inducing outages of my life have all happened after holiday code freezes. Every. Single. One.
Don't use rules. Practice good judgment, build tools to align incentives and deploy often; practice tolerating and recovering from pedestrian failures often too. https://t.co/GH7lef274z
I am not saying that you should make a habit of shipping a large feature at 4:55 pm on Friday and then sauntering out the door at 5. For fucks sake. Every engineer needs to learn and practice good technical judgment around deploy hygiene. LIke,
Don’t ship before you walk out the door on *any* day.
Don’t ship big, gnarly features right before the weekend, if you aren’t going to be around to watch them.
Instrument your code, and go and LOOK at the damn thing once it’s live.
Use feature flags and other tools that separate turning on code paths from deploys.
But you don’t need rules for this; in fact, rules actually inhibit the development of good judgment!
Policies (and enumerated exceptions to policies, and exceptions to exceptions) are a piss-poor substitute for judgment. Rules are blunt instruments that stunt your engineers' development and critical thinking skills.
Most deploy-related problems are readily obvious, if the person who has the context for the change in their heads goes and looks at it.
But if you aren’t looking for them, then sure — you probably won’t find out until user reports start to trickle in over the next few days.
So go and LOOK.
Stop shipping blind. Actually LOOK at what you ship.
I mean, if it takes 48 hours for a bug to show up, then maybe you better freeze deploys on Thursdays too, just to be safe! 🙄
I get why this seems obvious and tempting. The “safety” of nodeploy Friday is realized immediately, while the costs are felt later later. They’re felt when you lose Monday (and Tuesday) to debugging the big blob deplly. Or they get amortized out over time. Or you experience them as sluggish ship rates and a general culture of fear and avoidance, or learned helplessness, and the broad acceptance of fucked up situations as “normal”.
If pushing to production is a painful event, make it a *non-event* (same philosophy as chaos engineering)
But if recovering from deploys is long and painful and hard, then you should fix that. If you don’t tend to detect reliability events until long after the event, you should fix that. If people are regularly getting paged on Saturdays and Sundays, they are probably getting paged throughout the night, too. You should fix that.
On call paging events should be extremely rare. There’s no excuse for on call being something that significantly impacts a person’s life on the regular. None.
I’m not saying that every place is perfect, or that every company can run like a tech startup. I am saying that deploy tooling is systematically underinvested in, and we abuse people far too much by paging them incessantly and running them ragged, because we don’t actually believe it can be any better.
It can. If you work towards it.
Devote some real engineering hours to your deploy pipeline, and some real creativity to your processes, and someday you too can lift the Friday ban on deploys and relieve your oncall from burnout and increase your overall velocity and productivity.
On virtue signaling
Finally, I heard from a alarming number of people who admitted that Friday deploy bans were useless or counterproductive, but they supported them anyway as a purely symbolic gesture to show that they supported work/life balance.
This makes me really sad. I’m … glad they want to support work/life balance, but surely we can come up with some other gestures that don’t work directly counter to their goals of life/work balance.
That's it. Because if you make it a virtue signal, it will NEVER GET FIXED. Blocking Friday deploy is not a mark of moral virtue; it is a physical bash script patching over technical rot.
And technical rot is bad because it HURTS PEOPLE. It is in your interest to fix it.
Ways to begin recovering from a toxic deploy culture:
Have a deploy philosophy, make sure everybody knows what it is. Be consistent.
Build and deploy on every set of committed changes. Do not batch up multiple people’s commits into a deploy.
Train every engineer so they can run their own deploys, if they aren’t fully automated. Make every engineer responsible for their own deploys.
(Work towards fully automated deploys.)
Every deploy should be owned by the developer who made the changes that are rolling out. Page the person who committed the change that triggered the deploy, not whoever is oncall.
Set expectations around what “ownership” means. Provide observability tooling so they can break down by build id and compare the last known stable deploy with the one rolling out.
Never accept a diff if there’s no explanation for the question, “how will you know when this code breaks? how will you know if the deploy is not behaving as planned?” Instrument every commit so you can answer this question in production.
Shipping software and running tests should be fast. Super fast. Minutes, tops.
It should be muscle memory for every developer to check up on their deploy and see if it is behaving as expected, and if anything else looks “weird”.
Practice good deploy hygiene using feature flags. Decouple deploys from feature releases. Empower support and other teams to flip flags without involving engineers.
Each deploy should be owned by the developer who made the code changes. But your deploy pipeline needs to have a team that owns it too. I recommend putting your most experienced, senior developers on this problem to signal its high value.
Ultimately, I am not dogmatic about Friday deploys. Truly, I’m not. If that’s the only lever you have to protect your time, use it. But call it and treat it like the hack it is. It’s a gross workaround, not an ideal state.
Don’t let your people settle into the idea that it’s some kind of moral stance instead of a butt-ugly hack. Because if you do you will never, ever get rid of it.
There are plenty of good reasons to block deploys on Fridays, but it's not a good policy to cargo cult blindly. It imposes real costs and ultimately hinders you from achieving safe, boring deploys.
Remember: a team’s maturity and efficiency can be represented by how long it takes to get their shit into users’ hands after they write it. Ship it fast, while it’s still fresh in your developers’ heads. Ship one change set at a time, so you can swiftly debug and revert them. I promise your lives will be so much better. Every step helps. <3
You’ve convinced the security team and other stakeholders, you’ve gotten the integration running, you’re getting promising results from dev-test or staging environments… now it’s time to move from proof-of-concept to full implementation.Depending on your situation this might be a transition from staging to production, or it might mean increasing a feature flipper flag from 5% to 100%, or it might mean increasing coverage of an integration from one API endpoint to cover your entire developer footprint.
Taking into account Murphy’s Law, we expect that some things will go wrong during the rollout.Perhaps during coverage, a developer realizes that the schema designed to handle the app’s event mechanism can’t represent a scenario, requiring a redesign or a hacky solution.Or perhaps the metrics dashboard shows elevated error rates from the API frontend, and while there’s no smoking gun, the ops oncall decides to rollback the integration Just In Case it’s causing the incident.
This gives us another chance to practice empathy — while it’s easy, wearing the champion hat, to dismiss any issues found by looking for someone to blame, ultimately this poisons trust within your organization and will hamper success.It’s more effective, in the long run (and often even in the short run), to find common ground with your peers in other disciplines and teams, and work through to solutions that satisfy everybody.
Keeping the lights on
In all likelihood as integration succeeds, the team will rapidly develop experts and expertise, as well as idiomatic ways to use the product.Let the experts surprise you; folks you might not expect can step up when given a chance.Expertise flourishes when given guidance and goals; as the team becomes comfortable with the integration, explicitly recognize a leader or point person for each vendor relationship.Having one person explicitly responsible for a relationship lets them pay attention to those vendor emails, updates, and avoid the tragedy of the “but I thought *you* were” commons.This Integration Lead is also a center of knowledge transfer for your organization — they won’t know everything or help every user come up to speed, but they can help empower the local power users in each team to ramp up their teams on the integration.
As comfort grows you will start to consider ways to change your usage, for example growing into new kinds of data.This is a good time to revisit that security checklist — does the change increase PII exposure to your vendor?Would the new data lead to additional requirements such as per-field encryption?Don’t let these security concerns block you from gaining valuable insight using the new tool, but do take the chance to talk it over with your security experts as appropriate.
Throughout this organic growth, the Integration Lead remains core to managing your changing profile of usage of the vendor they shepherd; as new categories of data are added to the integration, the Lead has responsibility to ensure that the vendor relationship and risk profile are well matched to the needs that the new usage (and presumably, business value) is placing on the relationship.
Documenting the Intergation Lead role and responsibilities is critical. The team should know when to check in, and writing it down helps it happen.When new code has a security implication, or a new use case potentially amplifies the cost of an integration, bringing the domain expert in will avoid unhappy surprises.Knowing how to find out who to bring in, and when to bring them in, will keep your team getting the right eyes on their changes.
Security threats and other challenges change over time, too.Collaborating with your security team so that they know what systems are in use helps your team take note of new information that is relevant to your business. A simple example is noting when your vendors publish a breach announcement, but more complex examples happen too — your vendor transitions cloud providers from AWS to Azure and the security team gets an alert about unexpected data flows from your production cluster; with transparency and trust such events become part of a routine process rather than an emergency.
It’s all operational
Monitoring and alerting is a fact of operations life, and this has to include vendor integrations (even when the vendor integration is a monitoring product.)All of your operations best practices are needed here — keep your alerts clean and actionable so that you don’t develop pager fatigue, and monitor performance of the integration so that you don’t get blindsided by a creeping latency monster in your APIs.
Authentication and authorization are changing as the threat landscape evolves and industry moves from SMS verification codes to U2F/WebAuthn.Does your vendor support your SSO integration?If they can’t support the same SSO that you use everywhere else and can’t add it — or worse, look confused when you mention SSO — that’s probably a sign you should consider a different vendor.
A beautiful sunset
Have a plan beforehand for what needs to be done should you stop using the service.Got any mobile apps that depend on APIs that will go away or start returning permission errors?Be sure to test these scenarios ahead of time.
What happens at contract termination to data stored on the service?Do you need to explicitly delete data when ceasing use?
Do you need to remove integrations from your systems before ending the commercial relationship, or can the technical shutdown and business shutdown run in parallel?
In all likelihood these are contingency plans that will never be needed, and they don’t need to be fully fleshed out to start, but a little bit of forethought can avoid unpleasant surprises.
Year after year
Industry best practice and common sense dictate that you should revisit the security questionnaire annually (if not more frequently). Use this chance to take stock of the last year and check in — are you getting value from the service?What has changed in your business needs and the competitive landscape?
It’s entirely possible that a new year brings new challenges, which could make your current vendor even more valuable (time to negotiate a better contract rate!) or could mean you’d do better with a competing service.Has the vendor gone through any major changes?They might have new offerings that suit your needs well, or they may have pivoted away from the features you need.
Check in with your friends on the security team as well; standards evolve, and last year’s sufficient solution might not be good enough for new requirements.
Andy thinks out loud about security, society, and the problems with computers on Twitter.
❤️ Thanks so much reading, folks. Please feel free to drop any complaints, comments, or additional tips to us in the comments, or direct them to me on twitter.
All this pain will someday be worth it. 🙏❤️ charity + friends
“Get Aligned With Security”
by Lilly Ryan
If your team has decided on a third-party service to help you gather data and debug product issues, how do you convince an often overeager internal security team to help you adopt it?
When this service is something that provides a pathway for developers to access production data, as analytics tools often do, making the case for access to that data can screech to a halt at the mention of the word “production”. Progressing past that point will take time, empathy, and consideration.
I have been on both sides of the “adopting a new service” fence: as a developer hoping to introduce something new and useful to our stack, and now as a security professional who spends her days trying to bust holes in other people’s setups. I understand both sides of the sometimes-conflicting needs to both ship software and to keep systems safe.
This guide has advice to help you solve the immediate problem of choosing and deploying a third-party service with the approval of your security team. But it also has advice for how to strengthen the working relationship between your security and development teams over the longer term. No two companies are the same, so please adapt these ideas to fit your circumstances.
Understanding the security mindset
The biggest problems in technology are never really about technology, but about people. Seeing your security team as people and understanding where they are coming from will help you to establish empathy with them so that both of you want to help each other get what you want, not block each other.
First, understand where your security team is coming from. Development teams need to build features, improve the product, understand and ship good code. Security teams need to make sure you don’t end up on the cover of the NYT for data breaches, that your business isn’t halted by ransomware, and that you’re not building your product on a vulnerable stack.
This can be an unfamiliar frame of mind for developers. Software development tends to attract positive-minded people who love creating things and are excited about the possibilities of new technology. Software security tends to attract negative thinkers who are skilled at finding all the flaws in a system. These are very different mentalities, and the people who occupy them tend to have very different assumptions, vocabularies, and worldviews.
But if you and your security team can’t share the same worldview, it will be hard to trust each other and come to agreement. This is where practicing empathy can be helpful.
Before approaching your security team with your request to approve a new vendor, you may want to run some practice exercises for putting yourselves in their shoes and forcing yourselves to deliberately cultivate a negative thinking mindset to experience how they may react — not just in terms of the objective risk to the business, or the compliance headaches it might cause, but also what arguments might resonate with them and what emotional reactions they might have.
My favourite exercise for getting teams to think negatively is what I call the Land Astronaut approach.
The “Land Astronaut” Game
Imagine you are an astronaut on the International Space Station. Literally everything you do in space has death as a highly possible outcome. So astronauts spend a lot of time analysing, re-enacting, and optimizing their reactions to events, until it becomes muscle memory. By expecting and training for failure, astronauts use negative thinking to anticipate and mitigate flaws before they happen. It makes their chances of survival greater and their people ready for any crisis.
Your project may not be as high-stakes as a space mission, and your feet will most likely remain on the ground for the duration of your work, but you can bet your security team is regularly indulging in worst-case astronaut-type thinking. You and your team should try it, too.
Pick a service for you and your team to game out. Schedule an hour, book a room with a whiteboard, put on your Land Astronaut helmets. Then tell your team to spend half an hour brainstorming about all the terrible things that can happen to that service, or to the rest of your stack when that service is introduced. Negative thoughts only!
Start brainstorming together. Start out by being as outlandish as possible (what happens if their data centre is suddenly overrun by a stampede of elephants?). Eventually you will find that you’ll tire of the extreme worst case scenarios and come to consider more realistic outcomes — some of which which you may not have thought of outside of the structure of the activity.
After half an hour, or whenever you feel like you’re all done brainstorming, take off your Land Astronaut helmets, sift out the most plausible of the worst case scenarios, and try to come up with answers or strategies that will help you counteract them. Which risks are plausible enough that you should mitigate them? Which are you prepared to gamble on never happening? How will this risk calculus change as your company grows and takes on more exposure?
Doing this with your team will allow you all to practice the negative thinking mindset together and get a feel for how your colleagues in the security team might approach this request. (While this may seem similar to threat modelling exercises you might have done in the past, the focus here is on learning to adopt a security mindset and gaining empathy for this thought process, rather than running through a technical checklist of common areas of concern.)
While you still have your helmets within reach, use your negative thinking mindset to fill out the spreadsheet from the first piece in this series. This will help you anticipate most of the reasonable objections security might raise, and may help you include useful detail the security team might not have known to ask for.
Once you have prepared your list of answers to George’s worksheet and held a team Land Astronaut session together, you will have come most of the way to getting on board with the way your security team thinks.
Preparing for compromise
You’ve considered your options carefully, you’ve learned how to harness negative thinking to your advantage, and you’re ready to talk to your colleagues in security – but sometimes, even with all of these tools at your disposal, you may not walk away with all of the things you are hoping for.
Being willing to compromise and anticipating some of those compromises before you approach the security team will help you negotiate more successfully.
While your Land Astronaut helmets are still within reach, consider using your negative thinking mindset game to identify areas where you may be asked to compromise. If you’re asking for production access to this new service for observability and debugging purposes, think about what kinds of objections may be raised about this and how you might counter them or accommodate them. Consider continuing the activity with half of the team remaining in the Land Astronaut role while the other half advocates from a positive thinking standpoint. This dynamic will get you having conversations about compromise early on, so that when the security team inevitably raises eyebrows, you are ready with answers.
Be prepared to consider compromises you had not anticipated, and enter into discussions with the security team with as open a mind as possible. Remember the team is balancing priorities of not only your team, but other business and development teams as well. If you and your security colleagues are doing the hard work to meet each other halfway then you are more likely to arrive at a solution that satisfies both parties.
Working together for the long term
While the previous strategies we’ve covered focus on short-term outcomes, in this continuous-deployment, shift-left world we now live in, the best way to convince your security team of the benefits of a third-party service – or any other decision – is to have them along from day one, as part of the team.
Roles and teams are increasingly fluid and boundary-crossing, yet security remains one of the roles least likely to be considered for inclusion on a software development team. Even in 2019, the task of ensuring that your product and stack are secure and well-defended is often left until the end of the development cycle. This contributes a great deal to the combative atmosphere that is common.
Bringing security people into the development process much earlier builds rapport and prevents these adversarial, territorial dynamics. Consider working together to build Disaster Recovery plans and coordinating for shared production ownership.
If your organisation isn’t ready for that kind of structural shift, there are other ways to work together more closely with your security colleagues.
Try having members of your team spend a week or two embedded with the security team. You may even consider a rolling exchange – a developer for a security team member – so that developers build the security mindset, and the security team is able to understand the problems your team is facing (and why you are looking at introducing this new service).
At the very least, you should make regular time to meet with the security team, get to know them as people, and avoid springing things on them late in the project when change is hardest.
Riding off together into the sunset…?
If you’ve taken the time to get to know your security team and how they think, you’ll hopefully be able to get what you want from them – or perhaps you’ll understand why their objections were valid, and come up with a better solution that works well for both of you.
Investing in a strong relationship between your development and security teams will rarely lead to the apocalypse. Instead, you’ll end up with a better product, probably some new work friends, and maybe an exciting idea for a boundary-crossing new career in tech.
But this story isn’t over! Once you get the green light from security, you’ll need to think about how to roll your new service out safely, maintain it, and consider its full lifespan within your company. Which leads us to part three of this series, on rolling it out and maintaining it … both your integration and your relationship with the security team.
Lilly Ryan is a pen tester, Python wrangler, and recovering historian from Melbourne. She writes and speaks internationally about ethical software, social identities after death, teamwork, and the telegraph. More recently she has researched the domestic use of arsenic in Victorian England, attempted urban camouflage, reverse engineered APIs, wielded the Oxford comma, and baked a really good lemon shortbread.
I hear variations on this question constantly: “I’d really like to use a service like Honeycomb for my observability, but I’m told I can’t ship any data off site. Do you have any advice on how to convince my security team to let me?”
I’ve given lots of answers, most of them unsatisfactory. “Strip the PII/PHI from your operational data.” “Validate server side.” “Use our secure tenancy proxy.” (I’m not bad at security from a technical perspective, but I am not fluent with the local lingo, and I’ve never actually worked with an in-house security team — i’ve always *been* the security team, de facto as it may be.)
So I’ve invited three experts to share their wisdom in a three-part series of guest posts:
My ✨first-ever guest posts✨! Yippee. I hope these are useful to you, wherever you are in the process of outsourcing your tools. You are on the right path: outsourcing your observability to a vendor for whom it’s their One Job is almost always the right call, in terms of money and time and focus — and yes, even security.
All this pain will someday be worth it. 🙏❤️ charity + friends
“How to be a Champion”
by George Chamales
You’ve found a third party service you want to bring into your company, hooray!
To you, it’s an opportunity to deploy new features in a flash, juice your team’s productivity, and save boatloads of money.
To your security and compliance teams, it’s a chance to lose your customers’ data, cause your applications to fall over, and do inordinate damage to your company’s reputation and bottom line.
The good news is, you’re absolutely right. The bad news is, so are they.
Successfully championing a new service inside your organization will require you to convince people that the rewards of the new service are greater than the risks it will introduce (there’s a guide below to help you).
You’re convinced the rewards are real. Let’s talk about the risks.
The past year has seen cases of hackers using third party services to target everything from government agencies, to activists, to Target…again. Not to be outdone, attention-seeking security companies have been activelyhunting for companies exposing customer data then issuing splashy press releases as a means to flog their products and services.
A key feature of these name-and-shame campaigns is to make sure that the headlines are rounded up to the most popular customer – the clickbait lead “MBM Inc. Loses Customer Data” is nowhere near as catchy as “Walmart Jewelry Partner Exposes Personal Data Of 1.3M Customers.”
While there are scary stories out there, in many, many cases the risks will be outweighed by the rewards. Telling the difference between those innumerable good calls and the one career-limiting move requires thoughtful consideration and some up-front risk mitigation.
When choosing a third party service, keep the following in mind:
The security risks of a service are highly dependent on how you use it. You can adjust your usage to decrease your risk. There’s a big difference between sending a third party your server metrics vs. your customer’s personal information. Operational metrics are categorically less sensitive than, say, PII or PHI (if you have scrubbed them properly).
There’s no way to know how good a service’s security really is. History is full of compromised companies who had very pretty security pages and certifications (here’s Equifax circa September 2017). Security features are a stronger indicator, but there are a lot more moving parts that go into maintaining a service’s security.
Always weigh the risks vs. the rewards.
There’s risk no matter what you do – bringing in the service is risky, doing nothing is risky. You can only mitigate risks up to a point. Beyond that point, it’s the rewards that make risks worthwhile.
Context is critical in understanding the risks and rewards of a new service.
You can use the following guide to put things in context as you champion a new service through the gauntlet of management, security, and compliance teams. That context becomes even more powerful when you can think about the approval process from the perspective of the folks you’ll need to win over to get the okay to move forward.
In the next part of this series Lilly Ryan shares a variety of techniques to take on the perspective of your management, security and compliance teams, enabling you to constructively work through responses that can include everything from “We have concerns…” to “No” to “Oh Helllllllll No.”
Championing a new service is hard – it can be equally worthwhile. Good luck!
“A Security Guide for Third Party Services” Worksheet
Note to thoughtful service providers: You may want to fill parts of this out ahead of time and give it to your prospective customers. It will provide your champion with good fortune in the compliance wars to come. (Also available as a nicely formatted spreadsheet.)
Why this service?
This is the justification for the service – the compelling rewards that will outweigh the inevitable risks. What will be true once the service is online? Good reasons are ones that a fifth grader would understand.
Data it will / won’t collect?
Describe the classes or types of data the service will access / store and why that’s necessary for the service to operate. If there are specific types of sensitive data the service won’t collect (e.g. passwords, Personally Identifiable Information, Patient Health Information) explicitly call them out.
How is data be accessed?
Describe the process for getting data to the service. Do you have to run their code on your servers, on your customer’s computers?
Costs of NOT doing it?
This are the financial risks / liabilities of not going with this service. What’s the worst and average cost? Have you had costly problems in the past that could have been avoided if you were using this service?
Costs of doing it?
Include the cost for the service and, if possible, the amount of person-time it’s going to take to operate the service. Ideally less than the cost of not doing it.
Our Risk – how mad will important people be…
If it’s compromised.
What would happen if hackers or attention-seeking security companies publicly released the data you sent the service? Is it catastrophic or an annoyance?
When it goes down?
When this service goes down (and it will go down), will it be a minor inconvenience or will it take out your primary application and infuriate your most valuable customers?
Their Security – in order of importance
SSO & 2FA Support?
This is a security smoke test: If a service doesn’t support SSO or 2FA, it’s safe to assume that they don’t prioritize security. Also a good idea to investigate SSO support up front since some vendors charge extra for it (which is a shame).
This is another key indicator of the service’s maturity level since it takes time and effort to build in. It’s also something else they might make you pay extra for.
These aren’t guarantees of quality, but it does indicate that the company’s put in some effort and money into their processes. Check their website for general security compliance merit badges such as SOC2, ISO27001 or industry-specific things like PCI or HIPAA.
Security & privacy pages?
If there is, it means that they’re willing to publicly state that they do something about security. The more specific and detailed, the better.
Vendor’s security history?
Have there been any spectacular breaches that demonstrated a callous disregard for security, gross incompetence, or both?
Want to really poke and prod the internal security of your vendor? Ask if they can answer the following questions:
How many known vulnerabilities (CVEs) exist on your production infrastructure right now?
At what time (exactly) was the last successful backup of all your customer data completed?
What were the last three secrets accessed in the production environment?
Is it worth it?
Look back through the previous sections and ask whether it makes sense to:
* Use the 3rd party service
* Build it yourself
* Not do it at all Would a thoughtful person agree with you?
“So I’ve almost got our group at work up to Step 1 in your observability maturity model, but some of the devs that I work with want to turn OFF our lovely structured logging in prod for informational-level msgs due to their legacy philosophy (‘we only log errors in prod’). The reasons given are mostly philosophical (“I’m a dev and only interested when things error out, I don’t want any other noise in prod logs”, “I don’t want to slow my app down in prod”). Help?!?”
As I was reading this, I was itching to fly out and dive into battle with Eric. I know exactly where his opinionated devs are coming from. I used to say the same things! I even wrote a whole blog post about it.
These developers have internalized a set of rules and best practices for dealing with output data, in the context of “monolith application development in the early 2000s”.
Monolithic systems assumptions
Those systems had many common constraints and assumptions, such as:
We have a monolith service, or a very small number of services. We can model the system in our heads.
Logging is done to local disk, which can impact performance
Disks are expensive
Log lines are spat out inline with execution. A poorly placed printf can take the whole system down.
Investigation is rare, and usually means a human reading error logs.
Logging is of poor utility for understanding internal states or execution paths; you should just read the code or use a debugger. (There are few or network hops between functions.)
Logging is mostly useful for detecting certain terminal crash states or connection errors.
Monolithic logging best practices
We should be very stingy in what we log
Debuggers should be used for understanding internal states of the code
Logs are a last resort and record of crash dumps. We do not expect to use log data in the course of our daily work. We assume log-related manual investigation will be infrequent and of limited utility.
These were exactly the right lessons to learn in the era of expensive hardware and monolithic repos/artifacts. Many people still work in environments like this, and follow logging best practices like these. God bless, more power to em.
Distributed systems assumptions
But more and more of us face systems that are very different.
We have many services, possibly many MANY services. A representative request will have “many” hops across “many” services and routers and proxies and meshes and storage systems.
We cannot model the system in our heads; it would be a mistake to try. We rely on tooling as the source of truth for those systems.
You may or may not have access to those services, or the systems your code runs on. There may or may not be a logging facility, or a centralized log aggregator. Your only view of the system is through the instrumentation of your code.
Disks and system resources are cheap, ephemeral, all but disposable.
Data services are similarly cheap. We can almost entirely silo application performance off from the cost of writing perf data out.
Investigation is prohibitively slow and expensive for a human to do by hand. Many of the nodes or processes we need to inspect may no longer even exist, but their past states may still be relevant to us in understanding patterns to the present time.
Investigation should usually be done distributedly, across all instantiations of your code, however many there might be — and in real time
Investigation requires computation — not just string search. We need to ask on the fly involving math and percentiles and breakdowns and group by’s. And we need access to the raw requests in order to run accurate computations — no pre-aggregates.
The hardest part isn’t usually debugging the code, it’s figuring out where is the code you need to debug. Or what the errors or outliers have in common from the perspective of the code. Fixing the code itself is often comparatively trivial, once found.
What even is ‘logging’?
What even is ‘local disk’?
This isn’t optional: at some point of complexity or scale or distributedness, it becomes necessary if you want to work with these systems.
Logs can’t help you here.
And you aren’t going to get that kind of explorable data out of loglevel:ERROR, or by chopping up your telemetry into disconnected metrics devoid of context.
You are only going to get this kind of explorable, ad hoc, computation-friendly data if you take a radically new approach to how you output and aggregate telemetry. You’re going to need to replace your log lines and log levels with a different sort of beast: arbitrarily wide structured events that describe the request and its context, one event per
request per service.
If it helps, don’t think of them as log files any more. Think of them as events. Yes, you can stash this stream in a file, but why would you? on what disk? will that work for your serverless functions too? Just stream them over the network to wherever you want to put them.
Log levels are another confusing and unnecessary artifact of yesteryear that you no longer really need. The more you think of structured events as logs, the more tempted you may be to apply the old set of best practices. So just don’t think of them as logs at all.
How to gather and structure your data
Instead of dribbling little pebbles of log effluvia throughout your code, do this. (If you’re a honeycomb user, our beelines do it all automatically for you *and* pre-propagate the blobs with everything we know of your context.)
Initialize an empty blob at the beginning, when the request first enters the service.
Stuff any and all interesting detail about the request into that blob throughout the lifetime of the request.
Any unique id, any high-cardinality variable, any headers passed in, every full query, normalized query, and query execution time; every http call out to a remote service, every http execution time; any shopping cart id, first and last name, execution time — literally anything interesting, append to blob.
Then, when the request is about to exit or error, write the blob off to honeycomb or another service or disk somewhere.
You can see immediately how this method has radically different performance implications and risks than the earlier shotgun spray approach. No more “oops i accidentally put a print line INSIDE a for loop”. The write amplification profile is compressed. Most importantly, the incremental cost of capturing more detail about the request per service is nearly zero.
And now you have the kind of structured data that you can feed into something like a columnar store, or honeycomb, and run ad hoc queries to your heart’s delight.
Distributed systems logging events best practices:
Let’s sum up. (I’m including links to other past rants on this topic):
Feed this data into a columnar store or honeycomb or similar
Now use it every day. Not just as a last resort. Get knee deep in production every single day. Explore. Ask and answer rich questions about your systems, system quality, system behavior, outliers, error conditions, etc. You will be absolutely amazed how useful it is … and appalled by what you turn up. 🙂
No more doing multi-line regexps trying to look for the same request ID or user ID doing five suspicious things in a row.
No more regexps at all, for fuck’s sake.
No more bullshit percentiles that were computed at write time by averaging over a bunch of other averages
No more having to jump around from dashboards to logs trying to vainly eyeball correlate one spike with another. No more wondering why no two tools can agree if anything even exists or not
Just gather the detail you need to ask the questions when you need them, and store it in a single source of truth. It’s that simple.
No need to shame people from learning best practices that worked perfectly well for a long time. You can either let them learn the hard way that this transformation is non optional, or you can help them learn the easy way that it’s simply much better and easier to invest in this telemetry up front. You seem like a nice enough chap, which is probably why you chose door 2. (If you wanted to get tougher about it, have a few reformed folks in to tell their horror stories. Try some ex-twitter engineers.)
The hardest part seems to be getting people to unlearn all the best practices they once learned for dealing with logs. So just don’t call it logs anymore, if that helps. Call it “structured events”.
The company is growing like crazy, your engineering team keeps rising to the challenge, and you are ferociously proud of them. But some cracks are beginning to show, and frankly you’re a little worried. You have always advocated for engineers to have broad latitude in technical decisions, including choosing languages and tools. This autonomy and culture of ownership is part of how you have successfully hired and retained top talent despite the siren song of the Faceboogles.
But recently you saw something terrifying that you cannot unsee: your company is using all the languages, all the environments, all the databases, all the build tools. Shit!!! Your ops team is in full revolt and you can’t really blame them. It’s grown into an unsupportable nightmare and something MUST be done, but you don’t know what or how — let alone how to solve it while retaining the autonomy and personal agency that you all value so highly.
I hear a version of this everywhere I’ve gone for the past year or two. It’s crazy how often. I’ve been meaning to write my answer up for ages, and here it (finally) is.
First of all: you aren’t alone. This is extremely common among high-performing teams, so congratulations. Really!
There actually seems to be a direct link between teams that give engineers lots of leeway to own their technical decisions and that team’s ability to hire and retain top-tier talent, particularly senior talent. Everything is a tradeoff, obviously, but accepting somewhat more chaos in exchange for a stronger sense of individual ownership is usually the right one, and leads to higher-performing teams in the long run.
Second, there is actually already a well-trod path out of this hole to a better place, and it doesn’t involve sacrificing developer agency. It’s fairly simple! Just five short steps, which I will describe to you now.
How to build a golden path and reverse software sprawl
Assemble a small council of trusted senior engineers.
Task them with creating a recommended list of default components for developers to use when building out new services. This will be your Golden Path, the path of convergence (and the path of least resistance).
Tell all your engineers that going forward, the Golden Path will be fully supported by the org. Upgrades, patches, security fixes; backups, monitoring, build pipeline; deploy tooling, artifact versioning, development environment, even tier 1 on call support. Pave the path with gold. Nobody HAS to use these components … but if they don’t, they’re on their own. They will have to support it themselves.
Work with team leads to draw up an umbrella plan for adopting the Golden Path for their current projects as well as older production services, as much as is reasonable or possible or desirable. Come up with a timeline for the whole eng org to deprecate as many other tools as possible. Allocate real engineering time to the effort. Hell, make a party out of it!
After the cutoff date (and once things have stabilized), establish a regular process for reviewing and incorporating feedback about the blessed Path and considering any proposed changes, additions or removals.
There you go. That’s it. Easy, right??
(It’s not easy. I never said it was easy, I said it was simple. 👼🏼)
Your engineers are currently used to picking the best tool for the job by optimizing locally. What data store has a data model that is easiest for them to fit to their needs? Which language is fastest for I/O throughput? What are they already proficient in? What you need to do is start building your muscles for optimizing globally. Not in isolation of other considerations, but in conjunction with them. It will always be a balancing act between optimizing locally for the problem at hand and optimizing globally for operability and general sanity.
(Oh, incidentally, requiring an engineer to write up a proposal any time they want to use a non-standard component, and then defend their case while the council grills them in person — this will be nothing but good for them, guaran-fucking-teed.)
Let’s go into a bit more detail on each of the five points. But quick disclaimer: this is not a prescription. I don’t know your system, your team, your cultural land mines or technical interdependencies or anything else about your situation. I am just telling stories here.
1. Assemble your council
Three is a good number for a council. More than that gets unwieldy, and may have trouble reaching consensus. Less than three and you run into SPOFs. You never want to have a single person making unilateral decisions because a) the decision-making process will be weaker, b) it sets that person up for too much interpersonal friction, and c) it denies your other engineers the opportunity to practice making these kinds of decisions.
Your council members need technical breadth more than depth, and should be widely respected by engineers.
At least one member should have a long history with the company so they know lots of stupid little details about what’s been tried before and why it failed.
At least one member should be deeply versed in practical data and operability concerns.
They should all have enough patience and political skill to drive consensus for their decisions. Absolutely no bombthrowers.
If you’re super lucky, you just tap the three senior technologists who immediately come to mind … your mind and everyone else’s. If you don’t have this kind of automatic consensus, you may want to let teams or orgs nominate their own representative so they feel they have some say.
2. Task the council with defining a Golden Path
Your council cannot vanish for a week and then descend from the mountain lugging lists engraved on stone tablets. The process of discovery and consensus is what validates the result.
The process must include talking to and gathering feedback from your engineers, talking to experts outside the company, talking to teams at other companies who are farther along using that technology, coming up with detailed pro/con lists and reasons for their choices. Maybe sometimes it includes prototyping something or investigating the technical depths … but yeah no mostly it’s just the talking.
You need your council members to have enough political skill to handle these conversations deftly, building support and driving consensus through the process. Everybody doesn’t have to love the outcome, but it shouldn’t be a *surprise* to anyone by the end.
3. Know where you’re going
Your council should create a detailed written plan describing which technologies are going to be supported … and a stab at what “supported” means. (Ask the experts in each component what the best practices are for backups, versioning, dependency management, etc.)
You might start with something like this:
* Backend lang: Go 1.11 ## we will no longer be supporting
backend scripting languages
* Frontend lang: ReactJS v 16.5
* Primary db: Aurora v 2.0 ## Yes, we know postgres is "better",
but we have many mysql experts and 0 pg experts except the one guy
who is going to complain about this. You know who you are.
* Deploy pipeline: github -> jenkins + docker -> S3 -> custom k8s
* Message broker: kafka v 2.10, confluent build
* Mail: SES
* .... etc
Circulate the draft regularly for feedback, especially with eng managers. Some team reorganization will probably be necessary to bear the new weight of your support specifications, and managers will need some lead time to wrangle this.
This is also a great time to reconceive of the way on call works at your company. But I am not going to go into all that here.
4. Set a date, draft a plan: go!
Get approval from leadership to devote a certain amount of time to consolidating your stack and paying down a lump sum of tech debt. It depends on your stage of decay, but a reasonable amount of time might be “25% of engineering time for three months“. Whatever you agree to, make sure it’s enough to make the world demonstrably better for the humans who run it; you don’t want to leave them with a tire fire or you’ll blow your credibility.
The council and team leads should come up with a rough outer estimate for how long it would take to rewrite everything and move the whole stack on to the Golden Stack. (It’s probably impossible and/or would take years, but that’s okay.) Next, look for the quick wins or swollen, inflamed pain points.
If you are running two pieces of functionally similar software, like postgres and mysql, can you eliminate one?
If you are managing something yourself that AWS could manage for you (e.g. postfix instead of SES, or kafka instead of kinesis), can you migrate that?
If you are managing anything yourself that is not core to your business value, in fact, you should try to not manage it.
If you are running any services by hand on an AWS instance somewhere, could you try using a service?
If you are running your own monitoring software, etc … can you not?
If you have multiple versions of a piece of software, can you upgrade or consolidate on one version?
The hardest parts are always going to be the ones around migrating data or rewriting components. Not everything is worth doing or can afford to be done in the time span of your project time, and that’s okay.
Next, brainstorm up some carrots. Can you write templates so that anybody who writes a service using your approved library, magically gets monitoring checks without having to configure anything? Can you write a wrapper so they get a bunch of end-to-end tests for free? Anything you can do to delight people or save them time and effort by using your preferred components is worth considering.
(By the way, if you don’t have any engineers devoted to internal tooling, you’re probably way overdue at this point.)
Pay down as much debt as you can, but be pragmatic: it’s better to get rid of five small things than one large thing, from a support perspective. Your main goal is to shrink the number of types of software your team has to support, particularly databases.
Do look for ways to make it fun, like … running a competition to see who can move the most tools to AWS in a week, or throwing a hack week party, or giving dorky prizes like trophies that entitle you to put your manager on call instead of you for a day, etc.
5. Make the process sustainable
After your target date has come and gone, you probably want to hold a post mortem retrospective and do lots of listening. (Well — first might I recommend a bubble bath and a bottle of champagne? But then a post mortem.)
Nothing is ever fixed forever. The company’s needs are going to expand and contract, and people will come and go, because change is the only constant. So you need to bake some flex into your system. How are you going to handle the need for changes to the Golden Path? Monthly discussions? An email list? Quarterly meetings with a formal agenda? I’ve seen people do all of these and more, it doesn’t really matter afaict.
Nobody likes a cabal, though, so the original council should gradually rotate out. I recommend replacing one person at a time, one per quarter, and rotating in another senior engineer in their place. This provides continuity while giving others a chance to learn these technical and political skills.
In the end, engineers are still free to use any tool or component at any time, just like before, only now they are solely responsible for it, which puts pressure on them not to do it unless REALLY necessary. So if someone wants to propose adding a new tool to the default golden path, they can always add it themselves and gain some experience in it before bringing it to the council to discuss a formal place for it.
That’s all folks
See, wasn’t that simple?
(It’s never simple.)
I dearly wish more people would write up their experiences with this sort of thing in detail. I think engineering teams are too reluctant to show their warts and struggles to the world — or maybe it’s their executives who are afraid? Dunno.
Regardless, I think it’s actually a highly effective recruiting tool when teams aren’t afraid to share their struggles. The companies that brag about how awesome they are are the ones who come off looking weak and fragile. Whereas you can always trust the ones who are willing to laugh about all the ways they screwed up. Right?
In conclusion, don’t feel like an asshole for insisting on some process here. There should be friction around adding new components to your stack. (Add in haste, repent at leisure, as they say.) Anybody who argues with you probably needs to be exposed to way, way more of the support load for that software. That’s my professional opinion.
Anyway. You win or you die. Good luck with your sprawl.
On twitter this week, @srhtcn noted that “Many incidents happen during or right after release” and asked for advice on ways to fix this.
And he’s right! Rolling out new software is the proximate cause for the overwhelming majority of incidents, at companies of all sizes. Upgrading software is both a necessity and a minor insanity, considering how often it breaks things.
But it’s still risky. And most issues are still caused by humans and our pesky need for “improvements”. So what can be done?
It’s not ok for software releases to be scary and hazardous
First of all: If releasing is risky for you, you need to fix that. Make this a priority. Track your failures, practice post mortems, evaluate your on call practices and culture. Know if you’re getting better or worse. This is a project that will take weeks if not months until you can be confident in the results.
You have to fix it though, because these things are self-reinforcing. If shipping changes is scary and fraught, people will do it less and it will get even MORE scary and treacherous.
Likewise, if you turn it into a non-cortisol inducing event and set expectations, engineers will ship their code more often in smaller diffs and therefore break the world less.
Fixing deploys isn’t about eliminating errors, it’s about making your pipeline resilient to errors. It’s fundamentally about detecting common failures and recovering from them, without requiring human intervention.
Value your tools more
As an short term patch, you should run deploys in the mornings or whenever everyone is around and fresh. Then take a hard look at your deploy pipeline.
In too many organizations, deploy code is a technical backwater, an accumulation of crufty scripts and glue code, forked gems and interns’ earnest attempts to hack up Capistrano. It usually gives off a strong whiff of “sloppily evolved from many 2 am patches with no code review”.
This is insane. Deploy software is the most important software you have. Treat it that way: recruit an owner, allocate real time for development and testing, bake in metrics and track them over time.
If it doesn’t have an owner, it will never improve. And you will need to invest in frequent improvements even after you’re over this first hump.
Signal high organizational value by putting one of your best engineers on it.
Recruit help from the design side of the house as well. The “right” thing to do must be the fastest, easiest thing to do, with friendly prompts and good docs. No “shortcuts” for people to reach for at the worst possible time. You need user research and design here.
Track how often deploys fail and why. Managers should pay close attention to this metric, just like the one for people getting interrupted or woken up, and allocate time to fixing this early whenever it sags. Before it gets bad.
Allocate real time for development, testing, and training — don’t expect the work to get shoved into people’s “spare time” or post mortem cleanup time. Make sure other managers understand the impact of this work and are on board. Make this one of your KPIs.
In other words, make deploy tools a first class citizen of your technical toolset. Make the work prestigious and valued — even aspirational. If you do performance reviews, recognize the impact there.
(Btw, “how we hardened our deploys” is total Velocity-bait (&& other practitioner conferences) as well as being great for recruiting and general visibility in blog post form. People love these stories; there definitely aren’t enough of them.)
Turn software engineers into software owners
The canonical CI/CD advice starts with “ship early, ship often, ship smaller change sets”. That’s great advice: you should definitely do those things. But they are covered plenty elsewhere. What’s software ownership?
Software ownership is the natural end state of DevOps. Software engineers, operations engineers, platform engineers, mobile engineers — everyone who writes code should be own the full lifecycle of their software.
Software owners are people who:
Can deploy and roll back their own code
Are able to debug their own issues in prod (via instrumentation, not ssh)
If you’re lacking any one of those three ingredients, you don’t have ownership.
Why ownership? Because software ownership makes for better engineers, better software, and a better experience for customers. It shortens feedback loops and means the person debugging is usually the person with the most context on what has recently changed.
Some engineers might balk at this, but you’ll be doing them a favor. We are all distributed systems engineers now, and distributed systems require a much higher level of operational literacy. May as well start today.
Fail fast, fix fast
This is about shifting your mindset from one of brittleness and a tight grip, to one of flexibility where failures are no big deal because they happen all the time, don’t impact users, and give everyone lots of practice at detecting and recovering from them.
Here are a few of the best practices you should adopt with this practice.
The engineer who writes the code and merges the PR should also run the deploy
Everyone who writes code must be trained in how to deploy, roll back & revert to last known good state (before escalating if necessary). They should also know the basics of instrumentation, feature flagging and debugging in prod..
After deploying you MUST go verify: are your changes behaving as expected? Does anything else look .. unexpected? You have the most context on what to expect; just two minutes spent verifying that things look reasonable will catch the overwhelming majority of errors before users even notice.
Everyone who puts software in production needs to understand and feel responsible for the full lifecycle of their code, not just how it works in their IDE.
Baking: it’s not just for cookies
Shipping something to production is a process of incrementally gaining confidence, not a switch you can flip.
You can’t trust code until it’s been in prod a while, until you’ve seen it perform under a wide range of load and concurrency scenarios, in lots of partial failure modes. Only over time can you develop confidence in it not being terrible.
Nothing is production except production. Don’t rely on never failing; expect failure, embrace failure. Practice failure! Build guard rails around your production systems to help you find and fix problems quickly.
The changes you need to make your pipeline more resilient are roughly the same changes you need to safely test in production. These are a few of your guard rails.
Build canaries for your deploy process, so you can promote releases gracefully and automatically to larger subsets of your traffic as you gain confidence in them
Create cohorts. Deploy to internal users first, then any free tier, etc in order of ascending importance. Don’t jump from 10% to 25% to 50% and then 100% — some changes are related to saturating backend resources, and the 50%-100% jump will kill you.
Have robots check the health of your software as it rolls out to decide whether to promote the canary. Over time the robot checks will mature and eventually catch a ton of problems and regressions for you.
The quality of code is not knowable before it hits production. You may able to spot some problems, but you can never guarantee a lack of then. It takes time to bake a new release and gain incremental confidence in new code.
Get someone to own the deploy software
Value the work
Create a culture of software ownership
LOOK at what you’ve done after you do it
Be suspicious of new versions until they prove themselves