The cost of services is on everybody’s mind right now, with interest rates rising, economic growth slowing, and organizational budgets increasingly feeling the pinch. But I hear a special edge in people’s voices when it comes to their observability bill, and I don’t think it’s just about the cost of goods sold. I think it’s because people are beginning to correctly intuit that the value they get out of their tooling has become radically decoupled from the price they are paying.

In the happiest cases, the price you pay for your tools is “merely” rising at a rate several times faster than the value you get out of them. But that’s actually the best case scenario. For an alarming number of people, the value they get actually decreases as their bill goes up.

Observability 1.0 and the cost multiplier effect

Are you familiar with this chestnut?

“Observability has three pillars: metrics, logs, and traces.”

This isn’t exactly true, but it’s definitely true of a particular generation of tools—one might even say definitionally true of a particular generation of tools. Let’s call it “observability 1.0.”

From an evolutionary perspective, you can see how we got here. Everybody has logs… so we spin up a service for log aggregation. But logs are expensive and everybody wants dashboards… so we buy a metrics tool. Software engineers want to instrument their applications… so we buy an APM tool. We start unbundling the monolith into microservices, and pretty soon we can’t understand anything without traces… so we buy a tracing tool. The front-end engineers point out that they need sessions and browser data… so we buy a RUM tool. On and on it goes.

Logs, metrics, traces, APM, RUM. You’re now paying to store telemetry five different ways, in five different places, for every single request. And a 5x multiplier is on the modest side of the spectrum, given how many companies pay for multiple overlapping tools in the same category. You may also also be collecting:

Profiling data
Product analytics
Business intelligence data
Database monitoring/query profiling tools
Mobile app telemetry
Behavioral analytics
Crash reporting
Language-specific profiling data
Stack traces
CloudWatch or hosting provider metrics
…and so on.

So, how many times are you paying to store data about your user requests? What’s your multiplier? (If you have one consolidated vendor bill, this may require looking at your itemized bill.)

There are many types of tools, each gathering slightly different data for a slightly different use case, but underneath the hood there are really only three basic data types: the metric, unstructured logs, and structured logs. Each of these have their own distinctive trade-offs when it comes to how much they cost and how much value you can get out of them.

Metrics

Metrics are the great-granddaddy of telemetry formats; tiny, fast, and cheap. A “metric” consists of a single number, often with tags appended. All of the context of the request gets discarded at write time; each individual metric is emitted separately. This means you can never correlate one metric with another from the same request, or select all the metrics for a given request ID, user, or app ID, or ask arbitrary new questions about your metrics data.

Metrics-based tools include vendors like Datadog and open-source projects like Prometheus. RUM tools are built on top of metrics to understand browser user sessions; APM tools are built on top of metrics to understand application performance.

When you set up a metrics tool, it generally comes prepopulated with a bunch of basic metrics, but the useful ones are typically the custom metrics you emit from your application.

Your metrics bill is usually dominated by the cost of these custom metrics. At minimum, your bill goes up linearly with the number of custom metrics you create. Which is unfortunate, because to restrain your bill from unbounded growth, you have to regularly audit your metrics, do your best to guess which ones are going to be useful in the future, and prune any you think you can afford to go without. Even in the hands of experts, these tools require significant oversight.

Linear cost growth is the goal, but it’s rarely achieved. The cost of each metric varies wildly depending on how you construct it, what the values are, how often it gets hit, etc. I’ve seen a single custom metric cost $30k per month. You probably have dozens of custom metrics per service, and it’s almost impossible to tell how much each of them costs you. Metrics bills tend to be incredibly opaque (possibly by design).

Nobody can understand their software or their systems with a metrics tool alone, because the metric is extremely limited in what it can do. No context, no cardinality, no strings… only basic static dashboards. For richer data, we must turn to logs.

Unstructured logs

You can understand much more about your code with logs than you can with metrics. Logs are typically emitted multiple times throughout the execution of the request, with one or a small number of nouns per log line, plus the request ID. Unstructured logs are still the default, although this is slowly changing.

The cost of unstructured logs is driven by a few things:

Write amplification. If you want to capture lots of rich context about the request, you need to emit a lot of log lines. If you are printing out just 10 log lines per request, per service, and you have half a dozen services, that’s 60 log events for every request.
Noisiness. It’s extremely easy to accidentally blow up your log footprint yet add no value—e.g., by putting a print statement inside a loop instead of outside the loop. Here, the usefulness of the data goes down as the bill shoots up.
Constraints on physical resources. Due to the write amplification of log lines per request, it’s often physically impossible to log everything you want to log for all requests or all users—it would saturate your NIC or disk. Therefore, people tend to use blunt instruments like these to blindly slash the log volume:
- Log levels
- Consistent hashes
- Dumb sample rates

When you emit multiple log lines per request, you end up duplicating a lot of raw data; sometimes over half the bits are consumed by request ID, process ID, timestamp. This can be quite meaningful from a cost perspective.

All of these factors can be annoying. But the worst thing about unstructured logs is that the only thing you can do to query them is full text search. The more data you have, the slower it becomes to search that data, and there’s not much you can do about it.

Searching your logs over any meaningful length of time can take minutes or even hours, which means experimenting and looking around for unknown-unknowns is prohibitively time-consuming. You have to know what to look for in order to find it. Once again, as your logging bill goes up, the value goes down.

Structured logs

Structured logs are gaining adoption across the industry, especially as OpenTelemetry picks up steam. The nice thing about structured logs is that you can actually do things with the data other than slow, dumb string searches. If you’ve structured your data properly, you can perform calculations! Compute percentiles! Generate heatmaps!

Tools built on structured logs are so clearly the future. But just taking your existing logs and adding structure isn’t quite good enough. If all you do is stuff your existing log lines into key/value pairs, the problems of amplification, noisiness, and physical constraints remain unchanged—you can just search more efficiently and do some math with your data.

There are a number of things you can and should do to your structured logs in order to use them more effectively and efficiently. In order of achievability:

Instrument your code using the principles of canonical logs, which collects all the vital characteristics of a request into one wide, dense event. It is difficult to overstate the value of doing this, for reasons of usefulness and usability as well as cost control.
Add trace IDs and span IDs so you can trace your code using the same events instead of having to use an entirely separate tool.
Feed your data into a columnar storage engine so you don’t have to predefine a schema or indexes to decide which dimensions future you can search or compute based on.
Use a storage engine that supports high cardinality, with an explorable interface.

If you go far enough down this path of enriching your structured events, instrumenting your code with the right data, and displaying it in real time, you will reach an entirely different set of capabilities, with a cost model so distinct it can only be described as “observability 2.0.” More on that in a second.

Ballooning costs are baked into observability 1.0

To recap: high costs are baked into the observability 1.0 model. Every pillar has a price.

You have to collect and store your data—and pay to store it—again and again and again, for every single use case. Depending on how many tools you use, your observability bill may be growing at a rate 3x faster than your traffic is growing, or 5x, or 10x, or even more.

It gets worse. As your costs go up, the value you get out of your tools goes down.

Your logs get slower and slower to search.
You have to know what you’re searching for in order to find it.
You have to use blunt force sampling technique to keep log volume from blowing up.
Any time you want to be able to ask a new question, you first have to commit new code and deploy it.
You have to guess which custom metrics you’ll need and which fields to index in advance.
As volume goes up, your ability to find a needle in the haystack—any unknown-unknowns—goes down commensurately.

And nothing connects any of these tools. You cannot correlate a spike in your metrics dashboard with the same requests in your logs, nor can you trace one of the errors. It’s impossible. If your APM and metrics tools report different error rates, you have no way of resolving this confusion. The only thing connecting any of these tools is the intuition and straight-up guesses made by your most senior engineers. Which means that the cognitive costs are immense, and your bus factor risks are very real. The most important connective data in your system—connecting metrics with logs, and logs with traces—exists only in the heads of a few people.

At the same time, the engineering overhead required to manage all these tools (and their bills) rises inexorably. With metrics, an engineer needs to spend time auditing your metrics, tracking people down to fix poorly constructed metrics, and reaping those that are too expensive or don’t get used. With logs, an engineer needs to spend time monitoring the log volume, watching for spammy or duplicate log lines, pruning or consolidating them, choosing and maintaining indexes.

But all this the time spent wrangling observability 1.0 data types isn’t even the costliest part. The most expensive part is the unseen costs inflicted on your engineering organization as development slows down and tech debt piles up, due to low visibility and thus low confidence.

Is there an alternative? Yes.

The cost model of observability 2.0 is very different

Observability 2.0 has no three pillars; it has a single source of truth. Observability 2.0 tools are built on top of arbitrarily-wide structured log events, also known as spans. From these wide, context-rich structured log events you can derive the other data types (metrics, logs, or traces).

Since there is only one data source, you can correlate and cross-correlate to your heart’s content. You can switch fluidly back and forth between slicing and dicing, breaking down or grouping by events, and viewing them as a trace waterfall. You don’t have to worry about cardinality or key space limitations.

You also effectively get infinite custom metrics, since you can append as many as you want to the same events. Not only does your cost not go up linearly as you add more custom metrics, your telemetry just gets richer and more valuable the more key-value pairs you add! Nor are you limited to numbers; you can add any and all types of data, including valuable high-cardinality fields like “App Id” or “Full Name.”

Observability 2.0 has its own amplification factor to consider. As you instrument your code with more spans per request, the number of events you have to send (and pay for) goes up. However, you have some very powerful tools for dealing with this: you can perform dynamic head-based sampling or even tail-based sampling, where you decide whether or not to keep the event after it’s finished, allowing you to capture 100% of slow requests and other outliers.

Engineering time is your most precious resource

But the biggest difference between observability 1.0 and 2.0 won’t show up on any invoice. The difference shows up in your engineering team’s ability to move quickly, with confidence.

Modern software engineering is all about hooking up fast feedback loops. And observability 2.0 tooling is what unlocks the kind of fine-grained, exploratory experience you need in order to accelerate those feedback loops.

Where observability 1.0 is about MTTR, MTTD, reliability, and operating software, observability 2.0 is what underpins the entire software development lifecycle, setting the bar for how swiftly you can build and ship software, find problems, and iterate on them. Observability 2.0 is about being in conversation with your code, understanding each user’s experience, and building the right things.

Observability 2.0 isn’t exactly cheap either, although it is often less expensive. But the key difference between o11y 1.0 and o11y 2.0 has never been that either is cheap; it’s that with observability 2.0, when your bill goes up, the value you derive from your telemetry goes up too. You pay more money, you get more out of your tools. As you should.

Interested in learning more? We’ve written at length about the technical prerequisites for observability with a single source of truth (“observability 2.0” as we’ve called it here). Honeycomb was built to this spec; ServiceNow (formerly Lightstep) and Baselime are other vendors that qualify. Click here to get a Honeycomb demo.

CORRECTION: The original version of this document said that “nothing connects any of these tools.” If you are using a single unified vendor for your metrics, logging, APM, RUM, and tracing tools, this is not strictly true. Vendors like New Relic or Datadog now let you define certain links between your traces and metrics, which allows you to correlate between data types in a few limited, predefined ways. This is better than nothing! But it’s very different from the kind of fluid, open-ended correlation capabilities that we describe with o11y 2.0. With o11y 2.0, you can slice and dice, break down, and group by your complex data sets, then grab a trace that matches any specific set of criteria at any level of granularity. With o11y 1.0, you can define a metric up front, then grab a random exemplar of that metric, and that’s it. All the limitations of metrics still apply; you can’t correlate any metric with any other metric from that request, app, user, etc, and you certainly can’t trace arbitrary criteria. But you’re right, it’s not nothing. 😸

Questionable Advice: “My boss says we don’t need any engineering managers. Is he right?”

January 5, 2024January 10, 2024 mipsytipsyadvice, culture, engineering management, hierarchy, organizations, sociotechnical18 Comments

I recently joined a startup to run an engineering org of about 40 engineers. My title is VP Engineering. However, I have been having lots of ongoing conflict with the CEO (a former engineer) around whether or not I am allowed to have or hire any dedicated engineering managers. Right now, the engineers are clustered into small teams of 3-4, each of which has a lead engineer — someone who leads the group, but whose primary responsibility is still writing code and shipping product.

I have headcount to hire more engineers in the coming year, but no managers. My boss says we are a startup and can’t afford such luxuries. It seems obvious to me that we need engineering managers, but to him, it seems just as obvious that managers are unnecessary overhead and that all hands should be on deck writing code at our stage.

I don’t know how to make that argument. It seems so obvious to me that I actually struggle to put it into words or make the case for why we should hire EMs. Help?

— Unnecessary Overhead(?!?)

Oh boy, there’s a lot to unpack here.

It is unsurprising to me that your CEO does not understand why managers exist, given that he does not seem to understand why organizational structures exist. 🙈 Why is he micromanaging how you are structuring your org or what roles you are allowed to fill? He hired you to do a job, and he’s not letting you do it. He can’t even explain why he isn’t letting you do it. This does not bode well.

But I do think it’s an interesting question. So let’s pretend he isn’t holding your ability to do your damn job hostage until you defend yourself to his satisfaction. 😒

I can think of two ways to make the case for engineering managers: one is rather complicated, from first principles, and the other very simple, but perhaps unsatisfying.

I personally have a … vigorous … knee-jerk response to authority; I hate being told what to do. It’s only recently that I’ve found my way to an understanding of hierarchy that feels healthy and practical, and that was by looking at it through the lens of systems theory.

Why does hierarchy exist in organizations?

It makes sense that hierarchy comes with a lot of baggage. Many of us have had bad experience with managers — indeed, entire organizations — where hierarchy was used as a tool of oppression, where people rose up the leadership ranks by hoarding information and playing dominance games, and decisions got made by pulling rank.

Working at a place like that fucking sucks. Who wants to invest their creativity and life force into a place that feels like a Dilbert cartoon, knowing how little it will be valued or reciprocated, and that it will slowly but surely get crushed out of you?

But hierarchy is not intrinsically authoritarian. Hierarchy did not originate as a political structure that humans invented for controlling and dominating one another, it is in fact a property of self-organizing systems, and it emerges for the benefit of the subsystems. In fact, hierarchy is absolutely critical to the adaptability, resiliency, and scalability of complex systems.

Let’s start with few basic facts about systems, for anyone that may be unfamiliar.

Hierarchy is a property of self-organizing systems

A system is “a network of interdependent components that work together to try to accomplish a common aim” (W. Edward Deming). A pile of sand is not a system, but a car is a system; if you take out its gas tank, the car cannot achieve its aim.

A subsystem is a collection of elements with a smaller aim inside a larger system. There can be many levels of subsystems that operate interdependently. The subsystems always work to support the needs of the larger system; if the subsystem instead optimizes for its own best interests, the whole system can fail (this is where the term “suboptimize” comes from 😄).

A system is self-organizing if it has the ability to make itself more complex, by diversifying, adapting, and improving itself. As systems self-organize and their complexity increases, they tend to generate hierarchy — an arrangement of systems and subsystems. In a stable, resilient and efficient system, subsystems can largely take care of themselves, regulate themselves, and serve the needs of the larger system, while the larger system coordinates between subsystems and helps them perform better.

Hierarchy minimizes the costs of coordination and reduces the amount of information that any given part of the system has to keep track of, preventing information overload. Information transfer and relationships within a subsystem are much more dense and have fewer delays than information transfer or relationships between subsystems.

(This should all sound pretty familiar to any software engineer. Modularization, amirite?? 😍)

Applying this definition, we can say that a manager’s job is to coordinate between teams and help their team perform better.

The false binary of sociotechnical systems

You’ve probably heard this canard: “Engineers do the technical work, managers do the people work.” I hate it. ☺️ I think it misconstrues the fundamental nature of sociotechnical systems. The “socio” and “technical” of sociotechnical systems are not neatly separable, they are interwoven and interdependent. There is actually precious little that is purely technical work or purely people work; there is a metric shitload of glue work that draws upon both skill sets.

Consider a very partial list of tasks done by any functional engineering org, besides writing code:

Recruiting, networking, interviewing, training interviewers, synthesizing feedback, writing job descriptions and career ladders
Project management for each project or commitment, prioritizing backlog, managing stakeholders and resolving conflicts, estimating size and scope, running retrospectives
Running team meetings, having 1x1s, giving continuous growth feedback, writing reviews, representing the team’s needs
Architecture, code review, refactoring; capturing DORA and productivity metrics, managing alert volume to prevent burnout

A lot of this work can be done by engineers, and often is. Every company distributes the load somewhat differently. This is a good thing! You don’t WANT an org where this work is only done by managers. You want individual contributors to help co-create the org and have a stake in how it gets run. Almost all of this work would be done more effectively by someone with an engineering background.

So you can understand why someone might hesitate to spend valuable headcount on engineering managers. Why wouldn’t you want everyone in engineering to be writing and shipping code as their primary job? Isn’t that by definition the best way to maximize productivity?

Ehhh… 😉

Engineering managers are a useful abstraction

In theory, you could make a list of all the tasks that need to be done to coordinate with other teams and have each item be picked up by a different person. In practice, this is impractical because then everybody would need to know about everything. One of the primary benefits of hierarchy, remember, is to reduce information overload. Intra-team communication should be high-bandwidth and fast, inter-team communication should be more sparse.

As the company scales, you can’t expect everybody to know everyone else; we need abstractions in order to function. A manager is the point of contact and representative for their team, and they serve as routers for important information.

Graph theory: for each of n engineers to be connected ("aligned") w each other you need n(n-1)/2 links, i.e. you need 780 interactions just to keep consistency. Without a few engineering managers (and arks) he is allowing velocity to build up tech debt at blazing speed.
— HurtingBrain (@HurtingBrain) January 5, 2024

I sometimes imagine managers as the nervous system of the company body, carrying around messages from one limb to another to coordinate actions. Centralizing many or most of these functions into one person lets you take advantage of specialization, as a manager builds relationships and context and improves at their role, and this massively reduces context switching for everyone else.

Manager calendars vs maker calendars

Engineering labor takes concentration and focus. Context switching is expensive, and too many interrupts can be fatal. Management labor consists of context switching every hour or so, and being available for interruptions throughout the day. These are two very different modes of being, headspaces, and calendar schedules, and do not coexist well.

In general, you want people to be able to spend most of their time working on things that contribute to the success of the outcomes they are directly responsible for. Engineers can only do so much glue work before their calendar turns into Swiss cheese and they can no longer deliver on their commitments. Since managers’ calendars are already Swiss cheese, it’s typically less disruptive for them to take on a larger share of glue labor.

It isn’t up to managers to do all the glue work, but it is a manager’s job to make sure that everything that needs to get done, does gets done. It is a manager’s job to try to line up every engineer with work that is interesting and challenging, but not overwhelming, and to ensure that unpleasant labor gets equitably distributed. It’s also a manager’s job to make sure that if we are asking someone to do a job, they are equipped with the resources they need to succeed at that job. Including time to focus.

Management is a tool for accountability

When you’re an engineer, you are responsible for the software you develop, deploy, and maintain. When you’re a manager, you are responsible for your team and the organization as a whole.

Management is one way of holding people accountable for specific outcomes (building teams with the right skills, relationships, and processes to make good decisions and build value for the company), and equipping them with the resources (budget, tools, headcount) to achieve those outcomes. If you aren’t making building the organization someone’s number one job, it won’t be anyone’s number one job, which means it probably won’t get done very well. And whose responsibility will that be, Mr. CEO?

There’s a real upper limit to what you can reasonably expect tech leads, or engineers, or anyone whose actual job is shipping software to do in their “spare time”. If you’re trying to hold your tech leads responsible for building healthy engineering teams, tools, and processes, you are asking them to do two calendarily incompatible jobs with only one calendar. The likeliest scenario is that they will focus on the outcomes they feel comfortable owning (the technical ones), while you pile up organizational debt in the background.

In natural hierarchies, we look up for purpose and down for function. That, in a nutshell, is the more complicated argument for why we need engineering managers.

Choose Boring technology Culture

The simpler argument is this: most engineering orgs have engineering managers. That’s the default. Lots of people much smarter than you or me have spent lots of time thinking and tinkering with org structures over the years, and this is what we’ve got.

As Dan McKinley famously said, we should “choose boring technology“. Boring doesn’t mean bad, it means the capabilities and failure conditions are well understood. You only ever get a few innovation tokens, so you should spend those wisely on core differentiators that could make or break your business. The same goes for culture. Do you really want to spend one of your tokens on org structure? Why??

For better or for worse, the hierarchical org structure is well understood. There are plenty of people on the job market who are proficient at managing or working with managers, and you can hire them. You can get training, coaching, or read a lot of self-help books. There are various management philosophies you can coalesce around or use to rule people out. On the other hand, the manager-free experiments I’m aware of (e.g. holacracy at Medium and GitHub, or “Choose Your Own Work” at Linden Lab) have all been quietly abandoned or outgrown. Not, in my experience, because leaders went mad for power, but due to chaos, lack of focus, and poor execution.

When there is no explicit structure or hierarchy, the result is not freedom and egalitarianism, it’s “informal, unacknowledged, and unaccountable leadership”, as famously detailed in “The Tyranny of Structureless“. In reality, sadly, these teams tend to be chaotic, fragile, and frustrating. I know! I’m pissed too! 😭

This argument doesn’t necessarily prove your CEO is wrong, but I should think his bar for proof is much higher than yours. “I don’t want any of my engineers to stop writing code” is not an argument. But I’m also feeling like I haven’t quite addressed the core question of productivity, so let’s pick that up again once more.

More lines of code != more productivity

To briefly recap: we were talking about an org with ~40 engineers, broken up into 10 small clusters of 3-4 engineers, each with a tech lead. Your CEO is arguing that you can’t afford to lose any velocity, which he thinks is what would happen if anyone stops writing code full time.

Maybe. But everything I have ever experienced leads me to believe that a fewer number of larger teams, each helmed by an experienced engineering manager, should way outperform this gaggle of tiny groups. It’s not even close. And they can do so in a way that’s more efficient, sustainable, and humane than this scrappy death march.

And systems thinking shows us why! With fewer groups, but larger ones, you have less overall management overhead, and much less of the slow and costly intra-group coordination. You unlock rich, dense knowledge transfer within groups, which gives you more shared coverage of the surface area. With 7-9 engineers per group you can build a real on call rotation, which means fewer heroics and less burnout. The coordination that you do need to do can be more strategic, less tactical, and much more forward-looking.

Would five big teams ship as many lines of code as 10 small teams, even if five engineers become managers and stop writing code? Probably, but who cares? Your customers give zero fucks how many lines of code you write. They care about whether you are building the right things and solving problems that matter to them. What matters is moving the business forward, not churning out code. Don’t forget, the act of churning out code creates costs and externalities in and of itself.

What defines your velocity is that you spend your time on the right things. Learning to make good decisions about what to build is something every organization has to work out for itself, and it is always an ongoing work in progress. Engineering managers don’t do all the work or make all the decisions, but they are absolutely fucking vital, in my experience, to ensuring that work happens and is done well. As I wrote in my last piece, engineering managers are the embodiment of the feedback loops that systems use to learn and improve.

Are managers ever unnecessary overhead?

Sure, absolutely. Management is about coordinating between teams and helping teams run more optimally, so anything that decreases your need for coordination also decreases your need for management. If you are a small company, or if you have really senior folks who are used to working together, you need a lot less coordination. The next most relevant factor is probably the rate of change; if you’re growing fast or have a lot of turnover, or if there’s a lot of time pressure or frequent shifts in strategy, your need for managers goes up. But there are plenty of smaller orgs out there that are doing just fine without a lot of formal management.

Look, I’m not a fan of the word “overhead”, because a) it’s kind of rude and b) people who call managers “overhead” are typically people who disrespect or do not value the craft of management.

But management is, in fact, overhead. 😅 So is a lot of other glue work! By which I mean the work is important, but does not itself move the business forward; we should do as much of it as absolutely necessary and no more. The nature of glue work is such that it too-easily expands to consume all available time and space (and then some). Constraints are good. Feeling a bit underresourced is good, and should be the norm. It is incredibly easy for management to get a bit bloated, and managers can be very loath to acknowledge this, because it’s not like they ever feel any less stressed or stretched.[*]

Management is also very much like operations work in that when it’s being done well, it’s invisible. Evaluating managers can be very hard, especially in the near term, and making decisions about when it’s time to create or pay down organizational debt is a whole nother ball of wax, and way outside the scope of this post.

But yes, managers can absolutely be unnecessary overhead.

However, if you have 40 engineers all reporting to one VP, and nobody else whose number one job is the outcomes related to people, teams and org, I feel pretty safe in saying this is not a risk for you at this time.

<3 charity

[*] In fact, the reverse can be true; bloated management can create MORE work for managers, and they may counterintuitively feel LESS stretched or stressed with a leaner org chart. Bureaucracies do tend to pick up a momentum all their own. Especially when management gets all wrapped up in promotions and egos. Which is yet another good reason to ensure that management is not a promotion or a form of domination.

Becoming An Engineering Manager Can Make You Better At Life And Relationships

December 15, 2023January 10, 2024 mipsytipsyadvice, careers, culture, engineering management, ethics, management31 Comments

Original title: “Why Should You (Or Anyone) Become An Engineering Manager?”

The first piece I ever wrote about engineering management, The Engineer/Manager Pendulum, was written as a love letter to a friend of mine who was unhappy at work. He was an engineering director at a large and fast-growing startup, where he had substantially built out the entire infrastructure org, but he really missed being an engineer and building things. He wasn’t getting a lot of satisfaction out of his work, and he felt like there were other people who might relish the challenge and do it better than he could.

At the same time, it felt like a lot to walk away from! He had spent years building up not only the teams, but also his influence and reputation. He had grown accustomed to being in the room where decisions get made, and didn’t want to give that up or take a big step back in his career. He agonized over this for a long time (and I listened over many whiskeys). 🙂

To me it seemed obvious that his power and influence would only increase if he went back to engineering. You bring your credibility and your relationships along with you, and enthusiasm is contagious. So I wrote the piece with him in mind, but it definitely struck a nerve; it is still the most-read piece I have ever written.

That was in 2017. I’ve written a lot over the years since then about teams and management. For a long time, everything I wrote seemed to come out with a pretty noticeable bias against management, towards engineering:

I go back and read some of those pieces now, and the pervasive anti-manager slant actually makes me a bit uncomfortable, because the environment has changed quite a lot since then.

Miserable managers have miserable reports

When I started writing about engineering management, it seemed like there were a lot of unhappy, resentful, poorly trained managers, lots of whom would prefer to be writing code. Most people made the choice to switch to management for reasons that had nothing to do with the work itself.

Becoming a manager was seen as a promotion
It was the only form of career progression available at many places
Managers made a lot more money
It was the only way to get a seat at the table, or be in the loop
They were tired of taking orders from someone else

But a lot has changed. The emergence of staff+ engineering has been huge (two new books published in the last five years, and at least one conference!). The industry has broadly coalesced around engineering levels and career progression; a parallel technical leadership track is now commonplace.

We’ve become more aware of how fragile command-and-control systems are, and that you want to engage people’s agency and critical thinking skills. You want them to feel ownership over their labor. You can’t build great software on autopilot, or by picking up jira tasks. Our systems are becoming so complex that you need people to be emotionally and mentally engaged, curious, and continuously learning and improving, both as individuals and as teams.

I tell people all the time that you can do most of the "fun" management things (mentoring, coaching, watching people grow, contributing to decision making) as an IC without doing all the terrible parts of management (firing, budgeting, serious HR things).

— Jill Wetzler (@JillWetzler) September 5, 2019

At the same time, our expectations for managers have gone up dramatically. We’ve become more aware of the damage done by shoddy managers, and we increasingly expect managers to be empathetic, supportive, as well as deeply technical. All of this has made the job of engineering manager more challenging.

Ambitious engineers had already begun to drift away from management and towards the role of staff or principal engineer. And then came the pandemic, which caused managers (poorly supported, overwhelmed, squeezed between unrealistic expectations on both sides) to flee the profession in droves.

It’s getting harder to find people who are willing to be managers. On the one hand, it is fucking fantastic that people aren’t being driven into management out of greed, rage, or a lust for power. It is WONDERFUL that people are finding engineering roles where they have autonomy, ownership, and career progression, and where they are recognized and rewarded for their contributions.

On the other hand, engineering managers are incredibly important and we need them. Desperately.

Good engineering managers are force multipliers

A team with a good engineering manager will build circles around a team without one. The larger or more complex the org or the product, and the faster you want to move, the more true this is. Everybody understands the emotional component, that it feels nice to have a competent manager you trust. But these aren’t just squishy feels. This shit translates directly into velocity and quality. The biggest obstacles to engineering productivity are not writing lines of code too slowly or not working long hours, they are:

Working on the wrong thing
Getting bogged down in arguments, or being endlessly indecisive
Waiting on other teams to do their work, waiting on code review
Ramping new engineers, or trying to support unfamiliar code
When people are upset, distracted, or unmotivated
Unfinished migrations, migrations in flight, or having to support multiple systems indefinitely
When production systems are poorly understood and opaque, quality suffers, and firefighting skyrockets
Terrible processes, tools, or calendars that don’t support focus time
People who refuse to talk to each other
Letting bad hires and chronic underperformers stick around indefinitely

Engineers are responsible for delivering products and outcomes, but managers are responsible for the systems and structural support that enables this to happen.

Managers don’t make all the decisions, but they do ensure the decisions get made. They make sure that workstreams are are staffed and resourced sufficiently, that engineers are trained and improving at their craft. They pay attention to the contracts and commitments you have made with other teams, companies or orgs. They advocate for your needs at all levels of the organization. They connect dots and nudge and suggest ideas or solutions, they connect strategy with execution.

Breaking down a complex business problem into a software project that involves the collaboration of multiple teams, and ensuring that every single contributor has work to do that is challenging and pushes their boundaries while not being overwhelming or impossible… is really fucking hard. Even the best leaders don’t get it right every time.

In systems theory, hierarchy emerges for the benefit of the subsystems. Hierarchy exists to coordinate between the subsystems and help them improve their function; it is how systems create resiliency to unknown stressors. Which means that managers are, in a very real way, the embodiment of the feedback loops and meta loops that a system depends on to align itself and all of its parts around a goal, and for the system itself to improve over time.

For some people, that is motivation enough to try being a manager. But not for all (and that’s okay!!). What are some other reasons for going into management?

Why should you (or anyone) be a manager?

I can think of a few good reasons off the top of my head, like…

It gets you closer to how the business operates, and gives you a view into how and why decisions get made that translate eventually into the work you do as an engineer
Which makes the work feel more meaningful and less arbitrary, I think. It connects you to the real value you are creating in the world.
Many people reach a point where they feel a gravitational pull towards mentorship. It’s almost like a biological imperative to replicate yourself and pass on what you have learned to the next generation.
Many people also get to a point where they develop strong convictions about what not to do as a manager. They may feel compelled to use what they’ve learned to build happy teams and propagate better practices through the industry
One way to develop a great staff engineer is to take a great senior engineer and put them through 2-3 years of management experience.

But the main reason I would encourage you to try engineering management is a reason that I’m not sure I’ve ever heard someone articulate up front, which is that…it can make you better at life and relationships, in a huge and meaningful way.

Work is always about two things: what you put out into the world, and who you become while doing it.

I want to stop short of proclaiming that “being a manager will make you a better person!” — because skills are skills, and they can be used for good or ill. But it can.

It’s a lot like choosing to become a parent. You don’t decide to have kids because it sounds like a hoot (I hope); you go into it knowing it will be hard work, but meaningful work. It’s a way of processing and passing on the experiences that have shaped you and who you are. You also take up the mantle understanding that this will change you — it changes who you are as a person, and the relationships you have with others.

From the outside, management looks like making decisions and calling the shots. From the inside, management looks more like becoming intimately acquainted with your own limitations and motivations and those of others, plus a lot of systems thinking.

Yes, management absolutely draws on higher-level skills like strategy and planning, writing reviews, mediating conflict, designing org charts, etc. But being a good manager — showing up for other people and supporting them consistently, day after day — rests on a bedrock of some much more foundational skills.

The kind of skills you learn in therapy, not in classes

Self-regulation. Can you take care of yourself consistently — sleep, eat, leave the house, socialize, balance your moods, moderate your impulses? As an engineer, you can run your tank dry on occasion, but as a manager, that’s malpractice. You always need to have fuel left in the tank, because you don’t know when it will be called upon.

Self-awareness. Identifying your feelings in the heat of the moment, unpacking where they came from, and deciding how to act on them. It’s not about clamping down on your feelings and denying you have them. It’s definitely not about making your feelings into other people’s problems, or letting your reactions create even bigger problems for your future self. It’s about acting in ways that are fueled by your authentic emotional responses, but not ruled by them.

Understanding other people. You learn to read people and their reactions, starting with your direct reports. You build up a mental model for what motivates someone, what moves them, what bothers them, and what will be extremely challenging for them. You must develop a complex topographical map of how much you can trust each person’s judgment, on which aspect, in any given situation.

Setting good boundaries. Where is the line between supporting someone, advocating for them, encouraging them, pushing them … but not propping them up at all costs, or taking responsibility for their success? How is the manager/report relationship different from the coworker relationship, or the friend relationship? How do you navigate the times when you have to hold someone accountable because their work is falling short?

Sensitivity to power dynamics. Do people treat you differently as a manager than they did as a peer? Are there things that used to be okay for you to say or do that now come across as inappropriate or coercive? How does interacting with your reports inform the way you interact with your own manager, or how you understand what they say?

Hard conversations. Telling people things that you know they don’t want to hear, or things that will make them feel afraid, angry, or upset; and then sitting with their reactions, resisting the urge to take it all back and make everything okay.

The art of being on the same side. When you’re giving someone feedback, especially constructive feedback, it’s easy to trigger a defensive response. It’s SO easy for people to feel like you are judging them or criticizing them. But the dynamic you want to foster is one where you are both side by side, shoulder to shoulder, facing the same way, working together. You are giving them feedback because you care; feedback that could help them be even better, if they choose to accept it. You are on their side. They always have agency.

Did you learn these skills growing up? I sure as hell did not.

I grew up in a family that was very nice. It was very kind and loving and peaceful, but we did not tell each other hard things. When I went off to college and started dating, I had no idea how to speak up when something bothered me. There were sentences that lingered on the tip of my tongue for years but were never spoken out loud, over the ebb and fall of entire relationships. And if somebody raises their voice to me in anger, to this day, I crumble.

I also came painfully late to developing a so-called growth mindset. This is super common among “smart kids”, who get so used to perfect scores and praise for high achievements that any feedback or critique feels like failure…and failure feels like the end of the world.

It was only after I became a manager that I began to consciously practice skills like giving feedback, or receiving constructive criticism, or initiating hard conversations. I had to. But once I did, I started getting better.

Turns out, work is actually kind of an ideal sandbox for life skills, because the social contract is more explicit. These are structured relationships, with rules and conventions and expectations, and your purpose for coming together is clear: to succeed at business, to finish a project, to pull a paycheck. Even the element of depersonalization can be useful: it’s not you-you, it’s the professional version of you, performing a professional role. The stakes are lower than they would be with your mother, your partner, or your child.

You don’t have to be a manager to build these skills, of course. But it’s a great opportunity to do so! And there are tools — books and classes, mentors and review cycles. You can ask for feedback from others. Growth and development is expected in this role.

People skills are persistent

The last thing I will say is this — technical skills do decay and become obsolete, particularly language fluency, but people skills do not. Once you have built these muscles, you will carry them with you for life. They will enhance your ability to connect with people and build trust, to listen perceptively and communicate clearly, in both personal and professional relationships.

Whatever you decide to do with your life, these skills increase your optionality and make you more effective.

And that is the reason I think you should consider being an engineering manager. Like I said, I’ve never heard someone cite this as their reason for wanting to become a manager. But if you ask managers why they do it five or ten years later, you hear a version of this over and over again.

One cautionary note

As an engineer, you can work for a company whose leadership team you don’t particularly respect, whose product you don’t especially love, or whose goals you aren’t super aligned with, and it can be “okay.” Not terrific, but not terrible.

As a manager, you can’t. Or you shouldn’t. The conflicts will eat you up inside and/or prevent you from doing excellent work.

Your job consists of representing the leadership team and their decisions, pulling people into alignment with the company’s goals, and thinking about how to better achieve the mission. As far as your team goes, you are the face of The Man. If you can’t do that, you can’t do your job. You don’t get to stand apart from the org and throw rocks, e.g. “they told me I have to tell you this, but I don’t agree with it”. That does nothing but undermine your own position and the company’s. If you’re going to be a manager, choose your company wisely.

charity

P.S. My friend (from the start of the article) went back to being an engineer, despite his trepidation, and never regretted it once. His career has been up and to the right ever since; he went on to start a company. The skills he built as a manager were a huge boost to an already stellar career. 📈

LLMs Demand Observability-Driven Development

September 20, 2023December 21, 2024 mipsytipsyai, artificial intelligence, llms, observability-driven developmentLeave a comment

Originally posted on the Honeycomb blog on September 20th, 2023

Our industry is in the early days of an explosion in software using LLMs, as well as (separately, but relatedly) a revolution in how engineers write and run code, thanks to generative AI.

Many software engineers are encountering LLMs for the very first time, while many ML engineers are being exposed directly to production systems for the very first time. Both types of engineers are finding themselves plunged into a disorienting new world—one where a particular flavor of production problem they may have encountered occasionally in their careers is now front and center.

Namely, that LLMs are black boxes that produce nondeterministic outputs and cannot be debugged or tested using traditional software engineering techniques. Hooking these black boxes up to production introduces reliability and predictability problems that can be terrifying. It’s important to understand this, and why.

100% debuggable? Maybe not

Software is traditionally assumed to be testable, debuggable, and reproducible, depending on the flexibility and maturity of your tooling and the complexity of your code. The original genius of computing was one of constraint; that by radically constraining language and mathematics to a defined set, we could create algorithms that would run over and over and always return the same result. In theory, all software is debuggable. However, there are lots of things that can chip away at that beauteous goal and make your software mathematically less than 100% debuggable, like:

Adding concurrency and parallelism.
Certain types of bugs.
Stacking multiple layers of abstractions (e.g., containers).
Randomness.
Using JavaScript (HA HA).

There is a much longer list of things that make software less than 100% debuggable in practice. Some of these things are related to cost/benefit tradeoffs, but most are about weak telemetry, instrumentation, and tooling.

If you have only instrumented your software with metrics, for example, you have no way of verifying that a spike in api_requests and an identical spike in 503 errors are for the same events (i.e., you are getting a lot of api_requests returning 503) or for a disjoint set of events (the spike in api_requests is causing general congestion causing a spike in 503s across ALL events). It is mathematically impossible; all you can do is guess. But if you have a log line that emits both the request_path and the error_code, and a tool that lets you break down and group by arbitrary dimensions, this would be extremely easy to answer. Or if you emit a lot of events or wide log lines but cannot trace them, or determine what order things executed in, there will be lots of other questions you won’t be able to answer.

There is another category of software errors that are logically possible to debug, but prohibitively expensive in practice. Any time you see a report from a big company that tracked down some obscure error in a kernel or an ethernet device, you’re looking at one of the rare entities with 1) enough traffic for these one in a billion errors to be meaningful, and 2) enough raw engineering power to dedicate to something most of us just have to live with.

But software is typically understandable because we have given it structure and constraints.

IF (); THEN (); ELSE () is testable and reproducible. Natural languages, on the other hand, are infinitely more expressive than programming languages, query languages, or even a UI that users interact with. The most common and repeated patterns may be fairly predictable, but the long tail your users will create is very long, and they expect meaningful results there, as well. For complex reasons that we won’t get into here, LLMs tend to have a lot of randomness in the long tail of possible results.

So with software, if you ask the exact same question, you will always get the exact same answer. With LLMs, you might not.

LLMs are their own beast

Unit testing involves asserting predictable outputs for defined inputs, but this obviously cannot be done with LLMs. Instead, ML teams typically build evaluation systems to evaluate the effectiveness of the model or prompt. However, to get an effective evaluation system bootstrapped in the first place, you need quality data based on real use of an ML model. With software, you typically start with tests and graduate to production. With ML, you have to start with production to generate your tests.

Even bootstrapping with early access programs or limited user testing can be problematic. It might be ok for launching a brand new feature, but it’s not good enough for a real production use case.

Early access programs and user testing often fail to capture the full range of user behavior and potential edge cases that may arise in real-world usage when there are a wide range of users. All these programs do is delay the inevitable failures you’ll encounter when an uncontrolled and unprompted group of end users does things you never expected them to do.

Instead of relying on an elaborate test harness to give you confidence in your software a priori, it’s a better idea to embrace a “ship to learn” mentality and release features earlier, then systematically learn from what is shipped and wrap that back into your evaluation system. And once you have a working evaluation set, you also need to figure out how quickly the result set is changing.

Phillip gives this list of things to be aware of when building with LLMs:

Failure will happen—it’s a question of when, not if.
Users will do things you can’t possibly predict.
You will ship a “bug fix” that breaks something else.
You can’t really write unit tests for this (nor practice TDD).
Latency is often unpredictable.
Early access programs won’t help you.

Sound at all familiar? 😂

Observability-driven development is necessary with LLMs

Over the past decade or so, teams have increasingly come to grips with the reality that the only way to write good software at scale is by looping in production via observability—not by test-driven development, but observability-driven development. This means shipping sooner, observing the results, and wrapping your observations back into the development process.

Modern applications are dramatically more complex than they were a decade ago. As systems get increasingly more complex, and nondeterministic outputs and emergent properties become the norm, the only way to understand them is by instrumenting the code and observing it in production. LLMs are simply on the far end of a spectrum that has become ever more unpredictable and unknowable.

Observability—both as a practice and a set of tools—tames that complexity and allows you to understand and improve your applications. We have written a lot about what differentiates observability from monitoring and logging, but the most important bits are 1) the ability to gather and store telemetry as very wide events, ordered in time as traces, and 2) the ability to break down and group by any arbitrary, high-cardinality dimension. This allows you to explore your data and group by frequency, input, or result.

In the past, we used to warn developers that their software usage patterns were likely to be unpredictable and change over time; now we inform you that if you use LLMs, your data set is going to be unpredictable, and it will absolutely change over time, and you must have a way of gathering, aggregating, and exploring that data without locking it into predefined data structures.

With good observability data, you can use that same data to feed back into your evaluation system and iterate on it in production. The first step is to use this data to evaluate the representativity of your production data set, which you can derive from the quantity and diversity of use cases.

You can make a surprising amount of improvements to an LLM based product without even touching any prompt engineering, simply by examining user interactions, scoring the quality of the response, and acting on the correctable errors (mainly data model mismatches and parsing/validation checks). You can fix or handle for these manually in the code, which will also give you a bunch of test cases that your corrections actually work! These tests will not verify that a particular input always yields a correct final output, but they will verify that a correctable LLM output can indeed be corrected.

You can go a long way in the realm of pure software, without reaching for prompt engineering. But ultimately, the only way to improve LLM-based software is by adjusting the prompt, scoring the quality of the responses (or relying on scores provided by end users), and readjusting accordingly. In other words, improving software that uses LLMs can only be done by observability and experimentation. Tweak the inputs, evaluate the outputs, and every now and again, consider your dataset for representivity drift.

Software engineers who are used to boolean/discrete math and TDD now need to concern themselves with data quality, representivity, and probabilistic systems. ML engineers need to spend more time learning how to develop products and concern themselves with user interactions and business use cases. Everyone needs to think more holistically about business goals and product use cases. There’s no such thing as a LLM that gives good answers that don’t serve the business reason it exists, after all.

So, what do you need to get started with LLMs?

Do you need to hire a bunch of ML experts in order to start shipping LLM software? Not necessarily. You cannot (there aren’t enough of them), you should not (this is something everyone needs to learn), and you don’t want to (these are changes that will make software engineers categorically more effective at their jobs). Obviously you will need ML expertise if your goal is to build something complex or ambitious, but entry-level LLM usage is well within the purview of most software engineers. It is definitely easier for software engineers to dabble in using LLMs than it is for ML engineers to dabble in writing production applications.

But learning to write and maintain software in the manner of LLMs is going to transform your engineers and your engineering organizations. And not a minute too soon.

The hardest part of software has always been running it, maintaining it, and understanding it—in other words, operating it. But this reality has been obscured for many years by the difficulty and complexity of writing software. We can’t help but notice the upfront cost of writing software, while the cost of operating it gets amortized over many years, people, and teams, which is why we have historically paid and valued software engineers who write code more than those who own and operate it. When people talk about the 10x engineer, everyone automatically assumes it means someone who churns out 10x as many lines of code, not someone who can operate 10x as much software.

But generative AI is about to turn all of these assumptions upside down. All of a sudden, writing software is as easy as sneezing. Anyone can use ChatGPT or other tools to generate reams of code in seconds. But understanding it, owning it, operating it, extending and maintaining it… all of these are more challenging than ever, because in the past, the way most of us learned to understand software was by writing it.

What can we possibly do to make sure our code makes sense and works, and is extendable and maintainable (and our code base is consistent and comprehensible) when we didn’t go through the process of writing it? Well, we are in the early days of figuring that out, too. 🙃

If you’re an engineer who cares about your craft: do code reviews. Follow coding standards and conventions. Write (or generate) tests for it. But ultimately, the only way you can know for sure whether or not it works is to ship it to production and watch what happens.

This has always been true, by the way. It’s just more true now.

If you’re an engineer adjusting to the brave new era: take some of that time you used to spend writing lines of code and reinvest it back into understanding, shipping under controlled circumstances, and observing. This means instrumenting your code with intention, and inspecting its output. This means shipping as soon as possible into the production environment. This means using feature flags to decouple deploys from releases and gradually roll new functionality out in a controlled fashion. Invest in these—and other—guardrails to make the process of shipping software more safe, fine-grained, and controlled.

Most of all, it means developing the habit of looking at your code in production, through the lens of your telemetry, and asking yourself: does this do what I expected it to do? Does anything else look weird?

Or maybe I should say “looking at your systems” instead of “looking at your code,” since people might confuse the latter with an admonition to “read the code.” The days when you could predict how your system would behave simply by reading lines of code are long, long gone. Software behaves in unpredictable, emergent ways, and the important part is observing your code as it’s running in production, while users are using it. Code in a buffer can tell you very little.

This future is a breath of fresh air

This, for once, is not a future I am afraid of. It’s a future I cannot wait to see manifest. For years now, I’ve been giving talks on modern best practices for software engineering—developers owning their code in production, testing in production, observability-driven development, continuous delivery in a tight feedback loop, separating deploys from releases using feature flags. No one really disputes that life is better, code is better, and customers are happier when teams adopt these practices. Yet, only 11% of teams can deploy their code in less than a day, according to the DORA report. Only a tiny fraction of teams are operating in the way everybody agrees we all should!

Why? The answers often boil down to organizational roadblocks, absurd security/compliance policies, or lack of buy-in/prioritizing. Saddest of all are the ones who say something like, “our team just isn’t that good” or “our people just aren’t that smart” or “that only works for world-class teams like the Googles of the world.” Completely false. Do you know what’s hard? Trying to build, run, and maintain software on a two month delivery cycle. Running with a tight feedback loop is so much easier.

Just do the thing

So how do teams get over this hump and prove to themselves that they can have nice things? In my experience, only one thing works: when someone joins the team who has seen it work before, has confidence in the team’s abilities, and is empowered to start making progress against those metrics (which they tend to try to do, because people who have tried writing code the modern way become extremely unwilling to go back to the bad old ways).

And why is this relevant?

I hypothesize that over the course of the next decade, developing with LLMs will stop being anything special, and will simply be one skill set of many, alongside mobile development, web development, etc. I bet most engineers will be writing code that interacts with an LLM. I bet it will become not quite as common as databases, but up there. And while they’re doing that, they will have to learn how to develop using short feedback loops, testing in production, observability-driven development, etc. And once they’ve tried it, they too may become extremely unwilling to go back.

In other words, LLMs might ultimately be the Trojan Horse that drags software engineering teams into the modern era of development best practices. (We can hope.)

In short, LLMs demand we modify our behavior and tooling in ways that will benefit even ordinary, deterministic software development. Ultimately, these changes are a gift to us all, and the sooner we embrace them the better off we will be.

How to Communicate When Trust Is Low (Without Digging Yourself Into A Deeper Hole)

August 17, 2023January 10, 2024 mipsytipsyadvice, communication, culture, ethics28 Comments

This is based on an internal quip doc I wrote up about careful communication in the context of rebuilding trust. I got a couple requests to turn it into a blog post for sharing purposes; here you go.🌈✨🥂

In this doc I mention Christine, my wonderful, brilliant cofounder and CEO, and the time (years ago) when our relationship had broken down completely, forcing us to rebuild our trust from the ground up.

(Cofounder relationships can be hard. They are a lot like marriages; in their difficulty and intensity, yes, but also in that when you’re doing it with the right person, it’s all worth it. 💜)

Tips for Careful Communication

When a relationship has very little trust, you tend to interpret everything someone says in the worst possible light, or you may hear hostility, contempt, or dismissiveness where none exists. On the other side of the exchange, the conversation becomes a minefield, where it feels like everything you say gets misinterpreted or turned against you no matter how careful you are trying to be. This can turn into a death spiral of trust where every interaction ends up with each of you hardening against each other a little more and filing away ever more wounds and slights. 💔

Yet you HAVE to communicate in order to work together! You have to be able to ask for things and give feedback.

The way trust gets rebuilt is by ✨small, positive interactions✨. If you’re in a trust hole, you can’t hear them clearly, and they can’t hear you (or your intent) clearly. So you have to bend over backwards to overcommunicate and overcompensate.

There are lots of books out there on how to talk about hard topics. (We actually include a copy of “Crucial Conversations” in every new employee packet.) They are all pretty darn cheesy, but it’s worth reading at least one of them.

I’m not going to try and cover all of that territory. What follows is a very subjective list of tactics that worked for Christine and me when we were digging our way out of a massive trust deficit. Power dynamics can admittedly make things more difficult, but the mechanics are the same.

Acknowledge it is hard beforehand:

“I want to say something, but I am having a hard time with it.”
“I have something to say, but I don’t know how you’ll take it.”
“I need to tell you something and I am anxious about your reaction.”

What this does: forces you to slow down and be intentional about the words you’re going to use. It gives the other person a heads up that this was hard for you to say. Most of all, it shows that you do care about their feelings, and are trying to do your best for them (even if you whiff the landing).

… or check in afterwards.

“I’m not sure how that came across. Is there a better way I could have phrased it?”
“In my head that sounded like a compliment…how did you hear it?”
“Did that sound overly critical? I’m not trying to dwell on the past, but I could use your help in figuring out a better way.”

It’s okay if it’s minutes or hours or days later; if it’s still eating at you, ✨clear the fucking air.✨

Speak tentatively.

“Speak tentatively” is the exact opposite of the advice that people (especially women) tend to get in business. But it’s actually super helpful when the relationship is frayed because you are explicitly allowing that they may have a different perception, and making it safer to share it.

“From my perspective, it looks like these results might be missing some data… do you see the same thing?” opens the door for a friendly conversation based on concrete outcomes, whereas “You’re missing data” might sound accusatory and trigger fear and defensiveness.

Try to sound friendly.

Say “please” and “thank you” a lot. Add buffer words like “Hey there”, or “Good morning!” or “lol”. Even just using 🌱emojis🍃 will soften your response to an almost unsettling degree. This may seem almost insultingly simple, but it works. When trust is low, the lack of frills can easily be read as brusque or rude.

Take a breath.

If you are experiencing a physical panic response (sweating, heart racing, etc), announce that you need a few minutes before responding. Compose yourself. Firing off a reply while you are in fight-or-flight mode reliably leads to unintentional escalations.

If you need to take a few beats to read and process, take the time. But empty silence can also generate anxiety 🙂 so maybe say something to indicate “I’m listening, but I need a minute to absorb what you said”, or “I’m still processing”. (We often use “whoa…” as shorthand for this.)

(Alternately, if you find yourself really pissed, “whoa” becomes a great placeholder for yourself to get yourself under control 😬 before saying something you’ll regret having to deal with later.)

“The story in my head”.

When you are in a state where you are assuming the worst of someone and reading hostile intent into their words or actions, try to check yourself on those assumptions.

Repeat the words or behaviors back to them along with your interpretation, like: “The story in my head is that you asked me to send that status email because you don’t trust me to have done the work, or maybe even gathering evidence that I am not performing for a PIP.” This gives them the opportunity to reply and clarify what they actually meant.

Engineer positive interactions, even if you have to invent them.

Relationship experts say that there’s a magic ratio for happy, healthy relationships, which is at least five positive interactions for every one negative interaction. If you only interact with the people you have difficult relationships when you have something difficult to say, you are always going to dread it. Forever.

It might seem artificial at first, but look for chances to have any sort of positive interactions, and seize them.

Communicate positive intent.

In a low trust environment, you can assume everything you say will be read with a voice that is menacing, dismissive or sneering. It behooves you to pay extra attention to tone and voice, and to add extra words that overcommunicate your intended meaning. A neutral statement like “That number seems low”, or “Why is that number low?” will come out sounding brusque and accusatory, e.g. why isn’t that number growing? it’s your fault, you should know this, I blame you, you’re bad at your job. Not might: it will. Try to immunize your communication from distortion by saying things like,

“Hey, I know this just got dropped in your lap, but do you have any idea why this number is so low?”
“This number seems lower than usual. I’m wondering if it’s due to this other thing we tried. Do you have any better ideas?”
“I know it isn’t exactly in your wheel house, but can you help me understand this?”
“I’m new to this system and still trying to figure out how it works. Should this number be going down like this?”

It may seem excessive and time consuming, but it will save you time and effort overall because you will have fewer miscommunications to debug. ☺️

Give people the opening to do better.

We tend to make up our minds about people very quickly, and see them through that lens from then on. It takes work to open our selves up again.

“Assume positive intent” is a laudable goal, but in practice falls short. If every word someone says sounds accusatory or patronizing to you, what are you supposed to do with that advice? Just pretend you don’t hear it, or tell yourself they mean well? That’s not sustainable; your anger will only build up.

But if you can hold just enough space for the idea that they might mean well, then you can give them the opportunity to clarify (and hopefully use different words next time). Like,

Person A: “Why is that number low?”
Person B: “I’m not sure.”
(pause)
Person B: “…. Hey, sorry to interrupt, but the story in my head is that you think owning that number is part of my job, and now you’re upset with me, or you think I’m incompetent at my job.”
Person A: “OMG no, not at all. I’m just trying to figure out who understands this part of the system, since it seems like none of us do! 😃 Sorry for stressing you out!”

and maybe next time it will start off like…

Person A: “Hey, do you have any idea why this number is low? It’s a mystery … nobody I’ve talked to yet seems to know.” 🙂

Remember the handicaps, value the effort.

Ever meet someone you didn’t like online, and realize they’re terrific in person? Online communication loses sooooo much in transit. Christine and I know each other extremely well, and still sometimes we realize we’re reading way too much into each other’s written words. That’s when we try to remember to move it to “mouth words”, aka zoom or phone. Not as good as in person, but eons better than text.

Once you’ve met someone in person, it’s usually easier to read their written words in their voice, too.

Some people just aren’t great at written communication. Some people have neurodiversities that make it difficult for them to hear tone. Some people have English as a second language. And so on. Do give points for effort; if they’re trying, obviously, they care about your experience.

To the best of your ability, try to resist reading layers of meaning into textual communication; keep it simple, overcommunicate intent, and ask for clarity. And if someone is asking you for clarity, help them do a better job for you.

Helicopter Management and Other Mistakes

June 19, 2023July 14, 2023 mipsytipsybusiness, careers, culture, management4 Comments

You are a freshly minted manager. You come full of rage and frustration at the poor management you’ve endured and witnessed in tech, and you are god damn determined not to repeat all of those mistakes.

You are tired of reporting to a manager who isn’t transparent with you, who hoards critical information and isn’t forthcoming about changes that impact you. You are tired of not being listened to or treated like a cog, so you swear to really listen and take your reports seriously.

You have seen sooooo many managers who failed to develop their people or sponsor them for growth opportunities, who blamed their team and hung them out to dry instead of having their back behind closed doors. Managers who didn’t seem to care about you as people, or who never made it feel safe to say, “I need a mental health day”. Managers who dangled the promise of a promotion, but even though you are doing the work, the recognition never comes.

Fuck that shit. You aren’t going to do ANY of it.

And … you don’t! 🎉

🌸🍃 You make time in your 1x1s to ask about their personal lives and hobbies — you are careful not to pry or be intrusive, just showing that you care. You urge them to take vacation, often. You remind them, firmly, not to be a hero. You model the behavior of taking mental health days to show that not only is it safe, but managers need them too.

🌸🍃 You ask them about their career goals and aspirations. You make it your personal mission to get them promoted, so you frequently check in with them to make sure they’re on the right path. You keep an eye out for things they do that are above and beyond, and for strengths that make them special. They always know you are on their side.

🌸🍃 If you hear about a clash or a conflict between them and another team member, you quickly jump in to figure out what’s going on and make sure it gets resolved, with each person feeling heard before the conflict has time to marinate or fester.

🌸🍃 When reviews comes around, you write warm, passionate essays for each of your direct reports, listing all the things they have done and all the ways they have grown. You go in to manager calibrations fully prepped to advocate for each of your people to get the promotions and rewards they so richly deserve.

🌸🍃 If someone on your team ends up needing more help, whether that’s keeping them productive and on track or helping with prioritization or conflicts… whatever it is, you are there to help turn the situation around. This person was struggling under their old manager, maybe even close to being let go, but under your care they are thriving.

🌸🍃 Nobody ever leaves your team. This is a point of quiet pride for you. People want to transfer to your team, but never from. There may have been a couple close calls, but you are always able to save the day by talking it out with the person and figuring out what they need in order to stay.

🌸🍃 You take pride in your transparency and the democratized ethos of your team, where you collectively determine your priorities and no one feels pressured into doing something they don’t want to do.

Bottom line: you are a GREAT manager.

Right???

After all, your team ranks sky high on every company survey on employee happiness, manager trust, and autonomy and sense of purpose. Your team fucking LOVES YOU. You’re pretty sure they would follow you to your next job, if you left. So maybe you ARE the world’s greatest manager.

Or maybe…you are heavily optimizing for one aspect of the management role, the part where you interface with your direct reports as an ally and coach. You might even be optimizing to the extent that you are neglecting or outright harming other aspects of the job.

Rookie Mistake #1: Only Managing Down

But management means coaching and supporting the people on your team, right? What else is there?

Well.. a lot, actually. Like, the business needs to succeed, for starters. ☺️ And there are a bunch of other relationships that matter besides your own direct reports. A good, strong manager needs to care about:

Goals and planning. Managers are generally responsible for crafting a team roadmap out of the impossible mess of company strategy requirements, requests from other teams, product roadmap commitments, and KTLO (keep the lights on) work. Some companies also use OKRs.
Right-sizing workloads. There is always at least 10x as much work to be done as cycles to do work, which means estimating how much your team can deliver, planning that work, and dealing with the inevitable surprises that come up during execution. How do you balance urgency vs importance? It is YOUR job to make sure your team isn’t overcommitted.
Stakeholder management. Does your team have a reputation for delivering quality work when they say they will, more or less on schedule? Are you a good neighbor to other teams, or do they feel like anything they ask for goes into a black hole? This is largely determined by you.
Managing up. Your manager relies on you to provide enough visibility into your team that they can (at minimum) make good decisions where your team is involved, and help head off any problems or conflicts before they escalate.
Managing up (another sort) is the relationships you build and impressions you leave with your skip-level and other adjacent leaders. You are your team’s representative and ambassador. Leaders form a view of the org based on scraps of data. For the sake of your team: give good scraps.
Managing ~~out~~ horizontally. Building great relationships and a web of mutual support with your peers. Sharing context with each other. Managers are like the nervous system, carrying signal from point to point.
Contributing to the organization your team sits in, and its standards, policies, and structural integrity. This is the one most likely to suffer if a manager is laser focused on their own team. This means things like…applying the job ladder fairly and consistently, without playing favorites. Engaging in a dialogue with the ladder rather than bending the rules or making an exception.

As a manager, you have been granted certain formal powers by the org, to be used for the benefit of the org. This means you have a responsibility to care for the organization, and your team within that context.

You shouldn’t be advocating for the benefit of your team members, you have a greater responsibility to the rules and categories of the system, which you collectively maintain and agree upon. The system can’t survive if every manager is gaming the rules on behalf of their team. The system only works if every manager is playing fair.

As a line manager, the work you do interfacing with your team will likely be 50-75% of your time and energy … and impact. But this ratio changes as you go up the org chart. As a VP? Maybe 10-20% of your energy and time can go to your direct reports.

The higher up the ladder you go, the less important your bedside manner becomes and the more important your strategic direction becomes. You are first and foremost responsible for the company’s success, not for your reports’ feelings and career development.

Rookie mistake #2: Helicopter management

If rookie manager mistake number one is thinking that management consists of coaching and interfacing with your team, mistake number two is closely related. Mistake number two is … overmanaging the team, coddling people, and basically never allowing anyone to fail. I think of it as “helicopter managing”.

Helicopter management consists of overly identifying with your team and their needs and wants, instead of taking a step back and considering them in the full context of the organization, or letting them take risks and stand on their own two feet. You’re their manager, not their babysitter.

I have a personal story to illustrate this.

Once upon a time, many years ago, I had a team member who was energetic, highly talented, and a little high strung. I ended up spending a lot of time managing their relationships with other team members, keeping them on track with their projects, and helping them manage their emotional state in general. They nearly left in a dudgeon one time, and I think most managers would have let them go, but I saved the day and they stayed. I was actually really proud of the fact that I had retained them and kept them high-functioning for years. If you asked me, I would have shelved this under my successes, maybe even “proudest manager moments”.

Years later, though, I look back on this situation through very different eyes. Yes, I retained them at the company / on the team, with decent relationships, and they did a lot of good work! But should I have?

At what cost?

Most weeks, I probably spent 50-75% of my total emotional bandwidth on this one person’s needs. For years. Is this the best thing I could have done for the company with all that time and energy? Probably not!

Was this the best thing I could have done for them? I don’t even think it was that either! All that my coddling ultimately did was teach them the wrong lessons, and prevent them from learning the right ones. It delayed those lessons by a few years, and made learning them all the more painful when they finally came.

There are no clear bright lines here. But it’s worth checking in with yourself from time to time, and asking hard questions.

You spent all that time coaching and doing a diving save to retain that person …. but should you have? Is this really the best place for them to be at this point in their career?
Or let’s say you managed to turn around someone’s performance from failing to succeeding. Great!! But are you confident they are set up to excel, or are they always going to be hovering on the lower bound of acceptable performance? Are you going to be having this same discussion again next quarter?
Consider all that time you spent intimately entwined with every detail of every technical project your entire team was working on, reading every PR and design doc. Should you have? Or did you unintentionally deprive them of some agency, while cheating yourself out of time you should have spent becoming a better leader, strengthening your org, or understanding next year’s challenges?
Are you giving people only positive feedback? This is a common rookie manager mistake, and it often comes from a place of kindness, or overcompensating for overly negative environments. But you are not only cheating your people of opportunities for growth, you are teaching them that growth is something to be feared and avoided.
Or are you cheerleading people so intensely that they come away with a lopsided view of how valuable or advanced their skills are? Are you promoting fast and loose, so they grow to equate promotions with career development? Have you been spoon feeding them growth, or are they developing autonomy over time?

This can be especially unfortunate at higher levels, where autonomy is part of the definition of being a senior+ engineer. You might be stifling them and not allowing them to exercise that agency, or even develop that skill. For senior contributors, autonomy is what they bring. You gotta let them do it.

This shit is challenging. There are no simple answers. The “right” answer is often only obvious in retrospect, months or years later. Everyone needs help sometime, some of us more than others, and that’s okay.

But is it sustainable? What price will you pay?

What I do know is that if you haul someone over the finish line, that is not a success. If you’re going to be having the same hard conversations with them again in one month, three months, six months…that is not success. If they are going to have a rough landing the next time they change teams, that is not success, nor is it in their best interest. And if your team is overly dependent on you, you aren’t actually doing your job.

And honestly? People really WANT to be challenged. They crave it! Or at least the people you want to work with do.

Rookie Mistake #3: Your view of the system is incomplete

I’m only going to touch on this one very briefly; it’s long and complicated, and probably deserves a post of its own.

Systems thinking is a core skill for both managers and engineers. It’s not a skill we are born with; it takes a lot of practice and failure to develop good instincts for debugging complex systems. As an engineering manager, you may have spent 10+ years writing software and learning how computers work, but you have hardly begun to understand how business and organizational systems work.

This explains a lot when it comes to the empathy gap between engineers and management, I believe. 🙃

We spend a lot of time talking about empathy these days — empathy between teams, people, neurotypes; holding space for the fact that nobody is always at their best, etc. Yet engineers can still be incredibly dismissive and judgey towards management actions and organizational decisions.

We see a decision that doesn’t make sense to us, or that we wouldn’t have made, and we write it off as being selfish, uninformed, incompetent, stupid, money-grubbing, bureaucratic, untrustworthy, craven, selling out. Or — maybe worst of all — we shrug and say something cynical about how this kind of thing always happens in business. Or they’re out to get us, or they never listen to us, or it shows how much they don’t give a shit about us..

Far be it to me to excuse corporate venality, or to try and blow smoke up your ass about your leaders’ motives. But in many, many of these situations, this actually represents a failure of systems thinking when it comes to imagining the complex business, corporate, and people systems your leaders are operating in.

When you find yourself thinking things like:

“Why am I hearing this feedback so late, in such a roundabout way? Why didn’t they just come to me right away and tell me directly, and I could have fixed it so much sooner!”
“Why would they hire someone external to fill that role, instead of promoting the person who has been doing the work just fine in the meantime? Typical exec move; they never see the potential in the people they have, they always want to get someone who has already done the job before.”
“Why is our roadmap changing yet again? Why is this getting dropped in our lap? Our director doesn’t seem to know anything about building good software.”
“Why didn’t I get invited to that meeting, when it was about MY budget and MY workload? You can’t even get a seat at the table around here unless you have a director title or report to the CEO.”
“Why is that person being given ANOTHER chance? If they weren’t a straight white guy, they would have been fired a year ago.”

… or anything else that boils down to “other managers are stupid, hypocritical, or bad at their jobs”, stop yourself, and first try to understand “under what circumstances might their action be a reasonable one, or even the right one?”

Approaching people systems problems with curiosity, empathy, and the full awareness that you may never know the entire story (and there may be good reasons for this!) will make you a better coworker and a much more effective leader.

And if you are working as a manager at a company where you have enough evidence to prove that you cannot, should not take such a generous view of your peers, then maybe don’t. Like, if you have a professional responsibility to represent an organization you can not ethically represent… I would suggest not doing that. ¯\_(ツ)_/¯ If you can.

Your View of Your Manager Is Incomplete

One of the most challenging things to deal with, in my (limited) experience as an exec, is when you have a manager who is well-liked by their team but fundamentally ineffective in the role, so you have to replace them. Then you are left with a team left feeling bereft and confused, and you can’t just give them a list of all the ways their manager was actually dropping the ball and not doing their job well, because people deserve some privacy and dignity. You pretty much just have to suck it up, and hope that you have enough trust banked for them not to quit.

It’s entirely possible for a manager to be beloved by their entire team, while having a corrosive effect on the system around them. Sometimes it’s their very willingness to bend the system for their people’s benefit that generates that loyalty.

An unwary manager may create a sort of island within the company, where the team does not feel part of — may even feel separate from, superior to, or suspicious of — the broader org or the company. Team members may feel like “this is the only team I would ever want to belong to at this company”, or “my manager is great, they protect us from all the big company bullshit”, or “it’s us against the world”, or “nobody understands us, except our manager”. These are seductive dynamics to slip into because tribalism is so powerful. The more apart you feel from the company, the more tightly you may bond with each other.

I’m not judging you. I was that manager at Parse, to some extent, after the Facebook acquisition. I did not give a shit about the org, I only cared about my team. So…I get it. I still wouldn’t want me as a manager in my org.

What would you do?

Put yourself in your senior leadership’s shoes. What would YOU do if you had to choose between a good, reliable manager who gets the job done for the org and isn’t particularly beloved by their reports (she’s not awful, the team thinks she’s “fine”), vs a manager who is beloved by their reports and cares about their career growth deeply, but is weak at everything else? What will manager #2 cost the rest of the org, and who will bear those costs?

Don’t make your leadership make that choice. Be a manager who is good to your team, and good to the organization too.

Honestly, it is healthy for a manager to not identify too closely with their team. You should stand with them, but a step or two apart. After all, your job is to be pushing or pulling them in a direction, not just standing still and … marinating.

If you identify with them too closely, it can get very hard to tell your reports hard things. You may empathize with them so much that you put their feelings above the need to get shit done. You can be friends with your team, just like parents can be friends with their children, but the friendship doesn’t come first. Your formal role comes first.

On being a new manager who cares a lot

There’s something really beautiful about the energy and dedication that brand new managers bring to the role. Some of the most spectacular results I’ve ever seen for individual team members have come from teams with first time managers, who are determined and idealistic and pouring their whole heart and soul into the people reporting to them. They haven’t yet learned to pace themselves or to be more well-rounded with their time and energy, and sometimes a person can soak up that attention and turn a failing situation around into a thriving one.

I would never tell a manager that they should care less. Caring for people is the beating heart of this job. It doesn’t matter how efficient and effective you are at delivering feedback, managing people out, and planning roadmaps if you don’t truly give a shit about the people you serve. Even as you rise in the ranks and people-interfacing becomes a smaller % of what makes you good at your job, caring is still absolutely essential.

So I hope the message of this post doesn’t come across as “you think you’re a good manager, but here’s why you actually suck”. ☺️ You got into this role because you cared, and this is valuable. Never lose touch with it.

The message is simply that it took me years and years to learn that there is more to being a great manager than caring about my team. I hope you can learn it faster than I did.

Questionable Advice: “How can I drive change and influence teams…without power?”

June 5, 2023July 14, 2023 mipsytipsyadvice, deploys, devops, engineering, influence, management1 Comment

Last month I got to attend GOTO Chicago and give a talk about continuous deployment and high-performing teams. Honestly I did a terrible job, and I’m not being modest. I had just rolled off a delayed redeye flight; I realized partway through that I had the wrong slides loaded, and my laptop screen was flashing throughout the talk, which was horribly distracting and means I couldn’t read the speaker notes or see which slide was next. 😵 Argh!

Anyway, shit happens. BUT! I got to meet some longstanding online friends and acquaintances (hi JJ, Avdi, Matt!) and got to eat some of Hillel Wayne’s homemade chocolates, and the Q&A session afterwards was actually super fun.

My talk was about what high performing teams look like and why it’s so important to be on one (spoiler: because this is the #1 way to become a radically better engineer!!). Most of the Q&A topics therefore came down to some version of “okay, so how can I help my team get there?” These are GREAT questions, so I thought I’d capture a few of them for posterity.

But first… just a reminder that the actual best way to persuade people to listen to you is to make good decisions and display good judgment. Each of us has an implicit reputation score, which formal power can only overcome to an extent. Even the most junior engineer can work up a respectable reputation over time, and even principal engineers can fritter theirs away by shooting off at the mouth. 🥰

“how can I drive change when I have no power or influence?”

This first question came from someone who had just landed their first real software engineering job (congrats!!!):

“This is my first real job as a software engineer. One other junior person and myself just formed a new team with one super-senior guy who has been there forever. He built the system from scratch and knows everything about it. We keep trying to suggest ideas like the things you talked about in your talk, but he always shoots us down. How can we convince him to give it a shot?”

Well, you probably can’t. ☺️ Which isn’t the end of the world.

If you’re just starting to write software every day, you are facing a healthy learning curve for the next 3-5 years. Your one and only job is to learn and practice as much you possibly can. Pour your heart and soul into basic skills acquisition, because there really are no shortcuts. (Please don’t get hooked on chatGPT!!)

I know that I came down hard in my talk on the idea that great engineers are made by great teams, and that the best thing most people can do for their career is to join a high-performing, fast-moving team. There will come a time where this is true for you too, but by then you will have skills and experience, and it will be much easier for you to find a new job, one with a better culture of learning.

It is hard to land your first job as a software engineer. Few can afford to be picky. But as long as you are a) writing code every day, b) debugging code every day, and c) getting good feedback via code reviews, this job will get you where you need to go. When you’re fluent and starting to mentor others, or getting into higher level architecture work, or when you’re starting to get bored … then it’s time to start looking for roles with better teachers and a more collaborative team, so your growth doesn’t stall. (Please don’t fall into the Trap of the Premature Senior.)

This is an apprenticeship industry. You’re like a med student right now, who is just starting to do rounds under the supervision of an attending physician (your super-senior engineer). You can kinda understand why he isn’t inclined to listen to your opinions on his choice of stethoscope or how he fills out a patient chart. A better teacher would take time to listen and explain, but you already know he isn’t one. 🤷

I only have one piece of advice. If there’s something you want to try, and it involves doing engineering work, consider tinkering around and building it after hours. It’s real hard to say no to someone who cares enough to invest their own time into something.

“how can I drive change when I am a tech lead on a new team?”

“I have the same question! — except I’m a tech lead, so in theory I DO have some power and influence. But I just joined a new team, and I’m wondering what the best way is to introduce changes or roll them out, given that there are soooo many changes I’d like to make.”

(I wrote a somewhat scattered post a few years ago on engineers and influence, or influence without authority, which covers some related territory.)

As a tech lead who is new to a team, busting at the seams with changes I want to make, here’s where I’d start:

Understand why things are the way they are and get to know the personalities on your team a bit before you start pitching changes. (UNLESS they are coming to you with arms outstretched, pleading desperately for changes ~fast~ because everything is on fire and they know they need help. This does happen!)
Spend some time working with the old systems, even if you think you already understand. It’s not enough for you to know; you need to take the team on this journey with you. If you expect your changes to be at all controversial, you need to show that you respect their work and are giving it a chance.
Change one thing at a time, and go for the developer experience wins first. Address things that will visibly pay off for your team in terms of shipping faster, saving time, less frustration. You have no credibility in the beginning, so you want to start racking up wins before you take on the really hard stuff.
Roll up your sleeves. Nothing buys a leader more goodwill than being willing to do the scut work. Got a flaky test suite that everybody has been dreading trying to fix? I smell opportunity…
Pitch it as an experiment. If people aren’t sold on your idea for e.g. code review SLAs, ask if they’d be willing to try it out for three weeks just as an experiment.
Strategically shop it around to the rest of the team, if you sense there will be resistance…

At this point in my answer 👆 I outlined a technique for persuading a team and building support for a plan or an idea, especially when you already know it’s gonna be an uphill battle. Hillel Wayne said I should write it up in a blog post, so here it is! (I’ll do anything for free chocolate 😍)

“How can I get people on board with my controversial plan?”

So you have a great idea, and you’re eager to get started. Awesome!!! You believe it’s going to make people’s lives better, even though you know you are going to have to fight tooth and nail to make it happen.

What NOT to do:

Walk into the team meeting and drop your bomb idea on everyone cold:

“Hey, I think we should stop shipping product changes until we fix our build pipeline to the point where we can auto-deploy each merge set to production, one at a time, in under an hour.” ~ (for example)

…. then spend the rest of the hour grappling with everybody’s thoughts, feelings, and intense emotional reactions, before getting discouraged and slinking away, vowing to never have another idea, ever again.

What to do instead:

Suss out your audience. Who will be there? How are they likely to react? Are any of them likely to feel especially invested in the existing solution, maybe because they built it? Are any of them known for their strong opinions or being combative?

Great!!! Your first move is to have a conversation with each of them. Approach them in the spirit of curiosity, and ask what they think of your idea. Talking with them will also help you hash out the details and figure out if it is actually a good idea or not.

Your goal is to make the rounds, ask for advice, identify any allies, and talk your idea through with anybody who is likely to oppose you…before the meeting where you intend to unveil your plan. So that when that happens, you have:

given people the chance to process their reactions and ask questions in private
ensured that key people will not feel surprised, threatened, or out of the loop
already heard and discussed any objections
ideally, you have earned their support!

Even if you didn’t manage to convince every person, this was still a valuable exercise. By approaching people in advance, you are signaling that you respect them and their voice matters. You are always going to get people’s absolute worst reactions when you spring something on them in a group setting; any anxiety or dismay will be amplified tenfold. By letting them reflect and ask questions in private, you’re giving time for their better selves to emerge.

What to do instead…if you’re a manager:

As an engineer or a tech lead, you sometimes end up out front and visible as the owner of a change you are trying to drive. This is normal. But as a manager, there are far more times when you need to influence the group but not be the leader of the change, or when you need to be wary of sounding like you are telling people what to do. These are just a few of the many reasons it can be highly effective to have other people arguing on your behalf.

In the ideal scenario, particularly on technical topics, you don’t have to push for anything. All you do is pose the question, then sit back and listen as vigorous debate ensues, with key stakeholders and influential engineers arguing for your intended outcome. That’s a good sign that not only are they convinced, they feel ownership over the decision and its execution. This is the goal! 🌈

It’s not just about persuading people to agree with you, either. Instead of having a shitty dynamic where engineers are attached to the old way of doing things and you are “dragging them” into the newer ways against their will, you are inviting them to partner with you. You are offering them the opportunity to lead the team into the brave new world, by getting on board early.

(It probably goes without saying, but always start with the smallest relevant group of stakeholders, and not, say, all of engineering, or a group that has no ownership over the given area. 🙃 And … even this strategy will stop working rather quickly, if your controversial ideas all turn out to be disastrous. 😉)

“How do I know where to even start?!? 😱”

Before I wrap up, I want to circle back to the question from the tech lead about how to drive change on a team when you do have some influence or power. He went on to say (or maybe this was from a third questioner?*):

“There is SO MUCH I’d like to do or change with our culture and our tech stack. Where can I even start??”

Yeah, it can be pretty overwhelming. And there are no universal answers… as you know perfectly well, the answer is always “it depends.” ☺️ But in most cases you can reduce the solution space substantially to one of the two following starting points.

1. Can you understand what’s going on in your systems? If not, start with observability.

It doesn’t have to be elegant or beautiful; grepping through shitty text logs is fine, if it does the trick. But do any of the following make you shudder in recognition?:

If I get paged, I might lose the rest of the afternoon trying to figure out what happened
Our biggest problem is performance and we don’t know where the time is going
We have a lot of flaky, flappy alerts, and unexplained outages that simply resolve themselves without our ever truly understanding what happened.

If you can’t understand what’s going on in your system, you have to start with instrumentation and observability. It’s just too deadly, and too risky, not to. You’re going to waste a ton of time stabbing around in the dark trying to do anything else without visibility. Put your glasses on before you start driving down the freeway, please.

2. Can you build, test and deploy software in under an hour? If not, start with your deploy pipeline.

Specifically, the interval of time between when the code is written and when it’s being used in production. Make it shorter, less flaky, more reliable, more automated. This is the feedback loop at the heart of software engineering, which means that it’s upstream from a whole pile of pathologies and bullshit that creep in as a consequence of long, painful, batched-up deploys.

Here’s a talk I’ve given a few times on why this matters so much:

You pretty much can’t fail with one of those two; your lives will materially improve as you make progress. And the iterative process of doing them will uncover a great deal of shit you should probably know about.

Cheers! 🥂

charity.

* My apologies if I remembered anyone’s question inaccurately!

Ritual Brilliance: How a pair of Shrek ears shaped Linden Lab culture by making failure funny — and safe

May 30, 2023February 19, 2025 mipsytipsyblameless retros, culture, devops, linden lab2 Comments

[Originally posted on the now-defunct “Roadmap: A Magazine About Work” website, on May 30th, 2023. A pretty, nicely-formatted PDF version of this article can be downloaded here. Thanks to Molly McArdle for editing!]

If you talk to former Lindens about the company’s culture—and be careful, because we will do so at length—you will eventually hear about the Shrek ears.

When you saw a new person wearing the Shrek ears, a matted green-felt headband with ogre ears on it, you introduced yourself, congratulated them warmly, and begged to hear the story of how they came to be wearing them. Then you welcomed the new person to the team (“You’re truly one of us now!”) and shared a story about a time when you did something even dumber than they did.

My first job after (dropping out of) college was at Linden Lab, the home of Second Life. I joined in 2004 and stayed for nearly six years, during which the company grew from around 25 nerds in a room to around 400 employees who worked out of offices in Brighton, San Francisco, Menlo Park, and Singapore, or their own homes—wherever they were.

When I think back on that time now, almost two decades later, I’m puzzled by the Shrek ears phenomenon. I wasn’t exactly powerful then, at barely 20 years old. Not only was this my first real job, I was also the first woman engineer, and I made tons of mistakes. Shouldn’t I have found the practice of being systematically singled out and spotlighted for my errors humiliating, shaming, and traumatic?

Yet I remember loving the tradition and participating with joy and vigor. Everyone else seemed to love it, too. The practice spread beyond engineering and out into the rest of the company, not by fiat but because individual people would voluntarily track down the Shrek ears and put them on their own head. (I’m not imagining this, right?)

Step 1, break production; Step 2, put on Shrek ears

Here’s how it worked: The first time an engineer broke production or caused major outage, they would seek out the ears and put them on for the day. The ears weren’t a mark of shame—they were a badge of honor! Everyone breaks production eventually, if they’re working on something meaningful.

If people saw you wearing the ears, they would eagerly ask, “What happened? How did you find the problem? What was the fix?” Then they would regale you with their own stories of breaking production or tell you about the first outage they caused. If the person was self-flagellating or being too hard on themselves, the Shrek ears gave their colleagues an excuse to kindly but firmly correct it on the spot. It was Linden’s way of saying, Hey, we don’t do that here: “You did the reasonable thing! How can we make the system better, so the next person doesn’t stumble into the same trap?”

In those days, Linden was running a massively distributed system across multiple data centers on three continents, and doing so without the help of DevOps, CI/CD, GitHub, virtualization, the cloud, or infrastructure as code. We had an incredibly high-performing operations team, with a thousand-to-one server-to-ops engineer ratio, which was a real achievement in the days when the role required doing everything from racking and stacking boxes in the colocation center to developing your own automation software.

Failures were just fucking inevitable. In a world like that, devoid of the entire toolchain ecosystem we’ve come to rely on, you just had to learn to roll with it, absorb the hits, and keep moving fast. You could only test so much in staging; it was more important to get it out into production and watch it—understand it—there. It was better to invest in swift recovery, graceful degradation, and decoupling services than to focus on trying to prevent anything from going wrong. (Still is, as a matter of fact.)

This might all sound a little overwrought to you—maybe even dangerous or irresponsible. Didn’t we care about quality? Were we bad engineers?

The Shrek ears were “blameless retros” before there were blameless retros

I assure you, we cared. The engineers I worked with at Linden were of at least as high a caliber as the engineers I later worked with at Facebook (and a whole lot more diverse). In this specific place and time, the Shrek ears were what we needed to alleviate paralysis and fear of production, and to encourage the sharing of knowledge—even if anecdotal—about our systems.

In retrospect, the Shrek ears were a brilliant piece of social jujitsu. There was an element of shock value or contrarianism in celebrating outages instead of getting all worked up about them. But the larger purpose of the ears was to reset people’s expectations (especially in the case of new hires) and reprogram them with a different set of values: Linden’s values.

In the years since those early days at Linden, the industry has developed an entire language and set of practices around dealing with the aftermath of incidents: blameless post mortems, retrospectives, and so on. But those tools weren’t available to us at the time. What we did have was the Shrek ears. A couple of times a month something would break, the ears would be claimed, and we would all go around reminding one another that failure is both inevitable and ridiculous, and that no one is going to get mad at you or fire you when it happens.

Failure is always a question of when, not if

It’s important to note that you never saw anyone get teased or shamed for wearing the ears or for breaking production. There was a script to follow, and we all knew it. We learned it from watching others put on the ears, or by donning them ourselves. On a day when the Shrek ears had appeared, people would gather around at lunch or at the bar after work and swap war stories, one-upping one another and laughing uproariously.

Every new engineer was told, “If you never break production, you probably aren’t
doing anything that really matters or taking enough risks.”

It’s also important to emphasize that the ears were opt-in, not opt-out. You didn’t have to do it. And if you did take them, you could expect a wave of sympathy, good humor, and support. It affirmed that you deserved to be here, that you were part of the team.

And though the Shrek ears started in engineering, people in sales, marketing, accounting, and other departments picked them up over the years. It was a process of voluntary adoption, not a top-down policy. Someone would announce in IRC that they were wearing the ears today, and why. The camaraderie and laughter that ensued were infectious—and made it easier and easier over time for people to be transparent about what wasn’t working.

Rituals exist to instill values and train culture

In Rituals for Work, Kursat Ozenc defines rituals as “actions that a person or group does repeatedly, following a similar pattern or script, in which they’ve imbued symbolism and meaning.” Ritual exists to instill a value, create a mindset, or train a reflex.

And this particular ritual was extremely effective at taking lots of scared engineers and teaching them, very quickly:

✨ It is safe to fail✨
✨ Failure is constant✨
✨ Failure is fucking hilarious✨

At Linden, failure was not something to be ashamed of or to hide from your teammates. We understood that it’s not something that happens only to careless or inexperienced people. In fact, the senior people have the funniest fuckups—because what they are trying to do is insanely hard. The Shrek ears taught us that you fail, you laugh, you drink whiskey, you move on.

Other companies had similar rituals around the same time—Etsy famously had the “three-armed sweater,” which they would pass around to whoever had last broken production. But I’ve never again worked at a place where mistakes were discussed as freely and easily across the entire company as they were at Linden Lab. And I think the Shrek ears had a lot to do with that.

Their point was never to single out the person who had made a mistake and humiliate them, but the exact opposite. By putting on the ears, you said not just “Hi, I made a mistake” but also “I’m going to be brave about it, so we can all collectively learn and improve.” It was a ritualized act of bravery rewarded by affirmation, empathy, and acceptance. At Linden, the Shrek ears weren’t just a terrific tool for promoting team coherence and creating a sense of belonging. They also provided structure to help individuals and teams recover from scary events, and even traumas.

In so many ways, Linden Lab was ahead of its time

Linden was an extremely strange workplace when I was there, and it inspired unusually strong devotion, which we self-deprecatingly referred to as “the Kool-Aid.” It can be difficult to convey just how radical and weird it was at the time because the world has changed so much since then, and so many of the company’s “weird” philosophies have since gone mainstream. (Though not all: using “Kool-Aid” as a casual phrase to denote “excessive enthusiasm” or “cult-like devotion” is now recognized by many as being in poor taste. After all, people actually died at the Jonestown massacre.)

In a lot of ways, Linden culture (and Second Life technology) was profoundly, recognizably modern, and similar to the best workplaces of today, 20+ years later.

Philip Rosedale, Linden’s founder and CEO, is an inventor and technologist who believed it was every inch as interesting and important to experiment with company culture as with the virtual worlds we built. Except we did it all from scratch: building the technology and the culture together. And this led us down some weird rabbit holes, such as a cron job that rsynced the entire file system down over thousands of live servers every night. And the Shrek ears.

There was a period when “Choose your own work” was a company core value, and there were effectively no managers. (Not every experiment worked!) We went all-in on a fully distributed company culture at a time when practically no one else had. We ran a massively distributed, high-concurrency virtual world at a time before microservices, sharded databases, config management virtualization, AWS, or SRE and DevOps.

I can understand why people now find this story horrifying

With the distance of time, I get why the Shrek ears might make you recoil. If you think “That sounds awful! What kind of monsters would do that to each other?”—you are far from alone. Any time I mention the story in public, a sizable minority of people are aghast and appalled. Representative quotes include:

“I hope you realize how many people you traumatized by doing this to them.”

“I wonder how many introverted people found this excruciating but were too
afraid to say so.”

“Office bullying is fucked up even with cute Shrek ears.”

Even:

“We heard about the Shrek ears from an engineer we interviewed. He was telling us how great they were, but we were all so horrified that we declined to hire him because of it.”

And they’re right. It sounds awful to us now. It really does! It sounds like we were singling people out for their failures, like a dunce cap. I wouldn’t be surprised to someday learn that, in fact, a small number of people did felt pressured into using the ears, or hated them and were too afraid to say something. But how do we account for the fact that this tradition was so deeply beloved by so many—and that we are still fondly reminiscing about it more than 15 years later? It had a purpose.

Linden Lab was an incredibly progressive company for its time: very anti-hierarchical, very much about empowering people to be creative and independent. It also was by far the most diverse company I’ve ever worked in (other than Honeycomb, which I cofounded and where I’m CTO), with lots of women and genderqueer and trans people and people of color. We were way out on the sensitive branch relative to tech at that time. It’s tough to square this knowledge of what Linden was like as a place with the reactions some people outside the organization have to the Shrek ears.

I think this is, above all, a sign of progress. So many questionable practices that were ordinary back then—like referring to everyone as “guys,” using terms like “master/slave” for replication, or throwing alcohol-sloshed parties—are now rightfully frowned upon. We have become more sensitive to people’s differences and more clued into the power dynamics of the workplace. It’s far from perfect, but it is a lot better.

As a ritual, the Shrek ears were powerful and did the job. They were also fun—proving once again that making something goofy is the best way to make it stick. But I can’t imagine plopping Shrek ears on a new hire who has just broken production in 2023. And honestly, I think that’s probably a good thing. It’s time for new rituals.

Choose Boring Technology Culture

May 1, 2023July 14, 2023 mipsytipsybusiness, culture, startupsLeave a comment

Honeycomb recently announced our $50M Series D funding round. We aren’t the type to hype this a lot; Emily summed it up crisply as, “Living another day on someone else’s money isn’t business success, even though it is a lovely vote of confidence.”

Agreed. The vote of confidence does mean more than usual, given the dire state of VC funding these days, but…raising money is not success. Building a viable, sustainable company is success.

Whenever we are talking to investors, something that inevitably comes up is what a bomb ass team 🌈 we have. They have always been impressed by our ability to recruit and retain marquee names, people we “shouldn’t have been able to get” at our stage; honestly it’s even better than they realize, because we have heavy hitters all up and down the company, most of whom simply aren’t as well known. 😉 (Fame, and this may shock you, is not a function of talent.)

People join Honeycomb for many reasons, but “culture” is one of the most commonly cited. We have never been shy about talking about the ways we think tech culture sucks, or the experiments we are running. But this has given rise to the occasional impression that we are primarily cultural innovators who occasionally write software. We really aren’t.

In fact, I’d say the opposite is true. We try to choose boring culture.

What The Fuck Does “Culture” Even Mean?

Ok, so this is where the problem starts. This is why it grates on my nerves any time someone starts making pronouncements about how “your culture is bad”, “culture is the problem”, “fix your broken culture”… AUUGGGHHH. Those sentences are MEANINGLESS.

What does “culture” even mean?? Let’s consult the interwebs:

Culture: “An umbrella term which encompasses the social behavior, institutions and norms for a group; knowledge, beliefs, arts, laws, customs, capabilities, and habits of those individuals”
Culture: “The shared values, goals, attitudes and practices that characterize an organization; working environment, company policies and employee behavior”
Culture: “Maintain tissue cells, bacteria, etc in conditions suitable for growth”

Well, at least that last one makes sense. 😛 But if culture means everything, then culture means nothing. That’s just not helpful!

Instead, let’s disambiguate company culture into two categories. There is the formal culture of the organization (meetings, mission/vision, management, job ladders, hiring practices, strategy, organizational structure, team dynamics, and so on), and there is the informal culture of the people, the ways that humor, playfulness, and practices manifest in groups and individuals.

Organizational culture is professional, formal, structural, institutional. Managerial responsibilities, promotions, compensation plans, and fiduciary duties are just a few of the .many aspects of organizational culture.

Informal culture is chaotic, joyful, free-spirited, and fun, individualized, inherently anarchic and bottoms-up. It’s things like writing release notes in limerick form, bringing banana bread to work after an outage, long pun threads, slack channels dedicated to pets, competing on the number of employees named “Jess”* vs “Chris”*.

Organizational culture is the cake; informal culture is the frosting. Organizational culture is what leaders are hired to build, informal culture is what bubbles up irrepressibly in the gaps. (I wish I had better names for these!) And when it comes to formal, organizational culture, you don’t want to be in the business of innovating.

Culture Serves The Business

As a leader, you should absolutely care about your culture, but your primary responsibility is the health of the business. The purpose of your culture is to make your business succeed. It does not serve you, and it does not serve the people you care about, to be unclear on this front.

I don’t mean to make it sound like this is simple or easy. It is not. You are dealing with people’s lives and livelihoods, and it is all about tradeoffs. What might be best for an individual in the long run (for example, leaving your company to pursue another opportunity) might harm your business in the near term. Yet you might decide to celebrate them in leaving and not pressure them to stay, because you believe that what’s best for your business in the longer term is for employees to be able to trust their managers when they say, “I believe that working here is the best thing you can do for your career right now.”

The transactional nature of work relationships is how they differ from e.g. family relationships. You can form intense bonds and deep friendships with the people you work with — you may even form bonds that transcend your work relationship — but your relationship at work comes with terms and conditions.

Your company culture can’t be everything to everyone. Nor should you try.

You HAVE to care more about the health of the business than about culture for culture’s own sake. Even if — especially if — you have lots of strong opinions about culture, and there are lots of ways you want to deviate from common wisdom. Doing well at business is what earns you more innovation tokens to invest.

“Choose Boring Technology Culture”

Dan McKinley coined the phrase “choose boring technology” and the concept of innovation tokens nearly a decade ago.

“Boring” should not be conflated with “bad.” There is technology out there that is both boring and bad [2]. You should not use any of that. But there are many choices of technology that are boring and good, or at least good enough….The nice thing about boringness (so constrained) is that the capabilities of these things are well understood. But more importantly, their failure modes are well understood. — @mcfunley

The moral of the story is that innovation is costly, so you should choose standard, well-understood, rock-solid technologies insofar as you possibly can. You only get a few innovation tokens to spend, so you should spend them on technologies that can give you a true competitive advantage — not on, like, reinventing memcache for the hell of it.

The same goes for running a business, and the same goes for organizational culture. We have collectively inherited a set of default practices that work pretty well, like the 40 hour work week and having 1x1s with your manager. You CAN choose to do something different, but you should probably have a good reason. To the extent that you can learn from other people’s experience, you probably should, whether in business or in tech; innovation is expensive, and you only get so many tokens. Do you really want to spend one on a radical reinvention of your PTO policy? How does that serve you?

Innovation gets all the headlines, but I would posit that what most companies need is actually much simpler: organizational health.

Great Culture begins with Organizational Health

There’s this book by Patrick Lencione called “The Advantage: How Organizational Health Trumps Everything Else in Business.” (He is best known for writing “Five Dysfunctions of a Team“.) This guy is to organizational health what James Madison was to constitutional government: a very specific kind of genius.

I picked up “The Advantage” in 2020, around the time Honeycomb stopped teetering on the brink of failure, once it became clear we were likely to be around for a while. It made a huge impression on me. He makes the case that most businesses spend a ton of energy on trying to be “smart”, and relatively little on being “healthy”.

Healthy orgs are characterized by minimal politics, minimal confusion, high morale, high productivity, and low turnover. Health begets — and trumps — intelligence.

As Lencione says, an organization that is healthy will inevitably get smarter over time. People in a healthy organization will learn from each other, identify problems, and recover quickly from mistakes. Without politics and confusion, they will cycle through problems and rally much faster than dysfunctional rivals will. And they create an environment in which everyone else can do the same, which creates a multiplier effect.

The healthier an org is, the more of its collective intelligence it is able to tap into and use. Most orgs exploit only a fraction of the knowledge, experience, and intellectual capital available to them, but the healthy ones can tap into almost all of it.

Organizational Health Is Too Boring

No one would disagree with any of this, in principle. ☺️ EVERYBODY wants to work at a place where the mission, vision, and values are clear, meaningful and inspiring; where everyone is rallied around the same winning strategy; and where it’s crystal clear how your role specifically will contribute to that success. Everybody agrees that a healthy organizational culture leads to better outcomes.

So why isn’t every company like that?

Well, it is much easier said than done. ¯\_(ツ)_/¯ It is unglamorous work, difficult to measure, and at the end of the day we are always making risky decisions between conflicting tradeoffs based on partial information. We are imperfect meat sacks who lack self awareness, struggle to understand each other, and get hangry and snap. And the job is never done. You never “get there”, any more than you are ever perfectly healthy with perfect relationships.

But that doesn’t mean we shouldn’t try. We don’t have to be perfect to be a meaningfully better presence in people’s lives. We just have to be healthy enough to achieve our goals.

Nobody Wants An “Exciting” Company Culture

When you tell your partner you had an exciting day at work, do they respond with “uh oh 😬🔥🧯”?

All too often, excitement at work comes from strategic swerves, projects getting canceled, lack of focus, missteps or conflicts, anxiety and passive aggression, outages or downtime, outrageous demands coming from out of left field, or getting information at the last minute that you should have had ages ago. Living on the edge of your seat can be very stimulating! Firefighting is a huge rush, and if you’re part of the essential glue holding this creaky vessel together, you can get hooked on feeling desperately needed.

But this isn’t good for your cortisol levels, and it doesn’t move the company forward. When so much of your energy goes to bailing water and staying afloat, you don’t have much left over for rowing the oars. You want energy going to the oars.

Should work be exciting? ¯\_(ツ)_/¯ It’s not the adjective I would reach for. Emotional rollercoaster rides don’t provide the kind of circumstances that tend to unlock great design or engineering, or collaboration or focus. I would rather reach for words like achievement, fulfillment, pride, comradeship, or the joy of being part of something greater than yourself, not “exciting” or “fun”.

Leaders Worry Too Much About Making Work Fun

As a leader, your job is not to “make work fun”. You are not here to entertain your employees. Your responsibility is to build a formal culture that works, that supports the success of the business.

So what, am I dooming you to a life of bureaucratic beige and meetings without puns? Fuck no.

If formal organizational culture is like the architecture, then informal culture is furnishings, light displays, murals and banners — whatever you do on the inside. You don’t want someone getting overly creative with the load-bearing beams. Save that for when it’s time to paint “Frozen” murals on the walls and hang the matching icicle curtains.

You want formal culture to be boring, stable, reliable, load-bearing…because this creates a safe structure for people to bring the humor, the fun, the joy, the delight, without any fear of building collapse. The company doesn’t have to bring the fun; people bring the fun. Have you met people? People are fucking weirdos. 🥰 If you create an emotionally safe zone and the conditions for success, informal culture will thrive. 🪅People bring the fun🪅 — they always do.

The best informal culture is almost always bottoms up. But managers, execs, HR/People teams, etc can encourage informal culture. One of the most powerful things you can do is just participate. Show up for drinks, play the board games, keep the puns rolling, get silly with your team! Your participation gives people permission and shows that you value their creative cultural labor at work.

Of course there are no bright lines — companies can throw great parties! — but that’s not the job; building a healthy org is the job. Doing that right frees people up to have joy at work. It makes the celebrations that much bigger, the fun that much funnier.

Success Is Rocket Fuel For Fun

Think back to the most corporate fun you’ve ever had at work — the biggest parties, celebrations, blowouts, etc. Were they holiday parties and random occasions, or were they actually linked to great achievements? I bet they were the latter.

You don’t gather at work for the fun of it, you come together to do great things. It stands to reason the peak moments of joy and bonding are fueled by a sense of accomplishment.

Even on a smaller scale, levity and joy are inextricably linked to doing great work and making customers happy. For example, ops/SRE teams are notorious for their gallows humor around outages (ops is ALWAYS the funniest engineering team, in my experience ☺️). But dark humor is only funny when you are also taking your work seriously. Joking about the inevitability of data loss stops being funny real fast if you are actually playing fast and loose with customer backups.

In the absence of success, progress, and high performance, the kind of “frosting” behaviors that bring so much hilarity to work — joking and teasing, puns and stories — actually stop being fun and start making people feel distracted, irritated and on edge. You don’t want to hear a steady stream of jokes from somebody who keeps letting you down.

Side note: unhealthy orgs may have pockets of humor, but it often comes at the expense of other, less prestigious teams. Lots of people may feel too anxious, powerless or threatened to participate. Your experience of whether those companies are “fun” or not is likely to depend heavily on where you sit in the hierarchy. But a healthy org creates the level conditions for humor, playfulness and creativity throughout the org.

Investing Your Innovation Tokens

So yeah. Despite our reputation for cultural innovation, I’d say we’re actually pretty conservative when it comes to operating a company.

Not only are we not revolutionaries, we are actually trying to do as little differently as possible, because innovation is costly!! Instead, we (as a leadership team) are more focused on trying to execute well and improve upon our organizational health. For the past year, we have been laboring especially hard over strategy — the diagnosis, guiding policy, and set of coherent actions we need to win. Our first responsibility is to make the business succeed, after all.

Which brings us back to the topic of innovation tokens.

I started writing down some of the innovation tokens I feel like we’ve spent. But it dawned on me that when I look at most of the cultural experiments we run, and the things we talk about and write about publicly — stuff like the dangers of hierarchy, hiring, interviewing, high-leverage teams, engineering levels, rituals for engineering teams, etc — it doesn’t feel like innovation at all. It’s all just about trying to have a healthier organization. Hierarchy sucks because visible hierarchy has been shown to dampen people’s creativity, motivation and problem-solving skills. Engineering levels are important because they bring clarity. And so on.

What makes something rise to the level of an innovation token is the amount of time we end up asking other people to invest lots of their time into.

Like, we are 1.5 years into a 4 year experiment having an employee on our board of directors. We are about to spin up an internal Advisory Panel to more broadly distribute the impact of our employee board member around the company.
In the past, we have experimented with regular ethics discussion groups.
Last year we did a deep dive into company values with small breakout groups.
Some internal decisions around things like values are handled, not by estaff, but by a group of six people; one employee representative of each org, nominated by their VP; who do a deep dive into the material together and come back with a decision or recommendation.
We are about to start the process of developing our own leadership curriculum. We know that we need to equip our managers with better tools, and culturally indoctrinate new employees, so I am excited to build something with our cultural fingerprints all over it.

We run a lot of experiments around transparency, like, the agenda for exec staff meetings can be viewed by the whole company. After every board meeting, we present the same thing we showed to the board to the whole company during all-hands. We are transparent on salary bands. Stuff like that.

We are far from perfect; we have a long ways to go, and when I look around the org it’s hard not to only see all the work left to be done. But we are a lot healthier and better off than we were a year ago, which was better off than we were two years ago, let alone three.

The Experience Of Making This Will Be With Us Forever

A few months ago I was reading this lengthy profile of Sarah Polley in the New Yorker, as she was doing a bunch of press for her new movie, “Women Talking”. (The movie itself sounds incredibly intense; I am still trying to find time and emotional energy to watch it. Someday!)

One thing she said got lodged in my brain, and I’ve been unable to forget it ever since. She’s talking about the experience of having been a child actor, and how intensely it informs the experience she strives to create for everybody working on the set of one of her movies; where parents get to go home and have dinner with their kids, etc.

[He] told her, “If this film is everything we want it to be, maybe, if we are very lucky, it will affect a few people for a little while, in a way that is out of our control. The only thing that’s certain is that the experience of making it will be with all of us—it will become part of us—forever. So we must try our best to make it a good experience.”

Making a movie that lots of people want to see, one that was a good financial return on investment, buys you the ability to make even more movies, employ more people, take even bigger creative risks. If all you want to do is be a niche indie player, working on a shoestring budget, more power to you. But if you really believe in your ideas, and you want to see them go mainstream … you need mainstream success.

Sarah Polley makes movies. We make developer tools. ☺️ But the same thing is true of working at Honeycomb.

If we are very lucky, and work very hard, our work may help teams build better software and spend fewer, more meaningful hours at work, for a long time to come. I love our mission. But the only certain thing is that the experience of making it will be with all of us, become part of us, forever.

So we should try our best to make it a good experience. ☺️

charity.

Footnotes

(1) Inherited Defaults

How to access these inherited defaults can be a bit more complicated than I make it sound. Working as a manager at Facebook for two years taught me more about these defaults than anything else I’ve ever done in my career. Big companies have had to figure a lot of shit out in order to function at scale, which is why I often advise anyone who plans on starting a company or being a director/VP to do a stint at one. Will Larson’s book “An Elegant Puzzle” does a great job of laying out defaults and best practices for engineering orgs, and his blog has even more useful bits.. Otherwise, you might wanna get yourself an advisor or two with a lot of operator experience, and get used to asking questions like “how does this normally get done?”

(2) Corporate Fun

There’s plenty of stuff in the grey area between formal, organizational culture and informal, individual culture. Companies often stray into fun-like adjacencies like holiday parties, offsites, etc. Fostering a sense of “play” and informality is actually really important for making teams click with each other, and obviously the company should foot the bill if it’s a work function. Just be mindful of what you’re doing and what your goals are when you veer into the rocky shoals of Forced Corporate Fun. 😆

Questionable Advice: “People Used To Take Me Seriously. Then I Became A Software Vendor”

March 29, 2023July 14, 2023 mipsytipsyadvice, culture, startups, vendorsLeave a comment

I recently got a plaintive text message from my magnificent friend Abby Bangser, asking about a conversation we had several years ago:

“Hey, I’ve got a question for you. A long time ago I remember you talking about what an adjustment it was becoming a vendor, how all of a sudden people would just discard your opinion and your expertise without even listening. And that it was SUPER ANNOYING.

I’m now experiencing something similar. Did you ever find any good reading/listening/watching to help you adjust to being on the vendor side without being either a terrible human or constantly disregarded?”

Oh my.. This brings back memories. ☺️🙈

Like Abby, I’ve spent most of my career as an engineer in the trenches. I have also spent a lot of time cheerfully talking smack about software. I’ve never really had anyone question my experience[1] or my authority as an expert, hardened as I was in the flames of a thousand tire fires.

Then I started a software company. And all of a sudden this bullshit starts popping up. Someone brushing me off because I was “selling something”, or dismissing my work like I was fatally compromised. I shrugged it off, but if I stopped to think, it really bothered me. Sometimes I felt like yelling “HEY FUCKERS, I am one of your kind! I’m trying to HELP YOU. Stop making this so hard!” 😡 (And sometimes I actually did yell, lol.)

That’s what I remember complaining to Abby about, five or six years ago. It was all very fresh and raw at the time.

We’ll get to that. First let’s dial the clock back a few more years, so you can fully appreciate the rich irony of my situation. (Or skip the story and jump straight to “Five easy ways to make yourself a vendor worth listening to“.)

The first time I encountered “software for sale”

My earliest interaction with software vendors was at Linden Lab. Like most infrastructure teams, most of the software we used was open source. But somewhere around 2009? 2010? Linden’s data engineering team began auditioning vendors like Splunk, Greenplum, Vertica[2], etc for our data warehouse solution, and I tagged along as the sysinfra/ops delegate.

For two full days we sat around this enormous table as vendor after vendor came by to demo and plump their wares, then opened the floor for questions.

One of the very first sales guys did something that pissed me off me. I don’t remember exactly what happened — maybe he was ignoring my questions or talking down to me. (I’m certain I didn’t come across like a seasoned engineering professional; in my mid twenties, face buried in my laptop, probably wearing pajamas and/or pigtails.) But I do remember becoming very irritated, then settling in to a stance of, shall we say, oppositional defiance.

I peppered every sales team aggressively with questions about the operational burden of running their software, their architectural decisions, and how canned or cherry-picked their demos were. Any time they let slip a sign of weakness or betrayed uncertainty, I bore down harder and twisted the knife. I was a ✨royal asshole✨. My coworkers on the data team found this extremely entertaining, which only egged me on.

What the fuck?? 🫢😧🫠 I’m not usually an asshole to strangers.. where did that come from?

What open source culture taught me about sales

I came from open source, where contempt for software vendors was apparently de rigueur. (is it still this way?? seems like it might have gotten better? 😦) It is fascinating now to look back and realize how much attitude I soaked up before coming face to face with my first software vendor. According to my worldview at the time,

Vendors are liars
They will say anything to get you to buy
Open source software is always the safest and best code
Software written for profit is inherently inferior, and will ultimately be replaced by the inevitable rise of better, faster, more democratic open source solutions
Sales exists to create needs that ought never to have existed, then take you to the cleaners
Engineers who go work for software vendors have either sold out, or they aren’t good enough to hack it writing real (consumer facing) software.

I’m remembering Richard Stallman trailing around behind me, up and down the rows of vendor booths at USENIX in his St IGNUcious robes, silver disk platter halo atop his head, offering (begging?) to lay his hands on my laptop and bless it, to “free it from the demons of proprietary software.” Huh. (Remember THIS song? 🎶 😱)

Given all that, it’s not hugely surprising that my first encounter with software vendors devolved into hostile questioning.

(It’s fun to speculate on the origin of some of these beliefs. Like, I bet 3) and 4) came from working on databases, particularly Oracle and MySQL/Postgres. As for 5) that sounds an awful lot like the beauty industry and other products sold to women. 🤭)

Behind every software vendor lies a recovering open source zealot(???)

I’ve had many, many experiences since then that slowly helped me dismantle this worldview, brick by brick. Working at Facebook made me realize that open source successes like Apache, Haproxy, Nginx etc are exceptions, not the norm; that this model is only viable for certain types of general-purpose infrastructure software; that governance and roadmaps are a huge issue for open source projects too; and that if steady progress is being made, at the end of the day, somewhere somebody is probably paying those developers.

I learned that the overwhelming majority of production-caliber code is written by somebody who was paid to write it — not by volunteers. I learned about coordination costs and overhead, how expensive it is to organize an army of volunteers, and the pains of decentralized quality control. I learned that you really really want the person who wrote the code to stick around and own it for a long time, and not just on alternate weekends when they don’t have the kids (and/or they happen to feel like it).

I learned about game theory, and I learned that sales is about relationships. Yes, there are unscrupulous sellers out there, just like there are shady developers, but good sales people don’t want you to walk away feeling tricked or disappointed any more than you want to be tricked or disappointed. They want to exceed your expectations and deliver more value than expected, so you’ll keep coming back. In game theory terms, it’s a “repeated game”.

I learned SO MUCH from interviewing sales candidates at Honeycomb.[3] Early on, when nobody knew who we were, I began to notice how much our sales candidates were obsessed with value. They were constantly trying to puzzle out out how much value Honeycomb actually brought to the companies we were selling to. I was not used to talking or thinking about software in terms of “value”, and initially I found this incredibly offputting (can you believe it?? 😳).

Sell unto others as you would have them sell unto you

Ultimately, this was the biggest (if dumbest) lesson of all: I learned that good software has tremendous value. It unlocks value and creates value, it pays enormous ongoing dividends in dollars and productivity, and the people who build it, support it, and bring it to market fully deserve to recoup a slice of the value they created for others.

There was a time when I would have bristled indignantly and said, “we didn’t start honeycomb to make money!” I would have said that the reason we built honeycomb because we knew as engineers what a radical shift it had wrought in how we built and understood software, and we didn’t want to live without it, ever again.

But that’s not quite true. Right from the start, Christine and I were intent on building not just great software, but a great software business. It wasn’t personal wealth we were chasing, it was independence and autonomy — the freedom to build and run a company the way we thought it should be run, building software to radically empower other engineers like ourselves.

Guess what you have to do if you care about freedom and autonomy?

Make money. 🙄☺️

I also realized, belatedly, that most people who start software companies do so for the same damn reasons Christine and I did… to solve hard problems, share solutions, and help other engineers like ourselves. If all you want to do is get rich, this is actually a pretty stupid way to do that. Over 90% of startups fail, and even the so-called “success stories” aren’t as predictably lucrative as RSUs. And then there’s the wear and tear on relationships, the loss of social life, the vicissitudes of the financial system, the ever-looming spectre of failure … 👻☠️🪦 Startups are brutal, my friend.

Karma is a bitch

None of these are particularly novel insights, but there was a time when they were definitely news to me. ☺️ It was a pretty big shock to my system when I first became a software vendor and found myself sitting on the other side of the table, the freshly minted target of hostile questioning.

These days I am far less likely to be cited as an objective expert than I used to be. I see people on Hacker News dismissing me with the same scornful wave of the hand as I used to dismiss other vendors. Karma’s a bitch, as they say. What goes around comes around. 🥰

I used to get very bent out of shape by this. “You act like I only care because I’m trying to sell you something,” I would hotly protest, “but it’s exactly the opposite. I built something because I cared.” That may be true, but it doesn’t change the fact that vested interests can create blind spots, ones I might not even be aware of.

And that’s ok! My arguments/my solutions should be sturdy enough to withstand any disclosure of personal interest. ☺️

Some people are jerks; I can’t control that. But there are a few things I can do to acknowledge my biases up front, play fair, and just generally be the kind of vendor that I personally would be happy to work with.

Five easy ways to make yourself a vendor worth listening to

So I gave Abby a short list of a few things I do to try and signal that I am a trustworthy voice, a vendor worth listening to. (What do you think, did I miss anything?)

🌸 Lead with your bias.🌸
I always try to disclose my own vested interest up front, and sometimes I exaggerate for effect: “As a vendor, I’m contractually obligated to say this”, or “Take it for what you will, obviously I have religious convictions here”. Everyone has biases; I prefer to talk to people who are aware of theirs.

🌸 Avoid cheap shots.🌸
Try to engage with the most powerful arguments for your competitors’ solutions. Don’t waste your time against straw men or slam dunks; go up against whatever ideal scenarios or “steel man” arguments they would muster in their own favor. Comparing your strengths vs their strengths results in a way more interesting, relevant and USEFUL discussion for all involved.

🌸 Be your own biggest critic.🌸
Be forthcoming about the flaws of your own solution. People love it when you are unafraid to list your own product’s shortcomings or where the competition shines, or describe the scenarios where other tools are genuinely superior or more cost-effective. It makes you look strong and confident, not weak.

What would you say about your own product as an engineer, or a customer? Say that.

🌸 You can still talk shit about software, just not your competitors‘ software. 🌸
I try not to gratuitously snipe at our competitors. It’s fine to speak at length about technical problems, differentiation and tradeoffs, and to address how specifically your product compares with theirs. But confine your shit talking to categories of software where you don’t have a personal conflict of interest.

Like, I’m not going to get on twitter and take a swipe at a monitoring vendor (anymore 😇), but I might say rude things about a language, a framework, or a database I have no stake in, if I’m feeling punchy. ☺️ (This particular gem of advice comes by way of Adam Jacob.)

🌸 Be generous with your expertise.🌸
If you have spent years going deep on one gnarly problem, you might very well know that problem and its solution space more thoroughly than almost anyone else in the world. Do you know how many people you can help with that kind of mastery?! A few minutes from you could potentially spare someone days or weeks of floundering. This is a gift few can give.

It feels good, and it’s a nice break from battering your head against unsolvable problems. Don’t restrict your help to paying customers, and, obviously, don’t give self-serving advice. Maybe they can’t buy/don’t need your solution today, but maybe someday they will.

In conclusion

There’s a time and place for being oppositional. Sometimes a vendor gets all high on their own supply, or starts making claims that aren’t just an “optimistic” spin on the facts but are provably untrue. If any vendor is operating in poor faith they deserve to to be corrected.

But it’s a shitty, self-limiting stance to take as a default. We are all here to build things, not tear things down. No one builds software alone. The code you write that defines your business is just the wee tippy top of a colossal iceberg of code written by other people — device drivers, libraries, databases, graphics cards, routers, emacs. All of this value was created by other people, yet we collectively benefit.

Think of how many gazillion lines of code are required for you to run just one AWS Lambda function! Think of how much cooperation and trust that represents. And think of all the deals that brokered that trust and established that value, compensating the makers and allowing them to keep building and improving the software we all rely on.

We build software together. Vendors exist to help you. We do what we do best, so you can spend your engineering cycles doing what you do best, working on your core product. Good sales deals don’t leave anyone feeling robbed or cheated, they leave both sides feeling happy and excited to collaborate.[4]

🐝💜Charity.

[1] Yes, I know this experience is far from universal; LOTS of people in tech have not felt like their voices are heard or their expertise acknowledged. This happens disproportionately to women and other under-represented groups, but it also happens to plenty of members of the dominant groups. It’s just a really common thing! However that has not really been my experience — or if it has, I haven’t noticed — nor Abby’s, as far as I’m aware.

[2] My first brush with columnar storage systems! Which is what makes Honeycomb possible today.

[3] I have learned SO MUCH from watching the world class sales professionals we have at Honeycomb. Sales is a tough gig, and doing it well involves many disciplines — empathy, creativity, business acumen, technical expertise, and so much more. Selling to software engineers in particular means you are often dealing with cocky little shits who think they could do your job with a few lines of code. On behalf of my fellow ~~little shits~~ engineers, I am sorry. 🙈

[4] Like our sales team says: “Never do a deal unless you’d do both sides of the deal.” I fucking love that.