Outsource Your O11y: Get Aligned With Security (part 2/3)

This is part two of a three-part series of guest posts:

  1. How To Be A Champion, on how to choose a third-party vendor and champion them successfully to your security team.  (George Chamales)
  2. Get Aligned With Security, how to work with your security team to find the best possible outcome for all sides (Lilly Ryan)
  3. Now Roll It Out And Keep Them Happy, on how to operationalize your service by rolling out the integration and maintaining it — and the relationship with your security team — over the long run (Andy Isaacson)

All this pain will someday be worth it.  🙏❤️  charity + friends


“Get Aligned With Security”

by Lilly Ryan

If your team has decided on a third-party service to help you gather data and debug product issues, how do you convince an often overeager internal security team to help you adopt it?

When this service is something that provides a pathway for developers to access production data, as analytics tools often do, making the case for access to that data can screech to a halt at the mention of the word “production”. Progressing past that point will take time, empathy, and consideration.

I have been on both sides of the “adopting a new service” fence: as a developer hoping to introduce something new and useful to our stack, and now as a security professional who spends her days trying to bust holes in other people’s setups. I understand both sides of the sometimes-conflicting needs to both ship software and to keep systems safe.  

This guide has advice to help you solve the immediate problem of choosing and deploying a third-party service with the approval of your security team.  But it also has advice for how to strengthen the working relationship between your security and development teams over the longer term. No two companies are the same, so please adapt these ideas to fit your circumstances.

Understanding the security mindset

The biggest problems in technology are never really about technology, but about people. Seeing your security team as people and understanding where they are coming from will help you to establish empathy with them so that both of you want to help each other get what you want, not block each other.

First, understand where your security team is coming from. Development teams need to build features, improve the product, understand and ship good code. Security teams need to make sure you don’t end up on the cover of the NYT for data breaches, that your business isn’t halted by ransomware, and that you’re not building your product on a vulnerable stack.

This can be an unfamiliar frame of mind for developers.  Software development tends to attract positive-minded people who love creating things and are excited about the possibilities of new technology. Software security tends to attract negative thinkers who are skilled at finding all the flaws in a system.  These are very different mentalities, and the people who occupy them tend to have very different assumptions, vocabularies, and worldviews.   

But if you and your security team can’t share the same worldview, it will be hard to trust each other and come to agreement.  This is where practicing empathy can be helpful.

Before approaching your security team with your request to approve a new vendor, you may want to run some practice exercises for putting yourselves in their shoes and forcing yourselves to deliberately cultivate a negative thinking mindset to experience how they may react — not just in terms of the objective risk to the business, or the compliance headaches it might cause, but also what arguments might resonate with them and what emotional reactions they might have.

My favourite exercise for getting teams to think negatively is what I call the Land Astronaut approach.

The “Land Astronaut” Game

Imagine you are an astronaut on the International Space Station. Literally everything you do in space has death as a highly possible outcome. So astronauts spend a lot of time analysing, re-enacting, and optimizing their reactions to events, until it becomes muscle memory. By expecting and training for failure, astronauts use negative thinking to anticipate and mitigate flaws before they happen. It makes their chances of survival greater and their people ready for any crisis.

Your project may not be as high-stakes as a space mission, and your feet will most likely remain on the ground for the duration of your work, but you can bet your security team is regularly indulging in worst-case astronaut-type thinking. You and your team should try it, too.

The Game:

Pick a service for you and your team to game out.  Schedule an hour, book a room with a whiteboard, put on your Land Astronaut helmets.  Then tell your team to spend half an hour brainstorming about all the terrible things that can happen to that service, or to the rest of your stack when that service is introduced.  Negative thoughts only!

Start brainstorming together. Start out by being as outlandish as possible (what happens if their data centre is suddenly overrun by a stampede of elephants?). Eventually you will find that you’ll tire of the extreme worst case scenarios and come to consider more realistic outcomes — some of which which you may not have thought of outside of the structure of the activity.

After half an hour, or whenever you feel like you’re all done brainstorming, take off your Land Astronaut helmets, sift out the most plausible of the worst case scenarios, and try to come up with answers or strategies that will help you counteract them.  Which risks are plausible enough that you should mitigate them?  Which are you prepared to gamble on never happening?  How will this risk calculus change as your company grows and takes on more exposure?

Doing this with your team will allow you all to practice the negative thinking mindset together and get a feel for how your colleagues in the security team might approach this request. (While this may seem similar to threat modelling exercises you might have done in the past, the focus here is on learning to adopt a security mindset and gaining empathy for this thought process, rather than running through a technical checklist of common areas of concern.)

While you still have your helmets within reach, use your negative thinking mindset to fill out the spreadsheet from the first piece in this series.  This will help you anticipate most of the reasonable objections security might raise, and may help you include useful detail the security team might not have known to ask for.

Once you have prepared your list of answers to George’s worksheet and held a team Land Astronaut session together, you will have come most of the way to getting on board with the way your security team thinks.

Preparing for compromise

You’ve considered your options carefully, you’ve learned how to harness negative thinking to your advantage, and you’re ready to talk to your colleagues in security – but sometimes, even with all of these tools at your disposal, you may not walk away with all of the things you are hoping for.

Being willing to compromise and anticipating some of those compromises before you approach the security team will help you negotiate more successfully.

While your Land Astronaut helmets are still within reach, consider using your negative thinking mindset game to identify areas where you may be asked to compromise. If you’re asking for production access to this new service for observability and debugging purposes, think about what kinds of objections may be raised about this and how you might counter them or accommodate them. Consider continuing the activity with half of the team remaining in the Land Astronaut role while the other half advocates from a positive thinking standpoint. This dynamic will get you having conversations about compromise early on, so that when the security team inevitably raises eyebrows, you are ready with answers.

Be prepared to consider compromises you had not anticipated, and enter into discussions with the security team with as open a mind as possible. Remember the team is balancing priorities of not only your team, but other business and development teams as well.  If you and your security colleagues are doing the hard work to meet each other halfway then you are more likely to arrive at a solution that satisfies both parties.

Working together for the long term

While the previous strategies we’ve covered focus on short-term outcomes, in this continuous-deployment, shift-left world we now live in, the best way to convince your security team of the benefits of a third-party service – or any other decision – is to have them along from day one, as part of the team.

Roles and teams are increasingly fluid and boundary-crossing, yet security remains one of the roles least likely to be considered for inclusion on a software development team. Even in 2019, the task of ensuring that your product and stack are secure and well-defended is often left until the end of the development cycle.  This contributes a great deal to the combative atmosphere that is common.

Bringing security people into the development process much earlier builds rapport and prevents these adversarial, territorial dynamics. Consider working together to build Disaster Recovery plans and coordinating for shared production ownership.

If your organisation isn’t ready for that kind of structural shift, there are other ways to work together more closely with your security colleagues.

Try having members of your team spend a week or two embedded with the security team. You may even consider a rolling exchange – a developer for a security team member – so that developers build the security mindset, and the security team is able to understand the problems your team is facing (and why you are looking at introducing this new service).

At the very least, you should make regular time to meet with the security team, get to know them as people, and avoid springing things on them late in the project when change is hardest.

Riding off together into the sunset…?

If you’ve taken the time to get to know your security team and how they think, you’ll hopefully be able to get what you want from them – or perhaps you’ll understand why their objections were valid, and come up with a better solution that works well for both of you.

Investing in a strong relationship between your development and security teams will rarely lead to the apocalypse. Instead, you’ll end up with a better product, probably some new work friends, and maybe an exciting idea for a boundary-crossing new career in tech.

But this story isn’t over! Once you get the green light from security, you’ll need to think about how to roll your new service out safely, maintain it, and consider its full lifespan within your company.  Which leads us to part three of this series, on rolling it out and maintaining it … both your integration and your relationship with the security team.

 

Lilly Ryan is a pen tester, Python wrangler, and recovering historian from Melbourne. She writes and speaks internationally about ethical software, social identities after death, teamwork, and the telegraph. More recently she has researched the domestic use of arsenic in Victorian England, attempted urban camouflage, reverse engineered APIs, wielded the Oxford comma, and baked a really good lemon shortbread.

Outsource Your O11y: Get Aligned With Security (part 2/3)

Software Sprawl, The Golden Path, and Scaling Teams With Agency

gplanis

Stop me if you’ve heard this one before.

The company is growing like crazy, your engineering team keeps rising to the challenge, and you are ferociously proud of them.  But some cracks are beginning to show, and frankly you’re a little worried.  You have always advocated for engineers to have broad latitude in technical decisions, including choosing languages and tools.  This autonomy and culture of ownership is part of how you have successfully hired and retained top talent despite the siren song of the Faceboogles.

But recently you saw something terrifying that you cannot unsee: your company is using all the languages, all the environments, all the databases, all the build tools.  Shit!!!  Your ops team is in full revolt and you can’t really blame them.  It’s grown into an unsupportable nightmare and something MUST be done, but you don’t know what or how — let alone how to solve it while retaining the autonomy and personal agency that you all value so highly.

I hear a version of this everywhere I’ve gone for the past year or two.  It’s crazy how often.  I’ve been meaning to write my answer up for ages, and here it (finally) is.

First of all: you aren’t alone.  This is extremely common among high-performing teams, so congratulations.  Really!

There actually seems to be a direct link between teams that give engineers lots of leeway to own their technical decisions and that team’s ability to hire and retain top-tier talent, particularly senior talent.   Everything is a tradeoff, obviously, but accepting somewhat more chaos in exchange for a stronger sense of individual ownership is usually the right one, and leads to higher-performing teams in the long run.

Second, there is actually already a well-trod path out of this hole to a better place, and it doesn’t involve sacrificing developer agency.  It’s fairly simple!  Just five short steps, which I will describe to you now.

How to build a golden path and reverse software sprawl

  1. Assemble a small council of trusted senior engineers.
  2. Task them with creating a recommended list of default components for developers to use when building out new services.  This will be your Golden Path, the path of convergence (and the path of least resistance).
  3. Tell all your engineers that going forward, the Golden Path will be fully supported by the org.  Upgrades, patches, security fixes; backups, monitoring, build pipeline; deploy tooling, artifact versioning, development environment, even tier 1 on call support.  Pave the path with gold.  Nobody HAS to use these components … but if they don’t, they’re on their own.  They will have to support it themselves.
  4. Work with team leads to draw up an umbrella plan for adopting the Golden Path for their current projects as well as older production services, as much as is reasonable or possible or desirable.  Come up with a timeline for the whole eng org to deprecate as many other tools as possible.  Allocate real engineering time to the effort.  Hell, make a party out of it!
  5. After the cutoff date (and once things have stabilized), establish a regular process for reviewing and incorporating feedback about the blessed Path and considering any proposed changes, additions or removals.

There you go.  That’s it.  Easy, right??

(It’s not easy.  I never said it was easy, I said it was simple.  👼🏼)

Your engineers are currently used to picking the best tool for the job by optimizing locally. gpjon What data store has a data model that is easiest for them to fit to their needs?  Which language is fastest for I/O throughput?  What are they already proficient in?  What you need to do is start building your muscles for optimizing globally.  Not in isolation of other considerations, but in conjunction with them.  It will always be a balancing act between optimizing locally for the problem at hand and optimizing globally for operability and general sanity.

(Oh, incidentally, requiring an engineer to write up a proposal any time they want to use a non-standard component, and then defend their case while the council grills them in person — this will be nothing but good for them, guaran-fucking-teed.)

Let’s go into a bit more detail on each of the five points.  But quick disclaimer: this is not a prescription.  I don’t know your system, your team, your cultural land mines or technical interdependencies or anything else about your situation.  I am just telling stories here.

1. Assemble your council

Three is a good number for a council.  More than that gets unwieldy, and may have trouble reaching consensus.  Less than three and you run into SPOFs.  You never want to have a single person making unilateral decisions because a) the decision-making process will be weaker, b) it sets that person up for too much interpersonal friction, and c) it denies your other engineers the opportunity to practice making these kinds of decisions.

  • Your council members need technical breadth more than depth, and should be widely respected by engineers.
  • At least one member should have a long history with the company so they know lots of stupid little details about what’s been tried before and why it failed.
  • At least one member should be deeply versed in practical data and operability concerns.
  • They should all have enough patience and political skill to drive consensus for their decisions.  Absolutely no bombthrowers.

If you’re super lucky, you just tap the three senior technologists who immediately come to mind … your mind and everyone else’s.  If you don’t have this kind of automatic consensus, you may want to let teams or orgs nominate their own representative so they feel they have some say.

 

2.  Task the council with defining a Golden Path

Your council cannot vanish for a week and then descend from the mountain lugging lists engraved on stone tablets.  The process of discovery and consensus is what validates the result.

The process must include talking to and gathering feedback from your engineers, talking to experts outside the company, talking to teams at other companies who are farther along using that technology, coming up with detailed pro/con lists and reasons for their choices.  Maybe sometimes it includes prototyping something or investigating the technical depths … but yeah no mostly it’s just the talking.

You need your council members to have enough political skill to handle these conversations deftly, building support and driving consensus through the process.  Everybody doesn’t have to love the outcome, but it shouldn’t be a *surprise* to anyone by the end.

3.  Know where you’re going

Your council should create a detailed written plan describing which technologies are going to be supported … and a stab at what “supported” means.  (Ask the experts in each component what the best practices are for backups, versioning, dependency management, etc.)

You might start with something like this:

* Backend lang: Go 1.11           ## we will no longer be supporting
backend scripting languages
* Frontend lang: ReactJS v 16.5
* Primary db: Aurora v 2.0        ## Yes, we know postgres is "better", 
but we have many mysql experts and 0 pg experts except the one guy 
who is going to complain about this.  You know who you are.
* Deploy pipeline: github -> jenkins + docker -> S3 -> custom k8s 
deploy tooling
* Message broker: kafka v 2.10, confluent build
* Mail: SES
* .... etc

Circulate the draft regularly for feedback, especially with eng managers.  Some team reorganization will probably be necessary to bear the new weight of your support specifications, and managers will need some lead time to wrangle this.

This is also a great time to reconceive of the way on call works at your company.  But I am not going to go into all that here.

4. Set a date, draft a plan: go!

Get approval from leadership to devote a certain amount of time to consolidating your stack and paying down a lump sum of tech debt.  It depends on your stage of decay, but a reasonable amount of time might be “25% of engineering time for three months“.  Whatever you agree to, make sure it’s enough to make the world demonstrably better for the humans who run it; you don’t want to leave them with a tire fire or you’ll blow your credibility.

The council and team leads should come up with a rough outer estimate for how long itgpjava would take to rewrite everything and move the whole stack on to the Golden Stack.  (It’s probably impossible and/or would take years, but that’s okay.)  Next, look for the quick wins or swollen, inflamed pain points.

  • If you are running two pieces of functionally similar software, like postgres and mysql, can you eliminate one?
  • If you are managing something yourself that AWS could manage for you (e.g. postfix instead of SES, or kafka instead of kinesis), can you migrate that?
  • If you are managing anything yourself that is not core to your business value, in fact, you should try to not manage it.
  • If you are running any services by hand on an AWS instance somewhere, could you try using a service?
  • If you are running your own monitoring software, etc … can you not?
  • If you have multiple versions of a piece of software, can you upgrade or consolidate on one version?

The hardest parts are always going to be the ones around migrating data or rewriting components.  Not everything is worth doing or can afford to be done in the time span of your project time, and that’s okay.

Next, brainstorm up some carrots.  Can you write templates so that anybody who writes a service using your approved library, magically gets monitoring checks without having to configure anything?  Can you write a wrapper so they get a bunch of end-to-end tests for free?  Anything you can do to delight people or save them time and effort by using your preferred components is worth considering.

(By the way, if you don’t have any engineers devoted to internal tooling, you’re probably way overdue at this point.)

Pay down as much debt as you can, but be pragmatic: it’s better to get rid of five small things than one large thing, from a support perspective.  Your main goal is to shrink the number of types of software your team has to support, particularly databases.

Do look for ways to make it fun, like … running a competition to see who can move the most tools to AWS in a week, or throwing a hack week party, or giving dorky prizes like trophies that entitle you to put your manager on call instead of you for a day, etc.

gpcersei

5. Make the process sustainable

After your target date has come and gone, you probably want to hold a post mortem retrospective and do lots of listening.  (Well — first might I recommend a bubble bath and a bottle of champagne?  But then a post mortem.)

Nothing is ever fixed forever.  The company’s needs are going to expand and contract, gpdiedand people will come and go, because change is the only constant.  So you need to bake some flex into your system.  How are you going to handle the need for changes to the Golden Path?  Monthly discussions?  An email list?  Quarterly meetings with a formal agenda?  I’ve seen people do all of these and more, it doesn’t really matter afaict.

Nobody likes a cabal, though, so the original council should gradually rotate out.  I recommend replacing one person at a time, one per quarter, and rotating in another senior engineer in their place.  This provides continuity while giving others a chance to learn these technical and political skills.

In the end, engineers are still free to use any tool or component at any time, just like before, only now they are solely responsible for it, which puts pressure on them not to do it unless REALLY necessary.  So if someone wants to propose adding a new tool to the default golden path, they can always add it themselves and gain some experience in it before bringing it to the council to discuss a formal place for it.

That’s all folks

See, wasn’t that simple?

(It’s never simple.)

I dearly wish more people would write up their experiences with this sort of thing in detail.  I think engineering teams are too reluctant to show their warts and struggles to the world — or maybe it’s their executives who are afraid?  Dunno.

Regardless, I think it’s actually a highly effective recruiting tool when teams aren’t afraid to share their struggles.  The companies that brag about how awesome they are are the ones who come off looking weak and fragile.  Whereas you can always trust the ones who are willing to laugh about all the ways they screwed up.  Right?

In conclusion, don’t feel like an asshole for insisting on some process here.  There should be friction around adding new components to your stack.  (Add in haste, repent at leisure, as they say.)  Anybody who argues with you probably needs to be exposed to way, way more of the support load for that software.  That’s my professional opinion.

Anyway.  You win or you die.  Good luck with your sprawl.

charity

 

 

Software Sprawl, The Golden Path, and Scaling Teams With Agency