security

This is part three of a three-part series of guest posts:

How To Be A Champion, on how to choose a third-party vendor and champion them successfully to your security team. (George Chamales)
Get Aligned With Security, how to work with your security team to find the best possible outcome for all sides (Lilly Ryan)
Now Roll It Out And Keep Them Happy, on how to operationalize your service by rolling out the integration and maintaining it — and the relationship with your security team — over the long run (Andy Isaacson)

All this pain will someday be worth it. 🙏❤️ charity + friends

“Now Roll It Out And Keep Them Happy”

This is the third in a series of blog posts; previously we analyzed the security challenges of using a third party service, and we worked together with the security team to build empathy to deliver the project. You might want to read those first, since we are going to build on a lot of the ideas there to ship and maintain this integration.

Ready for launch

You’ve convinced the security team and other stakeholders, you’ve gotten the integration running, you’re getting promising results from dev-test or staging environments… now it’s time to move from proof-of-concept to full implementation. Depending on your situation this might be a transition from staging to production, or it might mean increasing a feature flipper flag from 5% to 100%, or it might mean increasing coverage of an integration from one API endpoint to cover your entire developer footprint.

Taking into account Murphy’s Law, we expect that some things will go wrong during the rollout. Perhaps during coverage, a developer realizes that the schema designed to handle the app’s event mechanism can’t represent a scenario, requiring a redesign or a hacky solution. Or perhaps the metrics dashboard shows elevated error rates from the API frontend, and while there’s no smoking gun, the ops oncall decides to rollback the integration Just In Case it’s causing the incident.

This gives us another chance to practice empathy — while it’s easy, wearing the champion hat, to dismiss any issues found by looking for someone to blame, ultimately this poisons trust within your organization and will hamper success. It’s more effective, in the long run (and often even in the short run), to find common ground with your peers in other disciplines and teams, and work through to solutions that satisfy everybody.

Keeping the lights on

In all likelihood as integration succeeds, the team will rapidly develop experts and expertise, as well as idiomatic ways to use the product. Let the experts surprise you; folks you might not expect can step up when given a chance. Expertise flourishes when given guidance and goals; as the team becomes comfortable with the integration, explicitly recognize a leader or point person for each vendor relationship. Having one person explicitly responsible for a relationship lets them pay attention to those vendor emails, updates, and avoid the tragedy of the “but I thought *you* were” commons. This Integration Lead is also a center of knowledge transfer for your organization — they won’t know everything or help every user come up to speed, but they can help empower the local power users in each team to ramp up their teams on the integration.

As comfort grows you will start to consider ways to change your usage, for example growing into new kinds of data. This is a good time to revisit that security checklist — does the change increase PII exposure to your vendor? Would the new data lead to additional requirements such as per-field encryption? Don’t let these security concerns block you from gaining valuable insight using the new tool, but do take the chance to talk it over with your security experts as appropriate.

Throughout this organic growth, the Integration Lead remains core to managing your changing profile of usage of the vendor they shepherd; as new categories of data are added to the integration, the Lead has responsibility to ensure that the vendor relationship and risk profile are well matched to the needs that the new usage (and presumably, business value) is placing on the relationship.

Documenting the Intergation Lead role and responsibilities is critical. The team should know when to check in, and writing it down helps it happen. When new code has a security implication, or a new use case potentially amplifies the cost of an integration, bringing the domain expert in will avoid unhappy surprises. Knowing how to find out who to bring in, and when to bring them in, will keep your team getting the right eyes on their changes.

Security threats and other challenges change over time, too. Collaborating with your security team so that they know what systems are in use helps your team take note of new information that is relevant to your business. A simple example is noting when your vendors publish a breach announcement, but more complex examples happen too — your vendor transitions cloud providers from AWS to Azure and the security team gets an alert about unexpected data flows from your production cluster; with transparency and trust such events become part of a routine process rather than an emergency.

It’s all operational

Monitoring and alerting is a fact of operations life, and this has to include vendor integrations (even when the vendor integration is a monitoring product.) All of your operations best practices are needed here — keep your alerts clean and actionable so that you don’t develop pager fatigue, and monitor performance of the integration so that you don’t get blindsided by a creeping latency monster in your APIs.

Authentication and authorization are changing as the threat landscape evolves and industry moves from SMS verification codes to U2F/WebAuthn. Does your vendor support your SSO integration? If they can’t support the same SSO that you use everywhere else and can’t add it — or worse, look confused when you mention SSO — that’s probably a sign you should consider a different vendor.

A beautiful sunset

Have a plan beforehand for what needs to be done should you stop using the service. Got any mobile apps that depend on APIs that will go away or start returning permission errors? Be sure to test these scenarios ahead of time.

What happens at contract termination to data stored on the service? Do you need to explicitly delete data when ceasing use?

Do you need to remove integrations from your systems before ending the commercial relationship, or can the technical shutdown and business shutdown run in parallel?

In all likelihood these are contingency plans that will never be needed, and they don’t need to be fully fleshed out to start, but a little bit of forethought can avoid unpleasant surprises.

Year after year

Industry best practice and common sense dictate that you should revisit the security questionnaire annually (if not more frequently). Use this chance to take stock of the last year and check in — are you getting value from the service? What has changed in your business needs and the competitive landscape?

It’s entirely possible that a new year brings new challenges, which could make your current vendor even more valuable (time to negotiate a better contract rate!) or could mean you’d do better with a competing service. Has the vendor gone through any major changes? They might have new offerings that suit your needs well, or they may have pivoted away from the features you need.

Check in with your friends on the security team as well; standards evolve, and last year’s sufficient solution might not be good enough for new requirements.

Andy thinks out loud about security, society, and the problems with computers on Twitter.

❤️ Thanks so much reading, folks. Please feel free to drop any complaints, comments, or additional tips to us in the comments, or direct them to me on twitter.

Have fun! Stay (a little bit) Paranoid!!

— charity

I hear variations on this question constantly: “I’d really like to use a service like Honeycomb for my observability, but I’m told I can’t ship any data off site. Do you have any advice on how to convince my security team to let me?”

I’ve given lots of answers, most of them unsatisfactory. “Strip the PII/PHI from your operational data.” “Validate server side.” “Use our secure tenancy proxy.” (I’m not bad at security from a technical perspective, but I am not fluent with the local lingo, and I’ve never actually worked with an in-house security team — i’ve always *been* the security team, de facto as it may be.)

So I’ve invited three experts to share their wisdom in a three-part series of guest posts:

How To Be A Champion, on how to choose a third-party vendor and champion them successfully to your security team. (George Chamales)
Get Aligned With Security, how to work with your security team to find the best possible outcome for all sides (Lilly Ryan)
Now Roll It Out And Keep Them Happy, on how to operationalize your service by rolling out the integration and maintaining it — and the relationship with your security team — over the long run (Andy Isaacson)

My ✨first-ever guest posts✨! Yippee. I hope these are useful to you, wherever you are in the process of outsourcing your tools. You are on the right path: outsourcing your observability to a vendor for whom it’s their One Job is almost always the right call, in terms of money and time and focus — and yes, even security.

All this pain will someday be worth it. 🙏❤️ charity + friends

“How to be a Champion”

by George Chamales

You’ve found a third party service you want to bring into your company, hooray!

To you, it’s an opportunity to deploy new features in a flash, juice your team’s productivity, and save boatloads of money.

To your security and compliance teams, it’s a chance to lose your customers’ data, cause your applications to fall over, and do inordinate damage to your company’s reputation and bottom line.

The good news is, you’re absolutely right. The bad news is, so are they.

Successfully championing a new service inside your organization will require you to convince people that the rewards of the new service are greater than the risks it will introduce (there’s a guide below to help you).

You’re convinced the rewards are real. Let’s talk about the risks.

The past year has seen cases of hackers using third party services to target everything from government agencies, to activists, to Target…again. Not to be outdone, attention-seeking security companies have been actively hunting for companies exposing customer data then issuing splashy press releases as a means to flog their products and services.

A key feature of these name-and-shame campaigns is to make sure that the headlines are rounded up to the most popular customer – the clickbait lead “MBM Inc. Loses Customer Data” is nowhere near as catchy as “Walmart Jewelry Partner Exposes Personal Data Of 1.3M Customers.”

While there are scary stories out there, in many, many cases the risks will be outweighed by the rewards. Telling the difference between those innumerable good calls and the one career-limiting move requires thoughtful consideration and some up-front risk mitigation.

When choosing a third party service, keep the following in mind:

- The security risks of a service are highly dependent on how you use it.
  You can adjust your usage to decrease your risk. There’s a big difference between sending a third party your server metrics vs. your customer’s personal information. Operational metrics are categorically less sensitive than, say, PII or PHI (if you have scrubbed them properly).
- There’s no way to know how good a service’s security really is.
  History is full of compromised companies who had very pretty security pages and certifications (here’s Equifax circa September 2017). Security features are a stronger indicator, but there are a lot more moving parts that go into maintaining a service’s security.
- Always weigh the risks vs. the rewards.

There’s risk no matter what you do – bringing in the service is risky, doing nothing is risky. You can only mitigate risks up to a point. Beyond that point, it’s the rewards that make risks worthwhile.

Context is critical in understanding the risks and rewards of a new service.

You can use the following guide to put things in context as you champion a new service through the gauntlet of management, security, and compliance teams. That context becomes even more powerful when you can think about the approval process from the perspective of the folks you’ll need to win over to get the okay to move forward.

In the next part of this series Lilly Ryan shares a variety of techniques to take on the perspective of your management, security and compliance teams, enabling you to constructively work through responses that can include everything from “We have concerns…” to “No” to “Oh Helllllllll No.”

Championing a new service is hard – it can be equally worthwhile. Good luck!

George Chamales is a useful person to have around. Please send critiques of this post to george@criticalsec.com

“A Security Guide for Third Party Services” Worksheet

Note to thoughtful service providers: You may want to fill parts of this out ahead of time and give it to your prospective customers. It will provide your champion with good fortune in the compliance wars to come. (Also available as a nicely formatted spreadsheet.)

Our Reasons
Why this service?	This is the justification for the service – the compelling rewards that will outweigh the inevitable risks. What will be true once the service is online? Good reasons are ones that a fifth grader would understand.
Our Data
Data it will / won’t collect?	Describe the classes or types of data the service will access / store and why that’s necessary for the service to operate. If there are specific types of sensitive data the service won’t collect (e.g. passwords, Personally Identifiable Information, Patient Health Information) explicitly call them out.
How is data be accessed?	Describe the process for getting data to the service. Do you have to run their code on your servers, on your customer’s computers?
Our Costs
Costs of NOT doing it?	This are the financial risks / liabilities of not going with this service. What’s the worst and average cost? Have you had costly problems in the past that could have been avoided if you were using this service?
Costs of doing it?	Include the cost for the service and, if possible, the amount of person-time it’s going to take to operate the service. Ideally less than the cost of not doing it.
Our Risk – how mad will important people be…
If it’s compromised.	What would happen if hackers or attention-seeking security companies publicly released the data you sent the service? Is it catastrophic or an annoyance?
When it goes down?	When this service goes down (and it will go down), will it be a minor inconvenience or will it take out your primary application and infuriate your most valuable customers?
Their Security – in order of importance
SSO & 2FA Support?	This is a security smoke test: If a service doesn’t support SSO or 2FA, it’s safe to assume that they don’t prioritize security. Also a good idea to investigate SSO support up front since some vendors charge extra for it (which is a shame).
Fine-grained permissions?	This is another key indicator of the service’s maturity level since it takes time and effort to build in. It’s also something else they might make you pay extra for.
Security certifications?	These aren’t guarantees of quality, but it does indicate that the company’s put in some effort and money into their processes. Check their website for general security compliance merit badges such as SOC2, ISO27001 or industry-specific things like PCI or HIPAA.
Security & privacy pages?	If there is, it means that they’re willing to publicly state that they do something about security. The more specific and detailed, the better.
Vendor’s security history?	Have there been any spectacular breaches that demonstrated a callous disregard for security, gross incompetence, or both?
BONUS Questions	Want to really poke and prod the internal security of your vendor? Ask if they can answer the following questions: How many known vulnerabilities (CVEs) exist on your production infrastructure right now? At what time (exactly) was the last successful backup of all your customer data completed? What were the last three secrets accessed in the production environment?
Our Decision
Is it worth it?	Look back through the previous sections and ask whether it makes sense to: * Use the 3rd party service * Build it yourself * Not do it at all Would a thoughtful person agree with you?

charity.wtf

charity wtf's about technology, databases, startups, engineering management, and whiskey.

Outsource Your O11y: Now Roll It Out And Keep Them Happy (part 3/3)