Managing Passwords and Application Secrets: Common Anti-Patterns

10 ways that development teams screw this up

Come gather ‘round all you developers, technical managers, and sys admins. Give your brainy heads a conspiratorial tilt in this direction.

A little closer. That’s better. Now, let’s talk about secrets.

No, I don’t mean that thing you do in the… with the… That’s… you really shouldn’t tell anyone about that.

No, I’m talking about the secrets you use at work and on side projects. Passwords. API keys. Database credentials. The admin dashboard password. The Redis connection string. The stuff your app needs to do its thing, but could cause quite a bit of damage if you were to, say, write them on a post-it note and leave it in this guy’s line of sight:

So, what’s the problem? Piece of cake, right?

You keep them encrypted, manage them securely, control access tightly, and rotate them on a regular basis. No one sends secrets over email or Slack. You never lose time debugging issues caused by missing or outdated config. You treat development secrets with the same care as production secrets.

All is well in the Kingdom Of Secrets.

BEEP, BEEP, BEEP. My, what a pleasant dream.

If your company is really on top of its ops practices, then maybe this dream is your reality. If so, kudos!

If not, you’re far from alone. This stuff is pretty tough to get right. Part 2 of this post will explain the various options for doing this securely, but to start I’m going to go through the some of the most common ways that development teams screw this up.

Anti-Pattern 1: The Shared Password

Your team uses the same shared password for everyone and everything.

Benefits:

• It’s easy.
• In addition to making accounts accessible to all, it takes care of API keys and credentials too since people can just sign in to each service’s dashboard and copy what they need.

Why It’s Bad:

• No access control.
• If a single employee or third party service is careless with your password and it leaks, every aspect of your business is at risk.
• Let’s be honest: if you’re doing this, you’re probably not changing the shared password whenever someone leaves the company. Remember Shelly The Vindictive Marketer who left in a cold rage after getting fired last Friday?

I hope you’re afraid.

Anti-Pattern 2: The FILE

Passwords and secrets are stored in a text or spreadsheet file. It usually lives in Dropbox, Google Drive, or some other widely accessible place.

Benefits:

• Not quite as easy as the shared password, but still pretty easy.
• Storage services use https so secrets are encrypted over the wire.

Why It’s Bad:

• If a single employee’s account gets hacked, there goes the neighborhood.
• If people are accessing the FILE in a web browser, you’re implicitly trusting every single browser extension anyone in your company has installed. This is one of several issues that makes dealing with secrets in a browser context fundamentally unsafe. More to come on this topic in future posts.
• You’re unnecessarily trusting a third party like Dropbox or Google. This may not be too big of a deal since they are presumably quite trustworthy, but it’s still better to avoid it.
• The FILE very quickly gets disorganized and outdated. Apart from being no fun to work with, this becomes a security issue in itself because it tends to lead to the next anti-pattern.

Anti-Pattern 3: Sharing Over Email

Secrets are emailed from whoever sets and stores them to whoever needs them.

Benefits

• Easy enough. It gets the password/API key/secret to whoever needs it so they can keep working.

Why It’s Bad

• If a single employee’s email is hacked, there goes Kansas.
• There’s no ability to remove access apart from resetting every password and rotating every API key, which come on, let’s not kid ourselves about that happening at 4pm on a Friday. Everything Shelly needs to satisfy her thirst for vengeance is a 500 millisecond Gmail search away.
• While it’s less of an issue in the Everyone Uses Gmail Era, emails to non-TLS enabled recipients won’t be encrypted over the wire.
• Webmail has the same issues with trusting extensions that I mentioned above about accessing the FILE in a browser.
• Unless you run your own email server, there’s the minor but unnecessary disadvantage of trusting a third party like Google with all your secrets.
• Plugins that do useful things in exchange for access to your inbox have become common. If a single plugin is compromised, you’re in (more) trouble.
• Causes frequent interruptions and annoyance since people are constantly pinging each other to ask for some password or API key.
• Long email conversations can cause something sensitive that’s buried in a thread to get accidentally sent to people who shouldn’t see it.

Anti-Pattern 4: Sharing Over Slack, Skype, SMS, iMessage, Whatsapp, etc.

Instead of email, secrets are shared over chat tools or text messages.

Benefits

• Basically the same benefits as email. It’s simple and it gets people what they need.
• Messages are encrypted over the wire.

Why It’s Bad

• These have similar drawbacks to sharing over email. Surface area for hacks is greatly increased, access control is lacking, trusting a third party is required, and people have to constantly bug each other to get their work done.
• Using tools like Slack that offer all those ever-so-useful bots and plugins can mean trusting additional third parties with sensitive data.

Anti-Pattern 5: Web-based Pastebin, Exploding Message, and Encrypted Chat Services

Secrets are shared via messaging services that encrypt and/or automatically delete messages after they’re received.

Benefits

• When implemented with one-time links, access control is fairly straightforward.
• If messages are deleted, this reduces the attack surface area since a hacked account won’t contain a backlog of secrets.

Why It’s Bad

• It’s nice that the service tells you it deletes your messages, but promises don’t mean a lot in the world of security. There’s no way to verify that the service didn’t retain your data somehow, either maliciously or by mistake.
• As mentioned previously, browsers are not sufficiently secure for dealing with secrets, even when a service uses encryption. Client-side encryption is useless against cross-site scripting (XSS) attacks, extension-based attacks, and compromise of the host server. End-to-end encryption implemented in-browser offers only marginal security benefits and can create a dangerously false expectation of privacy.

Anti-Pattern 6: The Git Repo, Unencrypted

This usually applies more to application secrets than passwords. API keys and credentials are stored right in the codebase in plain text.

Benefits:

• It’s easy for developers. No need for any special deployment steps or server configuration.
• Everything stays in sync — it’s one of the things git does best.

Why It’s Bad:

• Secrets access is now coupled to code access. You can’t give a developer or a third party service access to your codebase without also revealing all of your production secrets.
• There’s no ability to quickly cut off access since everyone who ever cloned the repo will have all the secrets in their git history.

Anti-Pattern 7: Gitignored Files and Environment Variables (The 12-Factor App Methodology)

The 12 factor app methodology popularized the idea of getting secrets out of source control and instead storing them in the environment. On servers, this means environment variables. In development, it usually means gitignored .env files, yaml files, or something similar.

Benefits:

• It’s a lot better than storing secrets in code. Instead of coupling production secrets access to codebase access, it gets coupled to production server access, which makes much more sense.
• It’s flexible and works well with platform-as-a-service (PaaS) providers like Heroku.

Why It’s Bad:

• There’s no obvious way to keep developers and servers in sync. This leads to sharing secrets over email, Slack, and other inappropriate tools.
• For the same reason, it’s error prone. It’s easy for values to become missing or outdated, which leads to time wasted on debugging, failed CI builds, and errors in production.
• It discourages secrets rotation, since it becomes a tedious process of updating values in many separate locations.
• It doesn’t scale well — all the previously mentioned problems are compounded as your team grows in size, more services are deployed, and your infrastructure becomes more complex. Before long, you end up with a sprawling configuration that is very difficult to keep track of.

Anti-Pattern 8: The Git Repo, Encrypted

Secrets are stored in the codebase and version-controlled, but are encrypted somehow so that only authorized users and/or servers can access them.

This one may be a bit controversial, as it is employed by some popular configuration management tools like Ansible and Chef, and is the basis for built-in secrets management in Rails 5.

Benefits:

• Git keeps secrets in sync and available, while codebase access is no longer strictly coupled to secrets access.

Why It’s Bad:

• Dealing with encryption keys securely can be an issue. For example, if you have multiple people on the team who deal with servers, you need a way to get the appropriate keys to all of them. People will be tempted to use email, Slack, etc. for convenience.
• Codebase access still gives access to the encrypted production secrets. You can’t revoke access since they will live on in everyone’s git history. Similarly, if a production private key leaks, everyone with access to the repository now has access to the production secrets.
• Encrypted data can’t be easily merged.
• Depending on the tool you use, it may encrypt production secrets but not development secrets. Development secrets should be encrypted too — more on this below.

Anti-Pattern 9: Not Protecting Development-Level Secrets

Some teams do a good job on avoiding the previously mentioned anti-patterns when it comes to production-level secrets, but don’t worry much about development secrets.

Benefits:

• Saves time on setting up a secure approach for secrets that likely carry less risk than their production-level equivalents.

Why It’s Bad:

• The line between what’s a production secret and what’s a development secret is inherently fuzzy. Plenty of so-called development secrets could be damaging if exposed.
• Development API keys aren’t always properly sandboxed. Some services may not even offer separate production and development API keys.
• Production-level secrets can end up in development for the purposes of debugging, deployments, infrastructure management, or other reasons.

Anti-Pattern 10: Custom Secrets Management

A custom system for managing secrets is built in-house.

Benefits:

• Your approach (hopefully) avoids all the other anti-patterns in this list.

Why It’s Bad:

• It’s a lot of work. Probably a lot more than you expect.
• Unless you spend a huge amount of time on it, it’s unlikely to be user-friendly and developer-friendly. It probably complicates deployments and server management, and gets harder to maintain over time. People will be tempted to route around the system insecurely if it gets in their way.
• It’s easy to make mistakes that effectively render the whole thing pointless. Are you using established, proven crypto libraries? If you’re rolling your own crypto and dealing with primitives, it’s almost certain to be broken somehow. Are you distributing keys securely? There are plenty of potential pitfalls. Are you signing and verifying data in addition to encrypting? If you’re using public key crypto, are you implementing a web of trust to verify public keys?

What are the alternatives?

If your team is following one or more of these anti-patterns and that makes you sad, then I have some good news for you: the situation can be fixed without too much effort.

Check out Part 2: Secure Strategies For Managing Passwords, API Keys, and Other Secrets, where I compare and contrast some better ways of managing secrets.

If your team shares API keys and other secrets over email, Slack, or Google Docs, EnvKey can help. You’ll improve security, speed up development, prevent bugs, and simplify ops with a ~15 minute time investment by keeping configuration securely in sync for developers and servers.

Go here to learn more and download the latest version of EnvKey.