Secure Strategies For Managing Passwords, API Keys, and Other Secrets

After going through some of the most common ways that companies screw up when managing their passwords and secrets, let’s consider a number of better options.

Passwords

To start, passwords. If there’s one thing all the experts agree on, it’s that they are lame.

Nobody likes them. We don’t like remembering them, and we like forgetting them even less.

But like them or not, we do need them, and we therefore need some way to keep track of them and distribute them. Enter the password manager.

Password Managers

In 2018, there’s really no excuse not to be using a password manager. If you aren’t, it’s very likely that you’re resorting to insecure password practices, needlessly wasting time and mental energy, or (most likely) all of the above.

There are a lot of password managers out there, and digging into the details of each one is more than I want to get into here. I personally use 1Password, and I’ve also heard good things about Dashlane and Keepass. Lastpass is also popular, though they’ve had some security issues in the past.

In the end, as long as it’s end-to-end encrypted and has a good track record, just use one that you like. Apart from the security benefits, it makes life a lot easier.

Application Secrets

Application secrets are things like API keys, database credentials, and admin passwords. Your app needs them to function, but they can cause a lot of damage if exposed.

These are tricker to deal with than passwords, and there’s also a wider variety of techniques being used to protect them.

Why is that?

1. They’re multi-dimensional — API keys have test and production versions. Credentials for databases, caches, and other services also differ across environments.
2. They usually need to be distributed far more widely than passwords. Every developer working on a project needs a copy of the development secrets, and test servers, staging servers, and production servers all need a version of the secrets too.
3. They’re used in development and server contexts. That means when things go wrong, it can halt development, cause downtime, and lead to major security issues. Keeping passwords organized is important too, but there are fewer moving parts involved.
4. Development workflows, server architectures, and deployment strategies vary widely across teams and projects. Because secrets management touches all these areas, approaches to it tend to vary a lot as well.

Now that you understand the challenges, let’s look at a number of reasonably secure strategies for dealing with these things.

Strategy 1: Cloud-Hosted, Not End-To-End Encrypted

Secrets are stored with a cloud hosting provider. This includes AWS Parameter Store, Google Cloud’s Firebase, Heroku config vars, and others.

Pros

• They tend to be pretty easy to integrate and make available to servers and other services on the same platform.
• While you do need to trust the provider to protect your secrets, you’re presumably already trusting them to run your production servers, so there isn’t much additional risk.
• No additional accounts or services are needed.

Cons

• They usually give you a web UI to edit secrets. This is convenient, but as I mentioned in Part 1, browsers aren’t really secure enough to manage secrets. This can be mitigated by avoiding full-permission browser extensions or using CLI tools instead of web UIs, but it’s better not to have the temptation.
• AWS Parameter Store, in particular, can still be pretty difficult to configure and integrate with an application. It’s not a big problem for AWS experts, but it will likely increase the amount of time you spend on server setup, deployments, integrations, maintenance, and other related things. If you’re an AWS newbie, you can expect it to make the already steep learning curve a fair amount steeper.
• Access control systems like IAM can become complicated and error-prone as teams and infrastructure expand.
• They don’t help you protect development-level secrets. Securing these as well as production secrets is an important component of a really solid secrets management approach.

Strategy 2: Self-Hosted, End-To-End Encrypted

Access to secrets is controlled by a central server on your own infrastructure. In most cases, secrets are also end-to-end encrypted so that the server itself doesn’t have access.

The well known options for this approach are open source. There’s Hashicorp’s vault, Lyft’s confidant, Square’s keywhiz, and a bunch of others.

Pros

• Secrets remain completely under your control and never end up on external servers (whether encrypted or not).
• Secrets are end-to-end encrypted. This allows you to decouple access to secrets from access to the storage layer.
• It depends on the tool, but some can be used to encrypt and control development secrets in addition to production secrets.
• Useful features like secrets generation, auto-expiration, and audit logs.

Cons

• Setting up a secrets server and integrating it with your applications is a significant project. It requires a fairly high level of devops sophistication, and will likely take between n days and n weeks depending on your infrastructure’s complexity and your team’s capabilities.
• There’s generally not enough attention paid to user friendliness and developer friendliness in these tools. Adding complexity to development and deployment workflows can slow down and frustrate engineers. They might then be tempted to route around the system in insecure ways when they’re in a hurry.
• Apart from setup and integration, you’ve got a new piece of critical infrastructure to maintain, upgrade, monitor, and make highly available. These are non-trivial, time-consuming, and highly unforgiving tasks.
• While some of these tools can be used to manage development-level secrets, production secrets tend to be given a lot more focus in terms of design and documentation. This means that you’re generally left to your own devices on fitting it into your development workflow.

Strategy 3: Encrypted With Configuration Management Tools

Many configuration management tools like Ansible, Chef, Kubernetes, and others have built-in or community-provided secrets management libraries.

Pros

• Fits in with your existing toolset and deployment workflows.
• Helps you to manage all your production secrets and configuration in one place.

Cons

• Managing encryption keys securely is often left to the user. It’s easy to make mistakes here that defeat the purpose of the whole thing.
• It can add complexity to your code, deployment scripts, and test server environments.
• It doesn’t do anything for development secrets. As mentioned above, a solid strategy should protect these as well as production secrets. The line is fuzzy.
• You can still end up with configuration sprawl as the number of services and applications you manage grows.
• Strong access control can still be difficult to achieve, particularly in a way that doesn’t require trusting additional third parties.

Strategy 4: End-To-End Encrypted Messages or Files

Tools like Keybase, Signal, and others can be used for this.

Pros

• Messages are end-to-end encrypted so that you don’t need to trust any third parties.
• Any tool worth its salt will offer native apps so that you aren’t dealing with secrets in a browser.
• Generally pretty quick to get up and running. It provides some basic security without needing to go all in on a more comprehensive approach.

Cons

• Relying on messages to keep developers and servers in sync is error-prone, tends to get disorganized quickly, and is a constant source of delays and interruptions.
• Some tools, like Keybase, have web messaging options in addition to their native apps. This tempts people to share secrets in a browser, which as mentioned above can be a problem due to all those devilishly convenient browser extensions looking over your shoulder.
• While tools like Keybase and Signal are the real deal when it comes to zero-knowledge encryption implementations, there are a lot of pretenders out there in the messaging space. If you choose a messenger that’s web-based or gets the crypto wrong, then you’re back in Anti-Pattern land.

Strategy 5: EnvKey

Disclaimer: EnvKey’s creator is the author of this post.

EnvKey is like a password manager, but for application secrets and configuration. It’s a cloud-hosted, end-to-end encrypted system that allows you to securely configure a development machine or server with a single environment variable, and then keep it in sync automatically going forward.

It also provides a desktop app where you can manage configuration and access across all your apps, teams, and environments in one place.

Pros

• Instead of days or weeks, EnvKey usually takes between 5 and 15 minutes to install and integrate. It was designed from the ground up for ease of use and developer-friendliness. Integration is as simple as pasting in a line or two of code and setting an environment variable.
• It doesn’t require trusting any third parties. EnvKey is designed to prevent anyone except authorized users and servers from having access to your secrets. This includes EnvKey itself. End-to-end encryption based on OpenPGP ensures that we can’t access our own users’ secrets.
• It gives you simple, common-sense access controls that are easy to manage and hard to screw up.
• No more configuration sprawl. It gives you a single place to manage secrets, configuration, and access control across teams, apps, and environments.

Cons

• It costs money. It costs a lot less than containing a breach or having your developers setup something more complicated, but after the free trial ends, you do have to plunk down that credit card.
• You have to trust EnvKey’s end-to-end encryption implementation. Our apps and client libraries are designed specifically not to trust the EnvKey server, but if you can’t trust the crypto, then this doesn’t help much. That’s why EnvKey relies on OpenPGP and high level, audited libraries for its crypto needs. All EnvKey clients — the app and its client libraries — are open source, and everything about our crypto and security choices is thoroughly documented.
• It’s a third party dependency. You’re relying on EnvKey to deliver configuration and secrets when your application processes start. That’s why it runs a high availability server architecture and offers failover to encrypted S3 backups in a separate region.
• You still need common sense. EnvKey minimizes the number of secrets you need to distribute and makes the path of least resistance a secure one, but at the end of the day, no tool can completely save you from yourself.

Almost-Honorable Mention 1: Encrypted Within Source Control

Encrypting secrets and keeping them in source control is another fairly popular solution to secrets management. While this is light years better than email, Slack, spreadsheets, and the like, I still consider it an anti-pattern. It’s not terrible, but there are much better options available.

Almost-Honorable Mention 2: Custom End-To-End Encrypted Systems

A lot of companies also build their own custom solutions to deal with secrets. This is usually a bad idea, but if you have particularly complex needs that no other tool can handle, this might be your only option. I’d still try to avoid it though — a large majority of teams that take the DIY approach would be better off if they had chosen any other option in this list.

Want to be notified when news posts about development, devops, and practical security come out?