Introducing the AWS Secrets Manager Credentials Provider for Jenkins
API keys and secrets are difficult to handle safely, and probably something you avoid thinking about. In this post I’ll show how the new AWS Secrets Manager Credentials Provider plugin allows you to marshal your secrets into one place, and use them securely from Jenkins.
When CI/CD pipelines moved to the public cloud, credential management did not evolve with them. If you’re in this situation, you may have seen a number of tactical workarounds to keep Jenkins builds talking to the services they depend on. The workarounds range from bad (hardcoding plaintext secrets into Git) to merely painful (wrangling Hiera EYAML), but their common feature is that they tend to make copies of secrets beyond the reach of automation. This increases their attack surface, makes routine key rotation impractical, and makes remediation difficult after a breach.
The good news is that there is a better way!
AWS Secrets Manager is a comprehensive solution for secure secret storage. You define a secret just once for your whole AWS account, then you give your consumers permission to use the secrets. Secrets Manager lets you manage a secret entry (name and metadata) separately from its value, and it integrates with other AWS services that you already use:
-
Secret entry management: Manual (Web console, AWS CLI) or with an infrastructure management tool (Terraform, CloudFormation etc.)
-
Secret value management: Manual (Web console, AWS CLI) or automatic (secret rotation Lambda function).
-
Access control: AWS IAM policies (for both applications and human operators).
-
Secret encryption: Amazon KMS automatically encrypts the secret value. Use either the account’s default KMS key, or a customer-managed KMS key.
-
Auditing: AWS CloudTrail and CloudWatch Events.
A couple of teams in my company started to use Secrets Manager from Jenkins jobs by calling the AWS CLI, but this remained a niche approach as it was quite unwieldy. There was clearly an appetite to integrate key developer apps with a centralised secrets store, but production-ready integrations were needed for wider adoption. So this year I created the AWS Secrets Manager Credentials Provider plugin for Jenkins, with help from friends in the Jenkins community, to do exactly that.
This is how you set it up…
-
Install the plugin from the Jenkins update center.
-
Give Jenkins read-only access to Secrets Manager with an IAM policy.
-
(Optional) Configure the plugin, either through the Global Configuration screen or Jenkins Configuration As Code.
This is how you use it…
-
Create your build secrets in AWS Secrets Manager. (You can start by uploading secrets via the AWS CLI. More sophisticated methods of secret creation are also available.)
-
View the credentials in the Jenkins UI, to check that Jenkins can see them.
-
Bind the credentials by ID in your Jenkins job.
The provider supports the following standard Jenkins credential types:
-
Secret Text
-
Username With Password
-
SSH User Private Key
-
PKCS#12 Certificate
And it has powerful advantages over quick-fix tactical solutions:
-
Your Jenkins jobs consume the credentials with no knowledge of Secrets Manager, so they stay vendor-independent.
-
The provider caches relevant Secrets Manager API calls, for a quicker and more reliable experience.
-
The provider integrates with the ecosystem of existing Jenkins credential consumers, such as the Git and SSH Agent plugins.
-
The provider records credential usage in the central Jenkins credentials tracking log.
-
Jenkins can use multiple credentials providers concurrently, so you can incrementally migrate credentials to Secrets Manager while consuming other credentials from your existing providers.
After the plugin’s first public release, developers at other companies adopted it too. It has had contributions so far from people at Elsevier, GoDaddy, and Northeastern University, as well as the fantastic Jenkins core team. We even got fan mail for our work!
In enterprise security, "The important things are always simple. The simple things are always hard. The easy way is always mined." (@thegrugq) It’s easy to buy a shiny ‘next generation' security appliance and drop it into your network. But it’s hard to embed the security fundamentals (like secrets management, OS patching, secure development) across your organisation. This Jenkins plugin is part of the effort [1] to take one of the persistent hard problems in security, and make it easier for everyone.