Secret Management and Kubernetes Part 1

What is Secret Management?

Over the past 12 months, the focus of much of my work has been assisting organisations in identifying and implementing a solution around two of the most common questions we see pop up within organisations striving to establish strong DevOps practices.

• How to manage secrets within our organisation?

• How to securely consume these secrets in our Kubernetes environments?

In this 2 part series, I want to outline the necessity of a secret management solution within any organisation, as well as what third-party secret management solutions are available for working with secrets in Kubernetes.

So what does the term Secret Management really mean? Secret Management is defined as the tools and methods for managing digital data to protect critical assets from unauthorised access, and is critical for all businesses, whether or not they employ DevOps because all businesses use digital secrets to some extent.

Let’s also start by defining what a secret is in terms of security.

Secrets include passwords, APIs, encryption keys, SSH keys, tokens, and other digital credentials that allow users and applications access to sensitive data, systems, and services. This is why it’s so important to keep secrets safe while they’re in transit and at rest, as well as to handle them securely.

As the number of systems, resources, and applications grows within a business, the number of secrets that must be handled and kept secure will rise as well. Establishing standards to manage secrets makes it easier to validate the integrity of the secret and also control various stages of a secret’s lifecycle which assist businesses in achieving security and regulatory obligations. Without a comprehensive secret management strategy, organisations risk being unable to maintain secrets effectively and thereby jeopardising both the integrity of secret storage and the security of secrets themselves.

Why do we need it?

Among the most prevalent reasons to invest in Secrets Management include:

• Auditing: Auditing is always vital throughout every organisation as it allows businesses to ensure regulatory compliance and provides vendor and client assurance. Regular auditing allows for the detection of security threats within any organisation also and leads to improved policies and procedures plus the ability to identify and validate who has access to confidential data. Auditing also aids in assuring that only the data associated with existing applications is stored.

• Avoid Plain Text Secrets: It is not uncommon to find sensitive data being shared utilising a shared text file containing all of the passwords to vital resources in the absence of Secret Management technologies. It’s also typical to receive emails or messages with the passwords for specific applications, resources, or accounts. Multiple security problems arise from sending and storing secrets in plain text, making it much easier for an attacker to get access to the target system. All they have to do is get a file, an email, or a text message.

• Dependencies on Individuals: Many businesses have employees who have been with the company for a long period, and there is frequently a reliance on these employees when it comes to secrets and sensitive information. Rather than the key information around an account, applications or even databases being stored in a centralised and globally accessible resource, one will instead discover the information retained and stored in the memory of these long-serving employees. As a result, there is an unwarranted and dangerous reliance on these individuals, which means in the case they are absent or leave the organisation, this information will be lost.

• Secret Rotation: The majority of businesses are expected to follow security standards like the Payment Card Industry Data Security Standard (PCI DSS), which mandates the rotation of application keys and other sorts of secrets every 90 days. Organisations frequently overlook the need for rotating secrets, which is critical because it defines a data life cycle, ensuring that if an attacker obtains them after a certain length of time, the duration in which they may be exploited is restricted.

• Reputation: Any business that can demonstrate it can safeguard both application and customer data will enhance its brand value and reputation, which also will lead to additional business opportunities. Internally, Engineering and Technical teams that have established sound security strategies in place will always instil confidence in executives and other divisions.

What should we look for?

In order to handle secrets securely and effectively, organisations should adopt a centralised secrets management solution that has the ability to handle many types of secrets in a secure and efficient manner.

The key elements any Secrets Management strategy must include to be successful are:

• Secret Creation: Solutions should support the ability to cater for creating and handling both static secrets, created manually by users, such as a password and also dynamic secrets generated automatically, such as an encryption key. Identify what type of secrets and confidential data your business and clients will need to create within your environment and ensure the solution supports their creation in the manner that aligns with your businesses security guidelines.

• Access Control: The security measures of any secrets management solution are critical and will often be the key evaluation criteria. When it comes to Secret Management, the ability to demonstrate and clearly define “who can create secrets?”, “who can read and access secrets and how?” are vital to its reliability and integrity. One of the strengths of many solutions is their ability to integrate with 3rd party authentication systems often already in place within the business. Know the authentication systems your business is currently utilising, and can they work with your choice of Secret Management to help simplify the process of authentication and authorisation. Identify how you wish to control the permissions associated with your secrets along with understanding the different levels of access required based on roles and the environment they are working in and what type of access should be in place, e.g A Developer having Read and Write access for Development and Read Only for Production.

• Storage: A crucial part of any solution will encompass data security and ensure that data is stored in a durable location. Map out whether secrets should be kept in a central location or handled using namespaces when data should be segregated between apps, users or teams. Follow up within your business and especially with clients to make yourself aware of the guidelines and standards any solution must implement to meet compliance in line with storage requirements. These requirements will govern the desired state of your secrets at rest based on security policies, and certify the solution can fulfil these requirements. Assure the long-term viability of both enterprises and client data by establishing a backup and disaster recovery plan in the event of a disaster. Understand the procedures used in the backup and restore processes to ensure that data is not transmitted or stored in clear text, clarify the integrity and security of the backup storage so that the whole solution is not jeopardised at the backup level.

• Documentation & Policies: Along with any Secret Management solution, policies and documentation must be in place to define how Secrets are handled and governed. Include any external standards about vital requirements such as mandatory rotation periods, data transmission security, and other obligations. Documentation should include information such as the steps for creating a secret, the naming standards being enforced. onboarding processes for teams and applications to utilise the solution, plus any other fundamental information. The key to making documentation successful involves making it as detailed and up to date as possible and available to all appropriate personal such as Engineers to Executives. Remember to keep the details of your documentation protected away from those who might use it as a way to identify weaknesses in your solution, furthermore either as part of the primary documentation or as a separate document a strategy for dealing with cybersecurity events and revoking compromised secrets. Note that the policies that govern your environment will compose the reference point for auditing, leading to missing policies not found or referenced in the documentation that will likely not have been enforced.

• Integration: Possibly the most vital decision is around integration and choosing a Secret Management tool that will make things go as smoothly as possible and take advantage of existing tools within the business, reducing costs and time. If existing agreements with cloud providers are already in place, do they provide an option we can leverage? How easy would it be to retrieve secrets from the Secret Management tool of choice? Will utilisation by existing applications, services, or an orchestration tool such as Kubernetes be possible.

What next … Secret Management and Kubernetes

Given that we’ve just discussed and defined Secret Management in regard to the first question we set out to answer, “How should we manage our secrets within our organisation,” let’s move on to the second question, “How should we use Secrets in Kubernetes in a secure manner?”

By design, Kubernetes uses a Secret object to handle sensitive data, which is encoded using Base64, which is effectively similar to plaintext. For the majority of organisations, storing confidential and sensitive data in a basic object that is not fully secured at rest and whose contents are widely available is an inadequate strategy for handling confidential and sensitive data within Kubernetes.

Here one can view what a Kubernetes Secret resource looks like

apiVersion: v1
kind: Secret
metadata:
  name: secret-application-one
  annotations:
    kubernetes.io/service-account.name: "application-one"
type: kubernetes.io/service-account-token
data:
  extra: aW5uYWJscg==

At present Kubernetes has the ability to handle a wide range of secret types, can be specified utilising both the annotation and type fields, for more information see Secret Types | Kubernetes .

As we mention the details of password are clearly visible and using a simple command or online tool to extract its contents

echo -n 'aW5uYWJscg==' | base64 --decode
innablr%

After creating a secret we need to identify how we will consume it within our pods and Kubernetes offers us two ways:

• Mount it as an environment variable

• Mount it as a volume.

The simplest option is to reference the secret as an environment variable from within the pod as seen here,

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_ACCOUNT
        valueFrom:
          secretKeyRef:
            name: applicationsecret
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: applicationsecret
            key: password
  restartPolicy: Never

However, not everyone is satisfied with sensitive data being maintained in environment variables, as environment variables frequently end up in logs, making monitoring who has access to these logs an additional duty.

As a response, many individuals have moved to storing secret information in volumes, which means that the hidden content of the container can be accessed as if it were a regular file. Kubernetes handles this by creating a file with the decoded value as the file content for each key in the secret’s data.

Below one can see how one can reference a Kubernetes secret using volumes.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: busybox
    volumeMounts:
    - name: boo
      mountPath: "/etc/boo"
      readOnly: true
  volumes:
  - name: boo
    secret:
      secretName: applicationsecret

Despite this, there is still a requirement for the secret to exist within the Kubernetes environment, making the data it controls vulnerable and still accessible.

As a result, businesses, engineers, and Kubernetes users will turn to and investigate the use of third-party secret management systems, which is where we will focus our attention now.

Kubernetes Secret Management

There are two main types of third-party tools available for handling Secret Management with Kubernetes.

- Open Source Tools
- Cloud Provider Tools

Rather than outlining how to implement them, we’ll cover a few popular and reputable ones enough to raise awareness of the options available and the primary benefits each provides.

Open source tools

Akeyless (Akeyless) Akeyless Vault is a centralised secrets management platform for managing and protecting all forms of Secrets offering a plugin for integration with Kubernetes,. This plugin uses the Kubernetes Mutating Admission Webhook to inject secrets into pods, with the init sidecar container fetching the secret before the application starts. This approach means applications will be able to access the secret at a filesystem path for use.

Key Features

• Offers segregation between different Kubernetes clusters and their secrets

• Access to Akeyless can be maintained using SSO,and AWS IAM)

• Utilises Distributed Fragments Cryptography

• Supports multiple options to authenticatication to Kubernetes clusters (API Key,Azure Active Directory,LDAP,Open ID,SAML,AKeyless Encryption Technology

Hashicorp Vault (Hashicorp Vault) Hashicorp Vault is a popular and established option for securing, storing, and controlling access to passwords, certificates, and other secrets. Vault currently employs the Vault Agent Injector service, which makes use of the Kubernetes mutating admission webhook to inject a Vault Agent container to handle the secrets, which are stored on the container’s file system.

Key Features

• Access to secrets can be enforced via Kubernetes service accounts and namespaces

• Implements encryption at rest and in transit using mTLS

• Known and proven solution

• Offers the ability to run side-by-side or within Kubernetes

• Secret rotation and centralised auditing

• Vault ensures the safety of multi-datacenter replication and disaster recovery so you can keep your applications running smoothly.

CyberArk Conjur (CyberArk) CyberArk Conjur is the open source core of CyberArk Application Access Manager (AAM) designed to provide secrets management for applications and non-human identities. The key to its Kubernetes integration involves enabling applications running in Kubernetes to authenticate with Conjur using the Kubernetes Authenticator that must exist in the same pod as the application. This achieved using either a sidecar or init container.

Key Features

• Access Request Management

• Implements encryption at rest with AES-256-GCM and in transit using mTLS

• Robust and efficient solution.

• Role Management

• Secret rotation and centralised auditing

In addition to these, there are a number of other open source solutions to explore that allow you to encrypt a Kubernetes secret and store it in a Git repository, such as:

• Kamus (Kamus)

• Sealed Secrets (Sealed Secrets)

• Secrets OPerationS (SOPS)

Cloud provider tools

AWS Secrets Manager (AWS Secrets Manager) A product of AWS, Secrets Manager provides protection of secrets needed to access your applications, services, and IT resources. Throughout the lifecycle of database credentials, API keys, and other secrets, the service allows you to effortlessly rotate, manage, and retrieve them.

Key Features

• Secure secrets storage

• Automatic secrets rotation without disrupting applications

• Automatic replication of secrets to multiple AWS Regions

• Ability to audit and monitor secrets usage

• Compliance with the Security standards required highly secure government organisations

Google Cloud Secrets Manager (Google Secret Manager) Secret Manager is a safe and easy-to-use solution for storing API keys, passwords, certificates, and other sensitive information. Secret Manager is a single source of truth for managing, accessing, and auditing secrets throughout Google Cloud.

Key Features

• Replication policies to control where secrets are replicated

• Secret versioning that encompasses aliases and pining to a specific version.

• Cloud IAM integration access to secrets the same way access to other resources

• Audit logging

• Encrypted by default at rest

Azure Key Vault (Azure Key Vault) Azure Key Vault is a Microsoft Azure service that allows users to store and use cryptographic keys. For high-value customer keys, Azure Key Vault offers a variety of key types and algorithms, as well as the use of Hardware Security Modules (HSM).

Key Features

• Key Vault authentication options

• Logging and monitoring

• Privileged access using Azure RBAC

• Backup and recovery to recover deleted vaults and vault objects.

Conclusion

In summary, we’ve looked at how Secret Management can be used to manage our Secrets within our organisation, and what to look for when selecting a tool, as well as how Kubernetes handles secrets and what alternatives are available to take our existing solution to the next level.

You might also be wondering, “What is the best Secret Management tool?” and “Which one should I use?”. Finding the best tool to use is a matter of figuring out which one best fits the demands of your environment and applications. Although Vault is the most popular and platform-agnostic solution on the list, cloud service providers' tools connect better into services within the same ecosystem. They do, however, require the use of an additional driver or plugin to interact with Kubernetes given they do not provide any native integration.

In part two we will take a look this and demonstrate an ever-growing approach to integrating Kubernetes with both Cloud-based Secret Managers and Hashicorp’s Vault.

Innablr is a leading cloud engineering and next generation platform consultancy. We are hiring! Want to be an Innablr? Reach out!