Secret Management Best Practices for Developers

The Leaked Secrets Problem

In 2024, GitGuardian reported that over 12.8 million new secrets were detected in public GitHub repositories — a 28% increase from the previous year. These secrets include API keys, database credentials, cloud provider tokens, private encryption keys, and OAuth secrets.

Each leaked secret is a potential breach. When an AWS access key is committed to a public repository, automated scanners detect it within minutes. Attackers use these keys to spin up cryptocurrency mining instances, access S3 buckets, exfiltrate data, and more — all before the developer even realizes they've pushed their credentials.

High-profile examples:

• Uber (2016): Attackers found AWS credentials in a private GitHub repo (accessed via stolen developer credentials). They used those keys to access an S3 bucket containing personal data of 57 million users and drivers.
• Samsung (2022): Internal GitLab repos were breached, exposing secret keys, source code, and credentials for Samsung's SmartThings platform.
• Toyota (2022): An access key to a customer data server was committed to a public GitHub repo for nearly five years before discovery.

The root cause is consistent: developers hardcode secrets in source code, commit them to version control, and the secrets persist in git history even after the code is "fixed."

What Counts as a Secret

A secret is any piece of information that grants access to a system, service, or data and must be kept confidential:

API keys and tokens:

• Cloud provider access keys (AWS, GCP, Azure)
• Payment processor keys (Stripe, PayPal)
• Third-party API keys (SendGrid, Twilio, OpenAI)
• OAuth client secrets and refresh tokens
• JWT signing secrets

Database credentials:

• Database usernames and passwords
• Connection strings containing credentials
• Redis/memcached authentication passwords

Encryption materials:

• Private SSH keys
• TLS/SSL private keys
• Encryption keys and initialization vectors
• HMAC secrets

Infrastructure credentials:

• Server passwords and SSH keys
• Docker registry credentials
• Kubernetes secrets
• CI/CD service account tokens
• Cloud IAM roles and service account keys

Application secrets:

• Session secret keys
• CSRF token secrets
• Webhook signing secrets
• Admin panel passwords
• Two-factor authentication backup codes

Rule of thumb: If someone could use the information to access a system or impersonate your application, it's a secret.

Environment Variables

Environment variables are the foundation of secret management — they separate configuration (including secrets) from code.

The Twelve-Factor App methodology (widely followed in modern development) states: "Store config in the environment." Secrets are a subset of configuration and should never be in code.

How to use environment variables for secrets:

// WRONG — hardcoded secret
const stripe = new Stripe('sk_live_abc123def456...');

// CORRECT — from environment variable
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);

# WRONG — hardcoded
db_password = "super_secret_password"

# CORRECT — from environment
import os
db_password = os.environ["DB_PASSWORD"]

Best practices for environment variables:

• Never set defaults for secret values in code. If the environment variable is missing, the application should fail immediately with a clear error — not silently use a default.
• Validate required environment variables at startup. Check that all necessary secrets are present before the application begins handling requests.
• Don't log environment variables. Avoid logging the full environment (process.env, os.environ) or any variable that might contain a secret.
• Don't pass secrets as command-line arguments. Command-line arguments are visible in process listings (ps aux). Use environment variables or files.

Limitations of environment variables:

• They're not encrypted in memory and can be read by anyone with access to the process
• They may be leaked in error reports, debug logs, or crash dumps
• Child processes inherit the parent's environment variables
• They don't support rotation without restarting the application

For these reasons, environment variables are a good starting point but should be combined with a secret manager for production systems.

.env Files and .gitignore

.env files are used to load environment variables during local development. They're a convenient way to keep secrets out of code while providing them to the application.

Example .env file:

DATABASE_URL=postgresql://user:password@localhost:5432/myapp
STRIPE_SECRET_KEY=sk_test_abc123...
JWT_SECRET=your-jwt-signing-secret-here
SENDGRID_API_KEY=SG.abc123...

Critical rule: .env files must NEVER be committed to version control.

Add to .gitignore:

# Secrets
.env
.env.local
.env.*.local
.env.production

Provide a .env.example file instead. This template shows which variables are needed without containing actual secrets:

DATABASE_URL=postgresql://user:password@localhost:5432/myapp
STRIPE_SECRET_KEY=sk_test_...
JWT_SECRET=generate-a-random-secret-here
SENDGRID_API_KEY=SG....

Commit .env.example to the repository so new developers know which variables to configure.

Common mistakes with .env files:

1. Committing .env to git. Even if you delete it in a later commit, it remains in git history. An attacker who clones the repo can find it.
2. Not adding .env to .gitignore before the first commit. If .env is committed even once, it's in the history permanently. Add .gitignore rules before creating the .env file.
3. Sharing .env files via Slack, email, or unencrypted channels. Use a secret manager or encrypted sharing tool.
4. Using the same .env file for development and production. Production secrets should never exist on developer machines. Use separate configurations per environment.

If you've already committed a secret to git:

1. Immediately revoke/rotate the compromised secret. Generate a new API key, change the password, rotate the token.
2. Consider the old secret compromised — even if the repo is private. You cannot reliably remove data from git history (rewriting history is possible but not guaranteed to propagate to all clones and forks).
3. Use tools like git-filter-repo or BFG Repo Cleaner to remove the secret from history (but still rotate the secret — this is a defense-in-depth measure, not a guarantee).

Secret Managers and Vaults

For production systems, a dedicated secret manager provides centralized, audited, encrypted storage for all secrets:

Cloud-provided secret managers:

• AWS Secrets Manager — stores and rotates secrets for AWS services; integrates with RDS, Lambda, ECS
• AWS Systems Manager Parameter Store — simpler, lower-cost option for string-based secrets
• Google Cloud Secret Manager — stores, accesses, and audits secrets in GCP
• Azure Key Vault — manages secrets, keys, and certificates for Azure services
• Cloudflare Workers Secrets — environment variables encrypted at rest for Cloudflare Workers

Self-hosted secret managers:

• HashiCorp Vault — the industry-standard open-source secret manager. Supports dynamic secrets (generates short-lived credentials on demand), secret rotation, and comprehensive audit logging.
• Doppler — SaaS secret manager with excellent developer experience; syncs secrets across environments
• Infisical — open-source alternative to Doppler

How secret managers improve security:

Centralized management. All secrets are in one place with consistent access controls, rather than scattered across .env files, CI/CD configurations, and developer machines.

Audit logging. Every access to a secret is logged — who accessed what, when, and from where. This is essential for compliance (SOC 2, HIPAA, PCI DSS) and incident investigation.

Encryption at rest and in transit. Secrets are encrypted in storage and only decrypted when accessed by authorized applications.

Dynamic secrets. Instead of static credentials, Vault can generate short-lived credentials on demand (e.g., a database user/password that expires in 1 hour). Even if compromised, the credential expires automatically.

Automatic rotation. Secret managers can rotate credentials automatically (changing database passwords, rotating API keys) without application downtime.

Access control. Fine-grained policies control which applications, services, and users can access which secrets.

CI/CD Pipeline Secrets

CI/CD pipelines need secrets to deploy, test, and build — but they're also a major attack surface for secret leaks.

GitHub Actions secrets:

# .github/workflows/deploy.yml
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: aws s3 sync ./build s3://my-bucket

GitHub Actions secrets are encrypted and only exposed to the workflow at runtime. They're masked in logs (GitHub replaces them with ***).

GitLab CI/CD variables:

deploy:
  script:
    - aws s3 sync ./build s3://my-bucket
  variables:
    AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

CI/CD secret best practices:

1. Use the platform's built-in secret storage. GitHub Secrets, GitLab Variables (masked/protected), CircleCI Contexts, Bitbucket Pipelines Variables — use these rather than embedding secrets in pipeline configuration files.

2. Scope secrets narrowly. Use environment-specific secrets (production secrets only accessible to the production deployment job, not the testing job).

3. Avoid echoing secrets. Never echo $SECRET in a pipeline script — even if the platform masks it, there are ways secrets can leak (e.g., base64-encoded output, URLs containing the secret).

4. Use OIDC/workload identity instead of long-lived keys. AWS, GCP, and Azure support workload identity federation — your CI/CD pipeline authenticates using short-lived tokens issued by the CI platform, eliminating the need to store cloud credentials as secrets at all.

5. Pin action versions. Use actions/checkout@v4.1.1 (specific version) rather than actions/checkout@v4 (latest). A compromised action at @v4 could exfiltrate secrets.

6. Restrict fork access. On public repositories, secrets are not available to pull requests from forks (by default). Don't change this setting — a malicious fork could create a PR that exfiltrates secrets.

Secret Rotation and Revocation

Secret rotation means regularly replacing secrets with new values. Secret revocation means immediately invalidating a compromised secret.

Why rotation matters:

• Limits the window of exposure if a secret is compromised
• Ensures that former employees' access is automatically revoked
• Meets compliance requirements (PCI DSS, SOC 2)
• Mitigates the risk of secrets that were leaked but not yet discovered

Rotation schedule recommendations:

• API keys and tokens: Every 90 days, or immediately upon suspected compromise
• Database passwords: Every 90 days (or use dynamic secrets with auto-expiration)
• JWT signing keys: Every 90-180 days (use key rotation with overlapping validity)
• Encryption keys: Annual rotation with key versioning (old keys kept for decrypting existing data)
• SSH keys: Annual rotation, or immediately when an employee leaves

Zero-downtime rotation strategy:

1. Generate the new secret
2. Configure the application to accept both old and new secrets simultaneously
3. Deploy the new secret to all application instances
4. Verify all instances are using the new secret
5. Revoke the old secret

For API keys and database credentials, this often means having a brief period where both old and new credentials are valid.

Immediate revocation procedure: When you discover a secret has been compromised:

1. Revoke immediately. Don't wait — disable the key/token/password right now.
2. Generate a new secret and deploy it to the application.
3. Audit access logs for the compromised secret to determine if it was exploited.
4. Investigate the leak source — how was the secret exposed?
5. Document and remediate — fix the process that allowed the leak.

Detection and Prevention Tools

Pre-commit detection (prevents secrets from entering git):

• git-secrets (AWS) — scans commits for AWS keys and configurable patterns
• pre-commit framework with detect-secrets hook — runs before each commit to block secrets
• truffleHog — scans git history for high-entropy strings and known secret patterns
• gitleaks — fast, configurable secret scanner for git repos and CI/CD

Example pre-commit configuration:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

CI/CD scanning:

• GitHub Secret Scanning — automatically detects known secret patterns in public repos (and private repos with GitHub Advanced Security)
• GitLab Secret Detection — SAST scanner that detects secrets in CI/CD pipeline
• GitGuardian — comprehensive secret detection platform for repos and CI/CD

Runtime detection:

• AWS GuardDuty — detects compromised credentials being used from unusual locations
• Cloudflare API Shield — detects and blocks requests using leaked API tokens
• Vault audit logs — alerts on unusual secret access patterns

Secret management checklist:

• [ ] No secrets hardcoded in source code
• [ ] .env files are in .gitignore
• [ ] .env.example committed as a template
• [ ] Production secrets in a dedicated secret manager
• [ ] CI/CD uses platform secret storage (not config files)
• [ ] Pre-commit hooks prevent secret commits
• [ ] Secret rotation schedule established and followed
• [ ] Revocation procedure documented and tested
• [ ] Access logging enabled for all secrets

Generate unique, strong secrets for every service with our password generator — use 32+ character random strings for API keys and JWT secrets.

---

Secret management isn't glamorous, but it's the difference between a routine security posture and a front-page breach. The pattern is always the same: a developer commits a credential, an attacker finds it, and what should have been a minor configuration task becomes a major incident. Adopt the practices in this guide — environment variables for development, a vault for production, pre-commit hooks for prevention, and rotation for resilience — and leaked secrets become a problem your team doesn't have.