Avoid Docker Hub Rate Limit with Secure Credential Management: Meet sm-docker-credential-helper

Docker Hub’s recent enforcement of pull rate limits—100 pulls per 6 hours for anonymous users and 200 for authenticated free-tier users—has created friction for developers and DevOps teams. This is especially problematic for CI/CD pipelines, where multiple jobs may share the same outbound IP address.

For organisations using self-hosted runners (e.g., GitHub Actions, GitLab CI, Jenkins, etc.) behind a shared NAT gateway, these limits can be hit quickly, even if individual jobs aren’t pulling many images. To avoid hitting those limits and securely manage credentials, the sm-docker-credential-helper offers a secure, centralised solution using AWS Secrets Manager.

What is sm-docker-credential-helper ?

sm-docker-credential-helper is a Docker credential helper that retrieves Docker Hub credentials directly from AWS Secrets Manager, allowing you to:

• Authenticate Docker pulls
• Avoid rate limits
• Store secrets securely
• Share a single set of credentials across environments

Why Is This Critical Now ?

Docker Hub Pull Rate Limits

• 100 pulls/6hr for anonymous users
• 200 pulls/6hr for free-tier authenticated users (per IP address)

If you’re using self-hosted CI runners sharing a NAT IP, Docker sees all requests as coming from one user, regardless of how many machines or containers are pulling. This means:

Even a modest pipeline with multiple stages can hit Docker Hub’s rate limits quickly—causing builds to fail unexpectedly.

With sm-docker-credential-helper, you can authenticate each request using centralised, securely stored credentials, ensuring rate limits are applied per authenticated account rather than shared IP.

How sm-docker-credential-helper Works

Prerequisites

• AWS credentials configured locally or on the runner
• Docker credentials stored in AWS Secrets Manager:

Setup Steps

1. Build the Helper (requires Golang installed locally)

   git clone https://github.com/vhoanguyen/sm-docker-credentials-helper
   cd sm-docker-credentials-helper
   VERSION=1.0.0 make build

2. Install the Binary to your $PATH

   cp ./bin/sm-login-linux-amd64 /usr/local/bin/docker-credential-sm-login

3. Configure Docker Update ~/.docker/config.json:

4. Create CFN Stack Resource

   AWSTemplateFormatVersion: '2010-09-09'
   Resources:
     MySecret:
       Type: AWS::SecretsManager::Secret
       Properties:
         Name: MySecret
         Description: "A sample secret for demonstration purposes"
         SecretString: |
   
         Tags:
           - Key: Environment
             Value: Production
         ResourcePolicy:
           Version: "2012-10-17"

5. Set Required Environment Variables

   export DOCKER_SECRET_NAME=MySecret
   export AWS_PROFILE=your-aws-profile

6. Create Docker Organisation or Personal Token and Update Secret Manager

• https://docs.docker.com/security/for-admins/access-tokens/
  • https://docs.docker.com/security/for-developers/access-tokens/

Perform simple tests

Test 1: Access to Secret Manager Resource

AWS_PROFILE=YOUR_AWS_PROFILE \ 
DOCKER_SECRET_NAME=MySecret \ 
/usr/local/bin/docker-credential-sm-login list

Test 2: Docker Login

echo https://index.docker.io/v1/ | AWS_PROFILE=YOUR_AWS_PROFILE DOCKER_SECRET_NAME=MySecret \
/usr/local/bin/docker-credential-sm-login get

Test 2: Run Hello World Image (check ~/.sm/ for debug logs )

AWS_PROFILE=YOUR_AWS_PROFILE \
DOCKER_SECRET_NAME=MySecret \
docker pull hello-world

Why Use This in self-hosted CI/CD

If you’re running pipelines with tools like:

• GitHub Actions (self-hosted)
• GitLab Runners
• Jenkins
• ArgoCD or custom Kubernetes jobs

…and all your runners exit through one shared IP, you’re highly vulnerable to rate-limiting—even with authenticated users.

By using sm-docker-credential-helper:

• All runners authenticate with valid Docker credentials
• You avoid being rate-limited due to shared IP
• Secrets are centrally stored, easily rotated, and not exposed in plain text

Benefits Recap

✅ Bypass Docker rate limits by authenticating every pull

✅ Avoid pipeline failures caused by NAT-based rate limits

✅ Secure Docker credentials using AWS Secrets Manager

✅ Easily share credentials across multiple machines/runners

✅ No local plaintext config—credentials are pulled on demand

Hoa Nguyen 2025-05-21T15:14:37+10:00