Skip to content

Security

Security Model

Prometheus Proxy is designed to be firewall-friendly:

  • The agent initiates an outbound gRPC connection to the proxy
  • No inbound ports need to be opened on the firewall
  • The proxy accepts connections only from registered agents
  • Stale agent connections are automatically cleaned up

TLS Encryption

Agents connect to the proxy using gRPC, which supports TLS with or without mutual authentication.

Mode Proxy Needs Agent Needs
No TLS Nothing Nothing
TLS (server only) Server cert + key CA cert (trust store)
Mutual TLS Server cert + key + CA cert Client cert + key + CA cert

See TLS Setup for detailed configuration instructions.

Auth Header Forwarding

When Prometheus scrape configurations include basic_auth or bearer_token, the proxy forwards the Authorization header to the agent over the gRPC channel. The agent then includes this header when fetching metrics from the target endpoint.

scrape_configs:
  # Bearer token authentication
  - job_name: 'app1'
    metrics_path: '/app1_metrics'
    bearer_token: 'eyJhbGciOiJIUzI1NiIs...'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

  # Basic auth
  - job_name: 'app2'
    metrics_path: '/app2_metrics'
    basic_auth:
      username: 'user'
      password: 's3cr3t'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

  # No auth
  - job_name: 'app3'
    metrics_path: '/app3_metrics'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

Credentials transmitted in plaintext without TLS

Without TLS, the Authorization header is transmitted in plaintext between the proxy and agent. The proxy logs a warning on the first request that includes an Authorization header when TLS is not enabled.

Always enable TLS when forwarding authentication headers.

When Prometheus scrape configs include basic_auth or bearer_token,
the proxy forwards the Authorization header to the agent over gRPC.

Without TLS, credentials are transmitted in plaintext!

Protect forwarded credentials with TLS:
java -jar prometheus-proxy.jar \
  --cert /path/to/server.crt \
  --key /path/to/server.key

java -jar prometheus-agent.jar \
  --config myconfig.conf \
  --trust /path/to/ca.crt

Scraping HTTPS Endpoints

If the agent needs to scrape HTTPS endpoints with self-signed certificates, you can disable SSL verification:

Disable SSL verification for HTTPS endpoints (development only!):

Via CLI:
java -jar prometheus-agent.jar --trust_all_x509 --config myconfig.conf

Via environment variable:
TRUST_ALL_X509_CERTIFICATES=true

Via config file:
agent.http.enableTrustAllX509Certificates = true

Development only

Only use trust_all_x509 in development or testing environments. In production, configure proper TLS certificates for your metrics endpoints.