Security
Security Model
Prometheus Proxy is designed to be firewall-friendly:
- The agent initiates an outbound gRPC connection to the proxy
- No inbound ports need to be opened on the firewall
- The proxy accepts connections only from registered agents
- Stale agent connections are automatically cleaned up
TLS Encryption
Agents connect to the proxy using gRPC, which supports TLS with or without mutual authentication.
| Mode | Proxy Needs | Agent Needs |
|---|---|---|
| No TLS | Nothing | Nothing |
| TLS (server only) | Server cert + key | CA cert (trust store) |
| Mutual TLS | Server cert + key + CA cert | Client cert + key + CA cert |
See TLS Setup for detailed configuration instructions.
Auth Header Forwarding
When Prometheus scrape configurations include basic_auth or bearer_token, the proxy forwards
the Authorization header to the agent over the gRPC channel. The agent then includes this
header when fetching metrics from the target endpoint.
scrape_configs:
# Bearer token authentication
- job_name: 'app1'
metrics_path: '/app1_metrics'
bearer_token: 'eyJhbGciOiJIUzI1NiIs...'
static_configs:
- targets: ['proxy-host.example.com:8080']
# Basic auth
- job_name: 'app2'
metrics_path: '/app2_metrics'
basic_auth:
username: 'user'
password: 's3cr3t'
static_configs:
- targets: ['proxy-host.example.com:8080']
# No auth
- job_name: 'app3'
metrics_path: '/app3_metrics'
static_configs:
- targets: ['proxy-host.example.com:8080']
Credentials transmitted in plaintext without TLS
Without TLS, the Authorization header is transmitted in plaintext between the proxy
and agent. The proxy logs a warning on the first request that includes an
Authorization header when TLS is not enabled.
Always enable TLS when forwarding authentication headers.
When Prometheus scrape configs include basic_auth or bearer_token,
the proxy forwards the Authorization header to the agent over gRPC.
Without TLS, credentials are transmitted in plaintext!
Protect forwarded credentials with TLS:
java -jar prometheus-proxy.jar \
--cert /path/to/server.crt \
--key /path/to/server.key
java -jar prometheus-agent.jar \
--config myconfig.conf \
--trust /path/to/ca.crt
Scraping HTTPS Endpoints
If the agent needs to scrape HTTPS endpoints with self-signed certificates, you can disable SSL verification:
Disable SSL verification for HTTPS endpoints (development only!):
Via CLI:
java -jar prometheus-agent.jar --trust_all_x509 --config myconfig.conf
Via environment variable:
TRUST_ALL_X509_CERTIFICATES=true
Via config file:
agent.http.enableTrustAllX509Certificates = true
Development only
Only use trust_all_x509 in development or testing environments. In production,
configure proper TLS certificates for your metrics endpoints.