Kubernetes

Run Prometheus Proxy on Kubernetes to scrape metrics from services that live inside firewalled or private clusters.

Topology

The two components play different roles, and that determines where each one runs:

The agent runs inside each target cluster, alongside the services it scrapes. It opens an outbound gRPC connection to the proxy, so the target cluster never needs an inbound firewall rule.
The proxy runs centrally, next to Prometheus. Agents dial in to it on the gRPC port (50051), and Prometheus scrapes it on the HTTP port (8080).

flowchart LR
  subgraph monitoring["Monitoring cluster"]
    prom["Prometheus"]
    proxy["Prometheus Proxy<br/>:8080 HTTP / :50051 gRPC"]
    prom -- "scrape :8080" --> proxy
  end

  subgraph target["Target cluster (behind firewall)"]
    agent["Prometheus Agent"]
    app["my-app :9100/metrics"]
    agent -- "scrape in-cluster" --> app
  end

  agent -- "outbound gRPC :50051" --> proxy

The agent always initiates the connection, so a target cluster only needs egress to the proxy's gRPC endpoint — no inbound ports are exposed.

Namespaces in these examples

The manifests below put the proxy in a monitoring namespace and the agent in default. Adjust the namespace fields and the in-cluster DNS names (e.g. prometheus-proxy.monitoring.svc.cluster.local) to match your own layout.

Deploying the Proxy

Deploy the proxy as a Deployment plus a Service. Admin and metrics endpoints are enabled through environment variables so Kubernetes can probe them and Prometheus can scrape them.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-proxy
  namespace: monitoring
  labels:
    app: prometheus-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-proxy
  template:
    metadata:
      labels:
        app: prometheus-proxy
    spec:
      containers:
        - name: proxy
          image: pambrose/prometheus-proxy:4.0.1
          ports:
            - { name: http, containerPort: 8080 }     # Prometheus scrapes here
            - { name: grpc, containerPort: 50051 }     # agents connect here
            - { name: metrics, containerPort: 8082 }
            - { name: admin, containerPort: 8092 }
          env:
            - { name: ADMIN_ENABLED, value: "true" }
            - { name: METRICS_ENABLED, value: "true" }
          livenessProbe:
            httpGet: { path: /ping, port: admin }
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet: { path: /healthcheck, port: admin }
            initialDelaySeconds: 10
            periodSeconds: 15
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-proxy
  namespace: monitoring
  labels:
    app: prometheus-proxy
spec:
  selector:
    app: prometheus-proxy
  ports:
    - { name: http, port: 8080, targetPort: http }
    - { name: grpc, port: 50051, targetPort: grpc }
    - { name: metrics, port: 8082, targetPort: metrics }

The Service above is a ClusterIP, which is enough when Prometheus and the agents live in the same cluster. Prometheus reaches the proxy at prometheus-proxy.monitoring.svc.cluster.local:8080, and same-cluster agents can use the same DNS name for the gRPC port.

Exposing gRPC to remote agents

When agents run in other clusters, they need to reach the proxy's gRPC port from outside. Expose it with a LoadBalancer Service (or an Ingress controller):

apiVersion: v1
kind: Service
metadata:
  name: prometheus-proxy-grpc
  namespace: monitoring
  labels:
    app: prometheus-proxy
spec:
  type: LoadBalancer
  selector:
    app: prometheus-proxy
  ports:
    - { name: grpc, port: 50051, targetPort: grpc }

gRPC needs HTTP/2 end to end

gRPC runs over HTTP/2. A plain LoadBalancer (L4) forwards it transparently. If you put an Ingress in front instead, it must support HTTP/2 / gRPC backends (for example, the NGINX Ingress nginx.ingress.kubernetes.io/backend-protocol: GRPC annotation). Terminate or pass through TLS as described under TLS.

Deploying the Agent

The agent's path mappings live in a ConfigMap. The proxy hostname is supplied separately through the PROXY_HOSTNAME environment variable, so the same config works regardless of where the proxy is exposed.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-agent-config
  namespace: default
data:
  agent.conf: |
    agent {
      pathConfigs: [
        {
          name: "My App metrics"
          path: my_app_metrics
          url: "http://my-app.default.svc.cluster.local:8080/metrics"
        }
      ]
    }

Each entry in pathConfigs maps a proxy path (what Prometheus scrapes) to the in-cluster url the agent fetches from. Point url at a Kubernetes Service DNS name such as http://my-app.default.svc.cluster.local:8080/metrics.

Updating targets without restarting the agent

With pathConfigs alone, changing the target list means updating the ConfigMap and restarting the agent pod — which interrupts scraping for every path the agent serves, not just the one that changed. Dynamic target discovery removes the restart: mount a second ConfigMap holding a paths list and point agent.discovery at it. The agent re-reads the file on an interval and reconciles — targets added to the ConfigMap start scraping within the interval, removed ones unregister, and untouched paths never stop.

apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-targets
data:
  targets.conf: |
    paths = [
      { name = "app1", path = "app1_metrics", url = "http://app1.default.svc.cluster.local:9090/metrics" }
      { name = "app2", path = "app2_metrics", url = "http://app2.default.svc.cluster.local:9090/metrics" }
    ]

Mount it into the agent pod (e.g. at /etc/prometheus-proxy/targets.conf) and enable discovery in the agent's main config:

agent {
  discovery {
    enabled = true
    file.path = "/etc/prometheus-proxy/targets.conf"
    reconcileIntervalSecs = 30
  }
}

Discovery polls the file rather than using inotify precisely because of how Kubernetes delivers ConfigMap updates (atomic symlink swaps that file watchers often miss). Note that a kubelet can take up to a minute to propagate a ConfigMap change to the mounted volume — that delay is Kubernetes's, and comes on top of the reconcile interval. A malformed or half-propagated file is skipped, keeping the last-known-good target set, so a bad edit never tears down working targets. Keep discovery targets in their own ConfigMap, separate from the agent's main config: whatever automation writes the target list then can't touch ports, TLS, or the proxy address.

Standalone Deployment

The common pattern is a single agent Deployment per cluster that scrapes several in-cluster endpoints:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-agent
  namespace: default
  labels:
    app: prometheus-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-agent
  template:
    metadata:
      labels:
        app: prometheus-agent
    spec:
      containers:
        - name: agent
          image: pambrose/prometheus-agent:4.0.1
          ports:
            - { name: metrics, containerPort: 8083 }
            - { name: admin, containerPort: 8093 }
          env:
            - name: PROXY_HOSTNAME
              value: "proxy.example.com:50051"   # the proxy's externally reachable address
            - { name: AGENT_CONFIG, value: /app/agent.conf }
            - { name: ADMIN_ENABLED, value: "true" }
            - { name: METRICS_ENABLED, value: "true" }
          volumeMounts:
            - { name: config, mountPath: /app/agent.conf, subPath: agent.conf }
          livenessProbe:
            httpGet: { path: /ping, port: admin }
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet: { path: /healthcheck, port: admin }
            initialDelaySeconds: 10
            periodSeconds: 15
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }
      volumes:
        - name: config
          configMap:
            name: prometheus-agent-config

Set PROXY_HOSTNAME to the proxy's externally reachable address (the LoadBalancer host from above), including the gRPC port if it is not the default 50051.

Sidecar

Alternatively, run the agent as a sidecar next to a single application pod and scrape it over localhost:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app:latest
          ports:
            - { name: metrics, containerPort: 9100 }
        - name: prometheus-agent
          image: pambrose/prometheus-agent:4.0.1
          env:
            - { name: PROXY_HOSTNAME, value: "proxy.example.com:50051" }
            - { name: AGENT_CONFIG, value: /app/agent.conf }
          volumeMounts:
            - { name: agent-config, mountPath: /app/agent.conf, subPath: agent.conf }
      volumes:
        - name: agent-config
          configMap:
            name: prometheus-agent-config   # agent.conf url points at http://localhost:9100/metrics

With the sidecar pattern the agent's url should point at http://localhost:9100/metrics rather than a Service DNS name.

Native sidecar containers

On Kubernetes 1.29+ you can run the agent as a native sidecar — an entry in initContainers with restartPolicy: Always — so it starts before the app container and is tracked by the pod's lifecycle. The plain multi-container form above works on all versions.

Prometheus Integration

Prometheus scrapes the proxy, using the agent path as the metrics_path. The path must match the path value from the agent ConfigMap.

Scrape configServiceMonitor (Operator)

Add a job to prometheus.yml pointing at the proxy Service:

scrape_configs:
  - job_name: 'my-app-via-proxy'
    metrics_path: '/my_app_metrics'
    static_configs:
      - targets: ['prometheus-proxy.monitoring.svc.cluster.local:8080']

If you run the Prometheus Operator (e.g. kube-prometheus-stack), use a ServiceMonitor instead:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-proxy
  namespace: monitoring
  labels:
    release: kube-prometheus-stack   # must match your Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: prometheus-proxy
  endpoints:
    # Add one endpoint per registered proxy path.
    - port: http
      path: /my_app_metrics
      interval: 30s

The release label must match your Prometheus instance's serviceMonitorSelector. Add one entry under endpoints for each registered proxy path.

Dynamic path discovery

Rather than listing every path by hand, enable the proxy's service-discovery endpoint and let Prometheus pull the target list with http_sd_config. See Service Discovery. (This is the Prometheus-facing half; the agent-facing half — picking up target changes inside the cluster without a restart — is dynamic target discovery above. The two compose: discovered paths appear in the service-discovery output like any other. Caveat: if you run an HA proxy pair, scrape with static_config instead — a standby proxy returns an empty discovery list, which http_sd_config treats as "delete these targets".)

TLS

To secure the proxy ↔ agent gRPC channel, bundle your certificates into a Secret and mount it into the proxy and agent containers:

# Bundle your certificate files into a Secret:
kubectl create secret generic prometheus-proxy-tls \
  --namespace monitoring \
  --from-file=cert.pem \
  --from-file=key.pem \
  --from-file=ca.pem

# Container spec excerpt — point the config at the mounted certs:
          env:
            - { name: PROXY_CONFIG, value: /app/tls.conf }
          volumeMounts:
            - { name: tls, mountPath: /app/certs, readOnly: true }
      volumes:
        - name: tls
          secret:
            secretName: prometheus-proxy-tls

The mounted config (tls.conf) references the certificate paths under /app/certs. See TLS Setup for the full configuration, including mutual authentication.

Health Probes & Resources

The proxy and agent Deployments above wire Kubernetes probes to the admin endpoints:

Probe	Endpoint	Meaning
`livenessProbe`	`GET /ping`	Returns `pong` while the process is up
`readinessProbe`	`GET /healthcheck`	Returns health status; non-200 if unhealthy

Both endpoints are served on the admin port (8092 for the proxy, 8093 for the agent), which requires ADMIN_ENABLED=true. The manifests also set conservative CPU/memory requests and limits — tune them to your scrape volume and payload sizes.

No official Helm chart

Prometheus Proxy does not currently publish a Helm chart. The raw manifests on this page are the supported way to deploy on Kubernetes; adapt them to Kustomize or your own chart as needed.

Next Steps

Agent Configuration

Path configs, HTTP client tuning, and scrape options

Agent Configuration
Security & TLS

Secure the proxy-agent channel with TLS and mutual authentication

Security
Service Discovery

Let Prometheus discover proxied targets dynamically

Service Discovery
:material-activity:{ .lg .middle } Monitoring

Scrape the proxy's and agent's own operational metrics

Monitoring