Agent

Prometheus Agent that connects to a Proxy to enable metrics scraping across firewalls.

The Agent is the component that runs inside the firewall alongside the services being monitored. It establishes an outbound connection to the Proxy, registers the metrics endpoints it can scrape, and responds to scrape requests by fetching metrics from the actual endpoints and returning them to the Proxy.

Architecture

The Agent maintains a persistent gRPC connection to the Proxy and handles:

• Connection Management: Establishes and maintains connection to proxy with automatic reconnection

• Path Registration: Registers configured metrics endpoints with the proxy

• Scrape Processing: Responds to scrape requests by fetching metrics from local endpoints

• Concurrent Scraping: Handles multiple concurrent scrape requests with configurable limits

• Heartbeat: Sends periodic heartbeat messages to maintain connection health

• Metrics Collection: Exposes operational metrics about the agent itself

Configuration

Agents are configured via HOCON configuration files that specify:

• Proxy connection details (hostname, port, TLS settings)

• Path configurations (endpoints to scrape and their metadata)

• Operational parameters (timeouts, concurrency limits, retry policies)

• Admin and metrics endpoints for monitoring the agent itself

Usage Examples

Basic Usage

val agent = Agent(AgentOptions(args))
agent.startSync()

Embedded Usage

val agentInfo = Agent.startAsyncAgent("config.conf", true)
println("Started agent: ${agentInfo.agentName}")

With Custom Initialization

val agent = Agent(options) {
    // Custom initialization logic
    logger.info { "Agent initialized with custom settings" }
}
agent.startSync()

Connection Lifecycle

1. Startup: Agent reads configuration and initializes services

2. Connection: Establishes gRPC connection to proxy

3. Registration: Registers configured paths with proxy

4. Operation: Processes scrape requests and sends heartbeats

5. Reconnection: Automatically reconnects on connection failures

6. Shutdown: Gracefully closes connections and cleans up resources

Error Handling

The Agent includes robust error handling for:

• Network connectivity issues with exponential backoff

• Individual scrape failures without affecting other scrapes

• Configuration errors with detailed error messages

• Resource exhaustion with proper cleanup

Parameters

See also