Technical Tutorials

Introduction

In the realm of modern infrastructure, Prometheus has emerged as the de facto standard for metrics collection and alerting. However, many organizations still rely on "legacy" systems—monolithic applications, mainframes, or proprietary software—that lack native support for the Prometheus exposition format. These systems often speak in SNMP traps, proprietary logs, or simple HTTP responses that are incompatible with modern observability stacks. Attempting to scrape these systems directly is often impossible due to security constraints or protocol limitations. The solution lies in building custom Prometheus exporters. An exporter is a lightweight service that sits between your legacy infrastructure and Prometheus, translating old data formats into the standardized metrics that Prometheus understands. This post will guide you through the architecture, implementation, and best practices for creating these bridges.

Understanding the Exporter Architecture

Before diving into code, it is crucial to understand the architectural pattern. A custom exporter acts as an adapter. It typically performs three core functions: 1. **Discovery/Connection:** It connects to the legacy system using an available protocol (e.g., JDBC for databases, SSH for servers, or HTTP APIs). 2. **Translation:** It extracts raw data and maps it to Prometheus metric types (Counter, Gauge, Histogram, or Summary). 3. **Exposition:** It exposes a `/metrics` endpoint in the plaintext Prometheus format, ready for scraping. The exporter itself should be stateless regarding the data it collects but stateful in its connection to the legacy system. It is best practice to keep the exporter separate from the legacy application to avoid performance overhead on the critical production system.

Choosing the Right Language

While you can build exporters in any language, Go (Golang) is the industry standard for Prometheus exporters. The official `prometheus/client_golang` library provides robust, low-level primitives that ensure high performance and minimal resource consumption. Python is also a viable option for simpler scripts, but Go offers better concurrency handling and static binary compilation, which simplifies deployment.

Practical Example: Building a Go Exporter

Let’s assume we have a legacy internal service that exposes a single number via a simple HTTP GET request at `http://legacy-system:8080/status`. We want to expose this as a Prometheus Gauge metric. First, ensure you have the necessary dependencies: ```bash go get github.com/prometheus/client_golang/prometheus go get github.com/prometheus/client_golang/prometheus/promhttp ``` Here is a complete, minimal implementation of a custom exporter:

package main

import (
	"io/ioutil"
	"log"
	"net/http"
	"strings"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// legacyStatusGauge is the metric we want to expose
var legacyStatusGauge = prometheus.NewGauge(
	prometheus.GaugeOpts{
		Name: "legacy_system_status_code",
		Help: "Current status code returned by the legacy system",
	},
)

// Collector implements the prometheus.Collector interface
type LegacyCollector struct{}

// Describe sends the super-set of all possible descriptors of metrics
func (c *LegacyCollector) Describe(ch chan<- *prometheus.Desc) {
	ch <- legacyStatusGauge.Desc()
}

// Collect is called by the Prometheus registry when collecting metrics
func (c *LegacyCollector) Collect(ch chan<- prometheus.Metric) {
	// Fetch data from the legacy system
	resp, err := http.Get("http://legacy-system:8080/status")
	if err != nil {
		log.Printf("Error fetching status: %v", err)
		// Do not return if the system is down; let Prometheus handle connection errors
		return
	}
	defer resp.Body.Close()

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		log.Printf("Error reading body: %v", err)
		return
	}

	// Parse the legacy response (assuming it's a number string)
	statusValue := 0.0
	if strings.TrimSpace(string(body)) != "" {
		// In a real scenario, use proper parsing logic
		statusValue = 1.0 
	}

	// Set the gauge value
	legacyStatusGauge.Set(statusValue)
	
	// Send the metric to Prometheus
	ch <- legacyStatusGauge
}

func main() {
	// Register the collector
	prometheus.MustRegister(&LegacyCollector{})

	// Expose the /metrics endpoint
	http.Handle("/metrics", promhttp.Handler())
	
	log.Println("Starting exporter on :9100")
	log.Fatal(http.ListenAndServe(":9100", nil))
}

Best Practices for Production

When moving from a prototype to production, consider the following optimizations: * **Error Handling:** Never let a collection error panic the exporter. If the legacy system is unreachable, the exporter should continue running and expose zero values or previous known values if appropriate, allowing Prometheus to generate alerts based on staleness. * **Authentication:** If your legacy system requires authentication, ensure credentials are handled securely via environment variables or secret management tools, never hardcoded. * **Metrics Naming:** Follow Prometheus naming conventions. Use underscores, not camelCase, and include the unit in the help text. Avoid "informational" labels that have high cardinality (like unique user IDs), as this can overwhelm the Prometheus time-series database. * **Health Checks:** Implement a `/health` endpoint alongside `/metrics` to allow container orchestrators (like Kubernetes) to verify the exporter is alive and able to connect to the backend.

Conclusion

Building custom Prometheus exporters is an essential skill for DevOps engineers managing hybrid or legacy-heavy environments. By decoupling the monitoring logic from the legacy application, you gain observability without compromising stability. While the initial setup requires effort, the long-term benefits of unified alerting, visualization, and data correlation across your entire infrastructure are invaluable. Start small, iterate on your metric definitions, and ensure your exporters are as lightweight and robust as the systems they monitor.