NGINX Prometheus Exporter

NGINX Prometheus exporter makes it possible to monitor NGINX or NGINX Plus using Prometheus.

Table of Contents

• Overview
• Getting Started
  • A Note about NGINX Ingress Controller
  • Prerequisites
  • Running the Exporter in a Docker Container
  • Running the Exporter Binary
• Usage
  • Command-line Arguments
• Exported Metrics
  • Common metrics
  • Metrics for NGINX OSS
    • Stub status metrics
  • Metrics for NGINX Plus
    • Connections
    • HTTP
    • SSL
    • HTTP Server Zones
    • Stream Server Zones
    • HTTP Upstreams
    • Stream Upstreams
    • Stream Zone Sync
    • Location Zones
    • Resolver
    • HTTP Requests Rate Limiting
    • HTTP Connections Limiting
    • Stream Connections Limiting
    • Cache
    • Worker
• Troubleshooting
• Releases
  • Docker images
  • Binaries
  • Homebrew
  • Snap
  • Scoop
  • Nix
• Building the Exporter
  • Building the Docker Image
  • Building the Binary
• Grafana Dashboard
• SBOM (Software Bill of Materials)
  • Binaries
  • Docker Image
• Provenance
• Contacts
• Contributing
• Support
• License

Overview

NGINX exposes a handful of metrics via the stub_status page. NGINX Plus provides a richer set of metrics via the API and the monitoring dashboard. NGINX Prometheus exporter fetches the metrics from a single NGINX or NGINX Plus, converts the metrics into appropriate Prometheus metrics types and finally exposes them via an HTTP server to be collected by Prometheus.

Getting Started

In this section, we show how to quickly run NGINX Prometheus Exporter for NGINX or NGINX Plus.

A Note about NGINX Ingress Controller

If you’d like to use the NGINX Prometheus Exporter with NGINX Ingress Controller for Kubernetes, see this doc for the installation instructions.

Prerequisites

We assume that you have already installed Prometheus and NGINX or NGINX Plus. Additionally, you need to:

• Expose the built-in metrics in NGINX/NGINX Plus:
  • For NGINX, expose the stub_status page at /stub_status on port 8080.
  • For NGINX Plus, expose the API at /api on port 8080.
• Configure Prometheus to scrape metrics from the server with the exporter. Note that the default scrape port of the exporter is 9113 and the default metrics path -- /metrics.

Running the Exporter in a Docker Container

To start the exporter we use the docker run command.

• To export NGINX metrics, run:

  docker run -p 9113:9113 nginx/nginx-prometheus-exporter:1.4.0 --nginx.scrape-uri=http://<nginx>:8080/stub_status

  where <nginx> is the IP address/DNS name, through which NGINX is available.

• To export NGINX Plus metrics, run:

  docker run -p 9113:9113 nginx/nginx-prometheus-exporter:1.4.0 --nginx.plus --nginx.scrape-uri=http://<nginx-plus>:8080/api

  where <nginx-plus> is the IP address/DNS name, through which NGINX Plus is available.

Running the Exporter Binary

• To export NGINX metrics, run:

  nginx-prometheus-exporter --nginx.scrape-uri=http://<nginx>:8080/stub_status

  where <nginx> is the IP address/DNS name, through which NGINX is available.

• To export NGINX Plus metrics:

  nginx-prometheus-exporter --nginx.plus --nginx.scrape-uri=http://<nginx-plus>:8080/api

  where <nginx-plus> is the IP address/DNS name, through which NGINX Plus is available.

• To scrape NGINX metrics with unix domain sockets, run:

  nginx-prometheus-exporter --nginx.scrape-uri=unix:<nginx>:/stub_status

  where <nginx> is the path to unix domain socket, through which NGINX stub status is available.

Note. The nginx-prometheus-exporter is not a daemon. To run the exporter as a system service (daemon), you can follow the example in examples/systemd. Alternatively, you can run the exporter in a Docker container.

Usage

Command-line Arguments

usage: nginx-prometheus-exporter [<flags>]


Flags:
  -h, --[no-]help                Show context-sensitive help (also try --help-long and --help-man).
      --[no-]web.systemd-socket  Use systemd socket activation listeners instead
                                 of port listeners (Linux only). ($SYSTEMD_SOCKET)
      --web.listen-address=:9113 ...
                                 Addresses on which to expose metrics and web interface. Repeatable for multiple addresses. ($LISTEN_ADDRESS)
      --web.config.file=""       Path to configuration file that can enable TLS or authentication. See: https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md ($CONFIG_FILE)
      --web.telemetry-path="/metrics"
                                 Path under which to expose metrics. ($TELEMETRY_PATH)
      --[no-]nginx.plus          Start the exporter for NGINX Plus. By default, the exporter is started for NGINX. ($NGINX_PLUS)
      --nginx.scrape-uri=http://127.0.0.1:8080/stub_status ...
                                 A URI or unix domain socket path for scraping NGINX or NGINX Plus metrics. For NGINX, the stub_status page must be available through the URI. For NGINX Plus -- the API. Repeatable for multiple URIs. ($SCRAPE_URI)
      --[no-]nginx.ssl-verify    Perform SSL certificate verification. ($SSL_VERIFY)
      --nginx.ssl-ca-cert=""     Path to the PEM encoded CA certificate file used to validate the servers SSL certificate. ($SSL_CA_CERT)
      --nginx.ssl-client-cert=""
                                 Path to the PEM encoded client certificate file to use when connecting to the server. ($SSL_CLIENT_CERT)
      --nginx.ssl-client-key=""  Path to the PEM encoded client certificate key file to use when connecting to the server. ($SSL_CLIENT_KEY)
      --nginx.timeout=5s         A timeout for scraping metrics from NGINX or NGINX Plus. ($TIMEOUT)
      --prometheus.const-label=PROMETHEUS.CONST-LABEL ...
                                 Label that will be used in every metric. Format is label=value. It can be repeated multiple times. ($CONST_LABELS)
      --log.level=info           Only log messages with the given severity or above. One of: [debug, info, warn, error]
      --log.format=logfmt        Output format of log messages. One of: [logfmt, json]
      --[no-]version             Show application version.

Exported Metrics

Common metrics

Name	Type	Description	Labels
nginx_exporter_build_info	Gauge	Shows the exporter build information.	branch, goarch, goos, goversion, revision, tags and version
promhttp_metric_handler_requests_total	Counter	Total number of scrapes by HTTP status code.	code (the HTTP status code)
promhttp_metric_handler_requests_in_flight	Gauge	Current number of scrapes being served.	[]
go_*	Multiple	Go runtime metrics.	[]

Metrics for NGINX OSS

Name	Type	Description	Labels
nginx_up	Gauge	Shows the status of the last metric scrape: 1 for a successful scrape and 0 for a failed one	[]

Stub status metrics

Name	Type	Description	Labels
nginx_connections_accepted	Counter	Accepted client connections.	[]
nginx_connections_active	Gauge	Active client connections.	[]
nginx_connections_handled	Counter	Handled client connections.	[]
nginx_connections_reading	Gauge	Connections where NGINX is reading the request header.	[]
nginx_connections_waiting	Gauge	Idle client connections.	[]
nginx_connections_writing	Gauge	Connections where NGINX is writing the response back to the client.	[]
nginx_http_requests_total	Counter	Total http requests.	[]

Metrics for NGINX Plus

Name	Type	Description	Labels
nginxplus_up	Gauge	Shows the status of the last metric scrape: 1 for a successful scrape and 0 for a failed one	[]

Connections

Name	Type	Description	Labels
nginxplus_connections_accepted	Counter	Accepted client connections	[]
nginxplus_connections_active	Gauge	Active client connections	[]
nginxplus_connections_dropped	Counter	Dropped client connections dropped	[]
nginxplus_connections_idle	Gauge	Idle client connections	[]

HTTP

Name	Type	Description	Labels
nginxplus_http_requests_total	Counter	Total http requests	[]
nginxplus_http_requests_current	Gauge	Current http requests	[]

SSL

Name	Type	Description	Labels
nginxplus_ssl_handshakes	Counter	Successful SSL handshakes	[]
nginxplus_ssl_handshakes_failed	Counter	Failed SSL handshakes	[]
nginxplus_ssl_session_reuses	Counter	Session reuses during SSL handshake	[]

HTTP Server Zones

Name	Type	Description	Labels
nginxplus_server_zone_processing	Gauge	Client requests that are currently being processed	server_zone
nginxplus_server_zone_requests	Counter	Total client requests	server_zone
nginxplus_server_zone_responses	Counter	Total responses sent to clients	code (the response status code. The values are: 1xx, 2xx, 3xx, 4xx and 5xx), server_zone
nginxplus_server_zone_responses_codes	Counter	Total responses sent to clients by code	code (the response status code. The possible values are here), server_zone
nginxplus_server_zone_discarded	Counter	Requests completed without sending a response	server_zone
nginxplus_server_zone_received	Counter	Bytes received from clients	server_zone
nginxplus_server_zone_sent	Counter	Bytes sent to clients	server_zone
nginxplus_server_ssl_handshakes	Counter	Successful SSL handshakes	server_zone
nginxplus_server_ssl_handshakes_failed	Counter	Failed SSL handshakes	server_zone
nginxplus_server_ssl_session_reuses	Counter	Session reuses during SSL handshake	server_zone

Stream Server Zones

Name	Type	Description	Labels
nginxplus_stream_server_zone_processing	Gauge	Client connections that are currently being processed	server_zone
nginxplus_stream_server_zone_connections	Counter	Total connections	server_zone
nginxplus_stream_server_zone_sessions	Counter	Total sessions completed	code (the response status code. The values are: 2xx, 4xx, and 5xx), server_zone
nginxplus_stream_server_zone_discarded	Counter	Connections completed without creating a session	server_zone
nginxplus_stream_server_zone_received	Counter	Bytes received from clients	server_zone
nginxplus_stream_server_zone_sent	Counter	Bytes sent to clients	server_zone
nginxplus_stream_server_ssl_handshakes	Counter	Successful SSL handshakes	server_zone
nginxplus_stream_server_ssl_handshakes_failed	Counter	Failed SSL handshakes	server_zone
nginxplus_stream_server_ssl_session_reuses	Counter	Session reuses during SSL handshake	server_zone

HTTP Upstreams

Note: for the state metric, the string values are converted to float64 using the following rule: "up" -> 1.0, "draining" -> 2.0, "down" -> 3.0, "unavail" –> 4.0, "checking" –> 5.0, "unhealthy" -> 6.0.

Name	Type	Description	Labels
nginxplus_upstream_server_state	Gauge	Current state	server, upstream
nginxplus_upstream_server_active	Gauge	Active connections	server, upstream
nginxplus_upstream_server_limit	Gauge	Limit for connections which corresponds to the max_conns parameter of the upstream server. Zero value means there is no limit	server, upstream
nginxplus_upstream_server_requests	Counter	Total client requests	server, upstream
nginxplus_upstream_server_responses	Counter	Total responses sent to clients	code (the response status code. The values are: 1xx, 2xx, 3xx, 4xx and 5xx), server, upstream
nginxplus_upstream_server_responses_codes	Counter	Total responses sent to clients by code	code (the response status code. The possible values are here), server, upstream
nginxplus_upstream_server_sent`	Counter	Bytes sent to this server	server, upstream
nginxplus_upstream_server_received	Counter	Bytes received to this server	server, upstream
nginxplus_upstream_server_fails	Counter	Number of unsuccessful attempts to communicate with the server	server, upstream
nginxplus_upstream_server_unavail	Counter	How many times the server became unavailable for client requests (state 'unavail') due to the number of unsuccessful attempts reaching the max_fails threshold	server, upstream
nginxplus_upstream_server_header_time	Gauge	Average time to get the response header from the server	server, upstream
nginxplus_upstream_server_response_time	Gauge	Average time to get the full response from the server	server, upstream
nginxplus_upstream_server_health_checks_checks	Counter	Total health check requests	server, upstream
nginxplus_upstream_server_health_checks_fails	Counter	Failed health checks	server, upstream
nginxplus_upstream_server_health_checks_unhealthy	Counter	How many times the server became unhealthy (state 'unhealthy')	server, upstream
nginxplus_upstream_server_ssl_handshakes	Counter	Successful SSL handshakes	server, upstream
nginxplus_upstream_server_ssl_handshakes_failed	Counter	Failed SSL handshakes	server, upstream
nginxplus_upstream_server_ssl_session_reuses	Counter	Session reuses during SSL handshake	server, upstream
nginxplus_upstream_keepalive	Gauge	Idle keepalive connections	upstream
nginxplus_upstream_zombies	Gauge	Servers removed from the group but still processing active client requests	upstream

Stream Upstreams

Note: for the state metric, the string values are converted to float64 using the following rule: "up" -> 1.0, "down" -> 3.0, "unavail" –> 4.0, "checking" –> 5.0, "unhealthy" -> 6.0.

Name	Type	Description	Labels
nginxplus_stream_upstream_server_state	Gauge	Current state	server, upstream
nginxplus_stream_upstream_server_active	Gauge	Active connections	server , upstream
nginxplus_stream_upstream_server_limit	Gauge	Limit for connections which corresponds to the max_conns parameter of the upstream server. Zero value means there is no limit	server , upstream
nginxplus_stream_upstream_server_connections	Counter	Total number of client connections forwarded to this server	server, upstream
nginxplus_stream_upstream_server_connect_time	Gauge	Average time to connect to the upstream server	server, upstream
nginxplus_stream_upstream_server_first_byte_time	Gauge	Average time to receive the first byte of data	server, upstream
nginxplus_stream_upstream_server_response_time	Gauge	Average time to receive the last byte of data	server, upstream
nginxplus_stream_upstream_server_sent	Counter	Bytes sent to this server	server, upstream
nginxplus_stream_upstream_server_received	Counter	Bytes received from this server	server, upstream
nginxplus_stream_upstream_server_fails	Counter	Number of unsuccessful attempts to communicate with the server	server, upstream
nginxplus_stream_upstream_server_unavail	Counter	How many times the server became unavailable for client connections (state 'unavail') due to the number of unsuccessful attempts reaching the max_fails threshold	server, upstream
nginxplus_stream_upstream_server_health_checks_checks	Counter	Total health check requests	server, upstream
nginxplus_stream_upstream_server_health_checks_fails	Counter	Failed health checks	server, upstream
nginxplus_stream_upstream_server_health_checks_unhealthy	Counter	How many times the server became unhealthy (state 'unhealthy')	server, upstream
nginxplus_stream_upstream_server_ssl_handshakes	Counter	Successful SSL handshakes	server, upstream
nginxplus_stream_upstream_server_ssl_handshakes_failed	Counter	Failed SSL handshakes	server, upstream
nginxplus_stream_upstream_server_ssl_session_reuses	Counter	Session reuses during SSL handshake	server, upstream
nginxplus_stream_upstream_zombies	Gauge	Servers removed from the group but still processing active client connections	upstream

Stream Zone Sync

Name	Type	Description	Labels
nginxplus_stream_zone_sync_zone_records_pending	Gauge	The number of records that need to be sent to the cluster	zone
nginxplus_stream_zone_sync_zone_records_total	Gauge	The total number of records stored in the shared memory zone	zone
nginxplus_stream_zone_sync_zone_bytes_in	Counter	Bytes received by this node	[]
nginxplus_stream_zone_sync_zone_bytes_out	Counter	Bytes sent by this node	[]
nginxplus_stream_zone_sync_zone_msgs_in	Counter	Total messages received by this node	[]
nginxplus_stream_zone_sync_zone_msgs_out	Counter	Total messages sent by this node	[]
nginxplus_stream_zone_sync_zone_nodes_online	Gauge	Number of peers this node is connected to	[]

Location Zones

Name	Type	Description	Labels
nginxplus_location_zone_requests	Counter	Total client requests	location_zone
nginxplus_location_zone_responses	Counter	Total responses sent to clients	code (the response status code. The values are: 1xx, 2xx, 3xx, 4xx and 5xx), location_zone
nginxplus_location_zone_responses_codes	Counter	Total responses sent to clients by code	code (the response status code. The possible values are here), location_zone
nginxplus_location_zone_discarded	Counter	Requests completed without sending a response	location_zone
nginxplus_location_zone_received	Counter	Bytes received from clients	location_zone
nginxplus_location_zone_sent	Counter	Bytes sent to clients	location_zone

Resolver

Name	Type	Description	Labels
nginxplus_resolver_name	Counter	Total requests to resolve names to addresses	resolver
nginxplus_resolver_srv	Counter	Total requests to resolve SRV records	resolver
nginxplus_resolver_addr	Counter	Total requests to resolve addresses to names	resolver
nginxplus_resolver_noerror	Counter	Total number of successful responses	resolver
nginxplus_resolver_formerr	Counter	Total number of FORMERR responses	resolver
nginxplus_resolver_servfail	Counter	Total number of SERVFAIL responses	resolver
nginxplus_resolver_nxdomain	Counter	Total number of NXDOMAIN responses	resolver
nginxplus_resolver_notimp	Counter	Total number of NOTIMP responses	resolver
nginxplus_resolver_refused	Counter	Total number of REFUSED responses	resolver
nginxplus_resolver_timedout	Counter	Total number of timed out request	resolver
nginxplus_resolver_unknown	Counter	Total requests completed with an unknown error	resolver

HTTP Requests Rate Limiting

Name	Type	Description	Labels
nginxplus_limit_request_passed	Counter	Total number of requests that were neither limited nor accounted as limited	zone
nginxplus_limit_request_rejected	Counter	Total number of requests that were rejected	zone
nginxplus_limit_request_delayed	Counter	Total number of requests that were delayed	zone
nginxplus_limit_request_rejected_dry_run	Counter	Total number of requests accounted as rejected in the dry run mode	zone
nginxplus_limit_request_delayed_dry_run	Counter	Total number of requests accounted as delayed in the dry run mode	zone

HTTP Connections Limiting

Name	Type	Description	Labels
nginxplus_limit_connection_passed	Counter	Total number of connections that were neither limited nor accounted as limited	zone
nginxplus_limit_connection_rejected	Counter	Total number of connections that were rejected	zone
nginxplus_limit_connection_rejected_dry_run	Counter	Total number of connections accounted as rejected in the dry run mode	zone

Stream Connections Limiting

Name	Type	Description	Labels
nginxplus_stream_limit_connection_passed	Counter	Total number of connections that were neither limited nor accounted as limited	zone
nginxplus_stream_limit_connection_rejected	Counter	Total number of connections that were rejected	zone
nginxplus_stream_limit_connection_rejected_dry_run	Counter	Total number of connections accounted as rejected in the dry run mode	zone

Cache

Name	Type	Description	Labels
nginxplus_cache_size	Gauge	Total size of the cache	cache
nginxplus_cache_max_size	Gauge	Maximum size of the cache	cache
nginxplus_cache_cold	Gauge	Is the cache considered cold	cache
nginxplus_cache_hit_responses	Counter	Total number of cache hits	cache
nginxplus_cache_hit_bytes	Counter	Total number of bytes returned from cache hits	cache
nginxplus_cache_stale_responses	Counter	Total number of stale cache hits	cache
nginxplus_cache_stale_bytes	Counter	Total number of bytes returned from stale cache hits	cache
nginxplus_cache_updating_responses	Counter	Total number of cache hits while cache is updating	cache
nginxplus_cache_updating_bytes	Counter	Total number of bytes returned from cache while cache is updating	cache
nginxplus_cache_revalidated_responses	Counter	Total number of cache revalidations	cache
nginxplus_cache_revalidated_bytes	Counter	Total number of bytes returned from cache revalidations	cache
nginxplus_cache_miss_responses	Counter	Total number of cache misses	cache
nginxplus_cache_miss_bytes	Counter	Total number of bytes returned from cache misses	cache
nginxplus_cache_expired_responses	Counter	Total number of cache hits with expired TTL	cache
nginxplus_cache_expired_bytes	Counter	Total number of bytes returned from cache hits with expired TTL	cache
nginxplus_cache_expired_responses_written	Counter	Total number of cache hits with expired TTL written to cache	cache
nginxplus_cache_expired_bytes_written	Counter	Total number of bytes written to cache from cache hits with expired TTL	cache
nginxplus_cache_bypass_responses	Counter	Total number of cache bypasses	cache
nginxplus_cache_bypass_bytes	Counter	Total number of bytes returned from cache bypasses	cache
nginxplus_cache_bypass_responses_written	Counter	Total number of cache bypasses written to cache	cache
nginxplus_cache_bypass_bytes_written	Counter	Total number of bytes written to cache from cache bypasses	cache

Worker

Name	Type	Description	Labels
nginxplus_worker_connection_accepted	Counter	The total number of accepted client connections	id, pid
nginxplus_worker_connection_dropped	Counter	The total number of dropped client connections	id, pid
nginxplus_worker_connection_active	Gauge	The current number of active client connections	id, pid
nginxplus_worker_connection_idle	Gauge	The current number of idle client connection	id, pid
nginxplus_worker_http_requests_total	Counter	The total number of client requests received	id, pid
nginxplus_worker_http_requests_current	Gauge	The current number of client requests that are currently being processed	id, pid

Connect to the /metrics page of the running exporter to see the complete list of metrics along with their descriptions. Note: to see server zones related metrics you must configure status zones and to see upstream related metrics you must configure upstreams with a shared memory zone.

Troubleshooting

The exporter logs errors to the standard output. When using Docker, if the exporter doesn’t work as expected, check its logs using docker logs command.

Releases

Docker images

We publish the Docker image on DockerHub, GitHub Container, Amazon ECR Public Gallery and Quay.io.

As an alternative, you can choose the edge version built from the latest commit from the main branch. The edge version is useful for experimenting with new features that are not yet published in a stable release.

Binaries

We publish the binaries for multiple Operating Systems and architectures on the GitHub releases page.

Homebrew

You can add the NGINX homebrew tap with

brew tap nginxinc/tap

and then install the formula with

brew install nginx-prometheus-exporter

Snap

You can install the NGINX Prometheus Exporter from the Snap Store.

snap install nginx-prometheus-exporter

Scoop

You can add the NGINX Scoop bucket with

scoop bucket add nginx https://github.com/nginxinc/scoop-bucket.git

and then install the package with

scoop install nginx-prometheus-exporter

Nix

First include NUR in your packageOverrides as explained in the NUR documentation.

Then you can use the exporter with the following command:

nix-shell --packages nur.repos.nginx.nginx-prometheus-exporter

or install it with:

nix-env -f '<nixpkgs>' -iA nur.repos.nginx.nginx-prometheus-exporter

Building the Exporter

You can build the exporter using the provided Makefile. Before building the exporter, make sure the following software is installed on your machine:

• make
• git
• Docker for building the container image
• Go for building the binary

Building the Docker Image

To build the Docker image with the exporter, run:

make container

Note: go is not required, as the exporter binary is built in a Docker container. See the Dockerfile.

Building the Binary

To build the binary, run:

make

Note: the binary is built for the OS/arch of your machine. To build binaries for other platforms, see the Makefile.

The binary is built with the name nginx-prometheus-exporter.

Grafana Dashboard

The official Grafana dashboard is provided with the exporter for NGINX. Check the Grafana Dashboard documentation for more information.

SBOM (Software Bill of Materials)

We generate SBOMs for the binaries and the Docker image.

Binaries

The SBOMs for the binaries are available in the releases page. The SBOMs are generated using syft and are available in SPDX format.

Docker Image

The SBOM for the Docker image is available in the DockerHub, GitHub Container registry, Amazon ECR Public Gallery and Quay.io repositories. The SBOMs are generated using syft and stored as an attestation in the image manifest.

For example to retrieve the SBOM for linux/amd64 from Docker Hub and analyze it using grype you can run the following command:

docker buildx imagetools inspect nginx/nginx-prometheus-exporter:edge --format '{{ json (index .SBOM "linux/amd64").SPDX }}' | grype

Provenance

We generate provenance for the Docker image and it's available in the DockerHub, GitHub Container registry, Amazon ECR Public Gallery and Quay.io repositories, stored as an attestation in the image manifest.

For example to retrieve the provenance for linux/amd64 from Docker Hub you can run the following command:

docker buildx imagetools inspect nginx/nginx-prometheus-exporter:edge --format '{{ json (index .Provenance "linux/amd64").SLSA }}'

Contacts

We’d like to hear your feedback! If you have any suggestions or experience issues with the NGINX Prometheus Exporter, please create an issue or send a pull request on GitHub. You can contact us directly via [email protected] or on the NGINX Community Slack in the #nginx-prometheus-exporter channel.

Contributing

If you'd like to contribute to the project, please read our Contributing guide.

Support

The commercial support is available for NGINX Plus customers when the NGINX Prometheus Exporter is used with NGINX Ingress Controller.

License

Apache License, Version 2.0.