dataplatform-probe

Monitoring service for dataplatform services.

Continuously sends GET requests to the dataplatform metadata-api in order to keep an eye on Keycloak.

Metrics

This app uses the prometheus_client library to expose metrics to Prometheus regarding pipeline latency through a http server on port 8000.

Name	Type	Description
probe_requests_created	Counter	Number of created requests
probe_requests_succeeded	Counter	Number of succeeded requests
probe_requests_failed	Counter	Number of failed requests
probe_request_duration	Gauge	The duration of the last succeeded requests

Configuration

The app is configurable by setting the following environment variables (* = required, no default):

Name	Description	Default
DATASET_ID*	Dataset ID
TASK_INTERVAL_SECONDS	Interval in seconds between requests	30
BETTERUPTIME_HEARTBEAT_URL	Send heartbeat to Better Uptime on successful task execution

In addition, OKDATA_CLIENT_ID, OKDATA_CLIENT_SECRET, and OKDATA_ENVIRONMENT, must also be set when deploying.

Development

A makefile with various commands is provided for convenience and to ease development. These are:

$ make init # Install dependencies
$ make format # Format using black
$ make lint # Lint using flake8
$ make run # Run application (as described below)

Running

By issuing the command make run, a local environment is configured comprising of the following services (see local-compose.yaml):

• Prometheus | http://localhost:9090
  Monitoring and alerting toolkit.

• Grafana (+ tns-db) | http://localhost:3000
  Observability and data visualization platform. Includes a provisioned datasource and dashboard for the application. Default username/password: admin/admin.

• HTTP server | http://localhost:8081
  Accepts requests from tasks, currently emulating okdata-metadata-api. By default configured to introduce some (more or less random) latency (between 0-3 seconds) for ~10 percent of requests, as well as failing with 401 for ~5 percent (configurable in local/http_server/http_server.py).

The run target sets LOCAL_RUN=true and LOCAL_SERVICES_ONLY=true. While the first environment variable enables "debug mode", the latter tells the application to use the "dummy" HTTP server mentioned above.

To test against real dataplatform services (while still running the application locally), set the appropriate environment variables listed above (i.e. credentials and dataset id) and use make run-dp.

$ docker-compose -f local-compose.yaml ps # Check services
$ docker-compose -f local-compose.yaml logs --follow --tail=10 # Tail service logs

Deploy

Deploy to both dev and prod is automatic via GitHub Actions on push to master.