feat: ORCA Format KV Cache Utilization in Inference Response Header

[BenjaminBraunDev]

What does the PR do?

This PR adds code to HTTPAPIServer::GenerateRequestClass::StartResponse inside src/http_server.cc to add both kv_cache_utilization and max_token_capacity metrics composed from the existing prometheus metrics in TensorRT-LLM Backend's nv_trt_llm_kv_cache_block_metrics metric family.

This is acomplished by parsing the serialized prometheus metrics text object provided to the Triton Sever frontend by the Triton Core libraries into a structured vector of metrics for a specific metric family.

Checklist

• I have read the Contribution guidelines and signed the Contributor License
  Agreement
• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• I ran pre-commit locally (pre-commit install, pre-commit run --all)
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Where should the reviewer start?

Changes are contained to 2 files:

• src/http_server.cc
• src/http_server.h (the former's header file)

The changes start in HTTPAPIServer::GenerateRequestClass::StartResponse() where the environment variable is checked and the header is written. There are 3 other funcitons below: MetricFamilyExtractor() which parses serialized prometheus metrics into a vector of PromMetric (which have a map of their metric labels), and ExtractKVMetrics() that pulls the values from the structured metrics and calculates the composite kv metrics, and finally OrcaKVMetricHeader() which forms the metrics into an endpoint-load-metrics header in the ORCA format specified by ORCA_METRIC_FORMAT. If there are no TensorRT-LLM Backend metrics, no metrics found for the header, or an invalid format type the header is simply not written.

The valid values for ORCA_METRIC_FORMAT are documented in the feature request (related issue linked below) and comments in StartResponse()

Test plan:

The feature is gated behind a feature flag in the form of the ORCA_METRIC_FORMAT environment variable. If unset, the feature is effectively disabled. Beyond that, the changes have been manually tested to not cause issue if either the queried metrics are not present (such as if TensorRT-LLM is not being used as the backend), or if the ORCA header metric type is invalid. In either case, nothing is parsed and no header is written. All code changes are wrapped in an #ifdef and are only included if metrics are enabled during the Triton Server build.

Caveats:

1. This feature only works on Triton Inference Server running with TensorRT-LLM Backend, as otherwise the KV-cache metrics are not included in the server metrics.

2. This change only implements the kv-cache utilization metics, but the functions it adds allows other metrics to be added easily (including metrics that don't require TensorRT-LLM Backend).

Background

This doc captures the overall requirements for model servers to integrate with llm instance gateway. More details in the Feature Request below.

Related Issues:

• Resolves Feature Request GitHub issue: #7865

Screenshots

Response header before changes (or if ORCA_METRIC_FORMAT environment variable is unset):

Response header with ORCA_METRIC_FORMAT="json":

Response header with ORCA_METRIC_FORMAT="http":

cc @yinggeh @krishung5 @jbkyang-nvi

[jbkyang-nvi]

@nnshah1 @indrajit96 are we merging into the already frozen r24.10? Or should this be onto main instead?

[nnshah1]

@nnshah1 @indrajit96 are we merging into the already frozen r24.10? Or should this be onto main instead?

good catch we should target main