Improving metrics with blob, kv, and state duration metrics

[seriousben]

Context

We were missing metrics around blob storage, state storage, and kv storage.
We also had buckets that started at 5 seconds.

What

• Adding metrics to have duration data per different read and write operations.
• Adding millisecond buckets to the histograms to help with tracking increases in latency for low latency operations.
• Audited use of info to make it less verbose or to properly log errors.

Testing

Run the graph behavior test and look at http://localhost:8900/metrics/service.

Details

# HELP blob_operations_duration_seconds blob store latencies in seconds
# TYPE blob_operations_duration_seconds histogram
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="0.001"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="0.005"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="0.01"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="0.05"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="0.1"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="0.5"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="1"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="5"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="10"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="25"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="50"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="75"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="100"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="250"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="500"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="750"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="1000"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="2500"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="5000"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="7500"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="10000"} 110
blob_operations_duration_seconds_bucket{op="get",otel_scope_name="blob-storage",le="+Inf"} 110
blob_operations_duration_seconds_sum{op="get",otel_scope_name="blob-storage"} 0.009765587000000008
blob_operations_duration_seconds_count{op="get",otel_scope_name="blob-storage"} 110
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="0.001"} 94
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="0.005"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="0.01"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="0.05"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="0.1"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="0.5"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="1"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="5"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="10"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="25"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="50"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="75"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="100"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="250"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="500"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="750"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="1000"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="2500"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="5000"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="7500"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="10000"} 104
blob_operations_duration_seconds_bucket{op="put",otel_scope_name="blob-storage",le="+Inf"} 104
blob_operations_duration_seconds_sum{op="put",otel_scope_name="blob-storage"} 0.053783284
blob_operations_duration_seconds_count{op="put",otel_scope_name="blob-storage"} 104
# HELP executors_online Number of executors online
# TYPE executors_online gauge
executors_online{otel_scope_name="scheduler_stats"} 1
# HELP fn_output_bytes_total number of bytes ingested for fn outputs
# TYPE fn_output_bytes_total counter
fn_output_bytes_total{otel_scope_name="service-api"} 83368
# HELP fn_outputs_total number of fn outputs
# TYPE fn_outputs_total counter
fn_outputs_total{otel_scope_name="service-api"} 65
# HELP invocation_bytes_total number of bytes ingested during invocations
# TYPE invocation_bytes_total counter
invocation_bytes_total{otel_scope_name="service-api"} 36629
# HELP invocations_total number of invocations
# TYPE invocations_total counter
invocations_total{otel_scope_name="service-api"} 17
# HELP kv_storage_read_duration_seconds K/V store read latencies in seconds
# TYPE kv_storage_read_duration_seconds histogram
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="0.001"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="0.005"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="0.01"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="0.05"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="0.1"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="0.5"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="1"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="5"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="10"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="25"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="50"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="75"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="100"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="250"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="500"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="750"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="1000"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="2500"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="5000"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="7500"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="10000"} 5
kv_storage_read_duration_seconds_bucket{op="get_ctx_state_key",otel_scope_name="kv-storage",le="+Inf"} 5
kv_storage_read_duration_seconds_sum{op="get_ctx_state_key",otel_scope_name="kv-storage"} 0.000036249
kv_storage_read_duration_seconds_count{op="get_ctx_state_key",otel_scope_name="kv-storage"} 5
# HELP kv_storage_write_duration_seconds k/v store write latencies in seconds
# TYPE kv_storage_write_duration_seconds histogram
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="0.001"} 0
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="0.005"} 1
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="0.01"} 1
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="0.05"} 1
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="0.1"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="0.5"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="1"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="5"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="10"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="25"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="50"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="75"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="100"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="250"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="500"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="750"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="1000"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="2500"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="5000"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="7500"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="10000"} 3
kv_storage_write_duration_seconds_bucket{op="put_ctx_state",otel_scope_name="kv-storage",le="+Inf"} 3
kv_storage_write_duration_seconds_sum{op="put_ctx_state",otel_scope_name="kv-storage"} 0.11577283299999999
kv_storage_write_duration_seconds_count{op="put_ctx_state",otel_scope_name="kv-storage"} 3
# HELP scheduler_invocations Scheduler invocation latencies in seconds
# TYPE scheduler_invocations histogram
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="0.001"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="0.005"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="0.01"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="0.05"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="0.1"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="0.5"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="1"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="5"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="10"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="25"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="50"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="75"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="100"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="250"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="500"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="750"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="1000"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="2500"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="5000"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="7500"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="10000"} 120
scheduler_invocations_bucket{otel_scope_name="scheduler_stats",le="+Inf"} 120
scheduler_invocations_sum{otel_scope_name="scheduler_stats"} 0.033748753000000006
scheduler_invocations_count{otel_scope_name="scheduler_stats"} 120
# HELP state_machine_read_duration_seconds State machine reading latency in seconds
# TYPE state_machine_read_duration_seconds histogram
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="0.001"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="0.005"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="0.01"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="0.05"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="0.1"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="0.5"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="1"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="5"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="10"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="25"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="50"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="75"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="100"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="250"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="500"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="750"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="1000"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="2500"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="5000"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="7500"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="10000"} 108
state_machine_read_duration_seconds_bucket{op="filter_join_cf",otel_scope_name="state_store",le="+Inf"} 108
state_machine_read_duration_seconds_sum{op="filter_join_cf",otel_scope_name="state_store"} 0.004169504
state_machine_read_duration_seconds_count{op="filter_join_cf",otel_scope_name="state_store"} 108
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="0.001"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="0.005"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="0.01"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="0.05"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="0.1"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="0.5"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="1"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="5"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="10"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="25"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="50"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="75"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="100"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="250"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="500"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="750"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="1000"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="2500"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="5000"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="7500"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="10000"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload",otel_scope_name="state_store",le="+Inf"} 34
state_machine_read_duration_seconds_sum{op="fn_output_payload",otel_scope_name="state_store"} 0.000403912
state_machine_read_duration_seconds_count{op="fn_output_payload",otel_scope_name="state_store"} 34
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="0.001"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="0.005"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="0.01"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="0.05"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="0.1"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="0.5"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="1"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="5"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="10"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="25"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="50"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="75"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="100"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="250"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="500"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="750"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="1000"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="2500"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="5000"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="7500"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="10000"} 39
state_machine_read_duration_seconds_bucket{op="fn_output_payload_by_key",otel_scope_name="state_store",le="+Inf"} 39
state_machine_read_duration_seconds_sum{op="fn_output_payload_by_key",otel_scope_name="state_store"} 0.0005760019999999999
state_machine_read_duration_seconds_count{op="fn_output_payload_by_key",otel_scope_name="state_store"} 39
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="0.001"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="0.005"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="0.01"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="0.05"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="0.1"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="0.5"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="1"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="5"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="10"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="25"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="50"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="75"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="100"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="250"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="500"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="750"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="1000"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="2500"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="5000"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="7500"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="10000"} 59
state_machine_read_duration_seconds_bucket{op="get_all_executors",otel_scope_name="state_store",le="+Inf"} 59
state_machine_read_duration_seconds_sum{op="get_all_executors",otel_scope_name="state_store"} 0.0012989149999999997
state_machine_read_duration_seconds_count{op="get_all_executors",otel_scope_name="state_store"} 59
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="0.001"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="0.005"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="0.01"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="0.05"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="0.1"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="0.5"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="1"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="5"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="10"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="25"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="50"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="75"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="100"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="250"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="500"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="750"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="1000"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="2500"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="5000"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="7500"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="10000"} 169
state_machine_read_duration_seconds_bucket{op="get_compute_graph",otel_scope_name="state_store",le="+Inf"} 169
state_machine_read_duration_seconds_sum{op="get_compute_graph",otel_scope_name="state_store"} 0.008419749000000004
state_machine_read_duration_seconds_count{op="get_compute_graph",otel_scope_name="state_store"} 169
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="0.001"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="0.005"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="0.01"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="0.05"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="0.1"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="0.5"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="1"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="5"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="10"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="25"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="50"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="75"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="100"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="250"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="500"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="750"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="1000"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="2500"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="5000"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="7500"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="10000"} 356
state_machine_read_duration_seconds_bucket{op="get_from_cf",otel_scope_name="state_store",le="+Inf"} 356
state_machine_read_duration_seconds_sum{op="get_from_cf",otel_scope_name="state_store"} 0.010014668000000003
state_machine_read_duration_seconds_count{op="get_from_cf",otel_scope_name="state_store"} 356
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="0.001"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="0.005"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="0.01"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="0.05"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="0.1"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="0.5"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="1"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="5"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="10"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="25"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="50"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="75"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="100"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="250"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="500"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="750"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="1000"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="2500"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="5000"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="7500"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="10000"} 1
state_machine_read_duration_seconds_bucket{op="get_gc_urls",otel_scope_name="state_store",le="+Inf"} 1
state_machine_read_duration_seconds_sum{op="get_gc_urls",otel_scope_name="state_store"} 0.000022708
state_machine_read_duration_seconds_count{op="get_gc_urls",otel_scope_name="state_store"} 1
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="0.001"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="0.005"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="0.01"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="0.05"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="0.1"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="0.5"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="1"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="5"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="10"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="25"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="50"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="75"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="100"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="250"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="500"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="750"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="1000"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="2500"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="5000"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="7500"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="10000"} 130
state_machine_read_duration_seconds_bucket{op="get_namespace",otel_scope_name="state_store",le="+Inf"} 130
state_machine_read_duration_seconds_sum{op="get_namespace",otel_scope_name="state_store"} 0.0017453350000000009
state_machine_read_duration_seconds_count{op="get_namespace",otel_scope_name="state_store"} 130
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.001"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.005"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.01"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.05"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.1"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.5"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="1"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="5"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="10"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="25"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="50"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="75"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="100"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="250"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="500"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="750"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="1000"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="2500"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="5000"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="7500"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="10000"} 46
state_machine_read_duration_seconds_bucket{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store",le="+Inf"} 46
state_machine_read_duration_seconds_sum{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store"} 0.000615415
state_machine_read_duration_seconds_count{op="get_raw_rows_from_cf_with_limits",otel_scope_name="state_store"} 46
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="0.001"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="0.005"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="0.01"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="0.05"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="0.1"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="0.5"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="1"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="5"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="10"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="25"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="50"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="75"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="100"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="250"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="500"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="750"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="1000"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="2500"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="5000"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="7500"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="10000"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_multi_key",otel_scope_name="state_store",le="+Inf"} 103
state_machine_read_duration_seconds_sum{op="get_rows_from_cf_multi_key",otel_scope_name="state_store"} 0.0014976210000000008
state_machine_read_duration_seconds_count{op="get_rows_from_cf_multi_key",otel_scope_name="state_store"} 103
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.001"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.005"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.01"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.05"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.1"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="0.5"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="1"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="5"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="10"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="25"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="50"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="75"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="100"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="250"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="500"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="750"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="1000"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="2500"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="5000"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="7500"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="10000"} 149
state_machine_read_duration_seconds_bucket{op="get_rows_from_cf_with_limits",otel_scope_name="state_store",le="+Inf"} 149
state_machine_read_duration_seconds_sum{op="get_rows_from_cf_with_limits",otel_scope_name="state_store"} 0.0027811300000000006
state_machine_read_duration_seconds_count{op="get_rows_from_cf_with_limits",otel_scope_name="state_store"} 149
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="0.001"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="0.005"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="0.01"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="0.05"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="0.1"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="0.5"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="1"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="5"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="10"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="25"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="50"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="75"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="100"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="250"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="500"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="750"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="1000"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="2500"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="5000"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="7500"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="10000"} 1
state_machine_read_duration_seconds_bucket{op="get_system_tasks",otel_scope_name="state_store",le="+Inf"} 1
state_machine_read_duration_seconds_sum{op="get_system_tasks",otel_scope_name="state_store"} 0.000035959
state_machine_read_duration_seconds_count{op="get_system_tasks",otel_scope_name="state_store"} 1
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="0.001"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="0.005"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="0.01"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="0.05"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="0.1"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="0.5"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="1"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="5"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="10"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="25"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="50"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="75"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="100"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="250"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="500"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="750"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="1000"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="2500"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="5000"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="7500"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="10000"} 57
state_machine_read_duration_seconds_bucket{op="get_task",otel_scope_name="state_store",le="+Inf"} 57
state_machine_read_duration_seconds_sum{op="get_task",otel_scope_name="state_store"} 0.0008034250000000002
state_machine_read_duration_seconds_count{op="get_task",otel_scope_name="state_store"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="0.001"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="0.005"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="0.01"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="0.05"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="0.1"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="0.5"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="1"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="5"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="10"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="25"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="50"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="75"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="100"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="250"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="500"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="750"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="1000"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="2500"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="5000"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="7500"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="10000"} 57
state_machine_read_duration_seconds_bucket{op="get_task_outputs",otel_scope_name="state_store",le="+Inf"} 57
state_machine_read_duration_seconds_sum{op="get_task_outputs",otel_scope_name="state_store"} 0.0015839609999999998
state_machine_read_duration_seconds_count{op="get_task_outputs",otel_scope_name="state_store"} 57
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="0.001"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="0.005"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="0.01"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="0.05"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="0.1"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="0.5"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="1"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="5"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="10"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="25"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="50"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="75"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="100"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="250"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="500"} 108
state_machine_read_duration_seconds_bucket{op="get_tasks_by_executor",otel_scope_name="state_store",le="750"} 108

........

Contribution Checklist

• If the python-sdk was changed, please run make fmt in python-sdk/.
• If the server was changed, please run make fmt in server/.
• Make sure all PR Checks are passing.

[diptanu]

Looks fine, remove the unwraps please

[diptanu]

No unwraps

[seriousben]

Since this happens at initialization time, I was thinking of keeping it.

[diptanu]

@seriousben What is left here? Bunch of new metrics here so we should merge this before making a new release

[seriousben]

@diptanu yeah, let's cut a release Monday.

I will have a bunch of PRs ready including this one.
I also fixed a bunch of edge cases.