[Docs]: update metric names for clarity and consistency #1822

haitwang-cloud · 2025-12-05T02:54:48Z

Pull Request Description

In vllm-project/vllm#18354, the vllm:gpu_cache_usage_perc metric was deprecated and vllm:kv_cache_usage_perc was introduced as its replacement.

Related Issues

Resolves: #[Insert issue number(s)]

Important: Before submitting, please complete the description above and review the checklist below.

Contribution Guidelines (Expand for Details)

We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:

Pull Request Title Format

Your PR title should start with one of these prefixes to indicate the nature of the change:

[Bug]: Corrections to existing functionality
[CI]: Changes to build process or CI pipeline
[Docs]: Updates or additions to documentation
[API]: Modifications to aibrix's API or interface
[CLI]: Changes or additions to the Command Line Interface
[Misc]: For changes not covered above (use sparingly)

Note: For changes spanning multiple categories, use multiple prefixes in order of importance.

Submission Checklist

PR title includes appropriate prefix(es)
Changes are clearly explained in the PR description
New and existing tests pass successfully
Code adheres to project style and best practices
Documentation updated to reflect changes (if applicable)
Thorough testing completed, no regressions introduced

By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.

…oss multiple files - Updated autoscaling configurations (apa.yaml, hpa.yaml, kpa.yaml, etc.) to use kv_cache_usage_perc instead of gpu_cache_usage_perc. - Modified application code (app.py, metrics.py, etc.) to reflect the new metric name. - Adjusted documentation and README files to align with the metric changes. - Updated tests and mock contexts to ensure compatibility with the new metric. - Reflected changes in Grafana dashboards and JSON configurations for observability. Signed-off-by: Tim <[email protected]>

gemini-code-assist · 2025-12-05T02:55:23Z

Summary of Changes

Hello @haitwang-cloud, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request systematically updates all references to a previously deprecated metric, gpu_cache_usage_perc, replacing it with kv_cache_usage_perc. The change spans across configuration files, application code, documentation, and tests, ensuring a unified and accurate representation of KV-cache usage. This improves the maintainability and clarity of the system's metrics, particularly for autoscaling and observability purposes.

Highlights

Metric Renaming: The deprecated gpu_cache_usage_perc metric has been consistently renamed to kv_cache_usage_perc across the codebase, documentation, and configuration files. This change aligns with previous deprecation efforts and enhances clarity and consistency in metric reporting and autoscaling configurations.
Configuration Updates: All relevant autoscaling configuration files (APA, HPA, KPA, fleet-autoscaling, multimetrics-apa) and example manifests have been updated to reference the new kv_cache_usage_perc metric.
Code and Documentation Alignment: The change is reflected in Python application code (app.py), Go controller logic, metric fetching mechanisms, and various documentation files, ensuring all components use the updated metric name.
Testing and Observability: Unit and integration tests for the pod autoscaler, metric fetching, and metric rules have been updated to validate the new metric name. Grafana dashboards have also been adjusted to query the vllm:kv_cache_usage_perc metric.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request correctly updates the metric name from gpu_cache_usage_perc to kv_cache_usage_perc across a large number of files. The changes are mostly consistent and well-executed. However, I've identified a few areas for improvement to enhance clarity and consistency further:

Metric Value Inconsistency: There's a notable inconsistency in how the kv_cache_usage_perc metric is used. In some configuration and test files, it's treated as a ratio (0-1), while in others, it's a percentage (0-100). The _perc suffix and the implementation in development/app/app.py suggest it should be a percentage. I've added comments to standardize this to a percentage value (0-100) across all relevant files.
Constant Naming: In pkg/metrics/metrics.go, the constant for this metric is still named GPUCacheUsagePerc, which is now misleading. I've suggested renaming it to KVCacheUsagePerc.
Grafana Dashboards: The legends in some Grafana dashboard files (observability/grafana/AIBrix_vLLM_Engine_Dashboard.json and samples/deepseek-r1/static/AIBrix Engine Dashboard (vLLM)-1741078999667.json) still refer to 'GPU Cache Usage'. While I couldn't comment directly on these lines as they weren't in the diff, updating them to 'KV Cache Usage' would improve consistency.

Addressing these points will help solidify the clarity and consistency this pull request aims to achieve.

gemini-code-assist · 2025-12-05T02:58:57Z

samples/volcano-engine/autoscaler.yaml

      path: metrics
-      targetMetric: gpu_cache_usage_perc
+      targetMetric: kv_cache_usage_perc
      targetValue: '0.3'


The targetValue is 0.3, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase (e.g., development/app/app.py). For consistency, should this be 30?

targetValue: '30'

gemini-code-assist · 2025-12-05T02:58:57Z

benchmarks/scenarios/autoscaling/deepseek-llm-7b-chat/kpa.yaml

      path: metrics
-      targetMetric: gpu_cache_usage_perc
+      targetMetric: kv_cache_usage_perc
      targetValue: '0.5'


The targetValue is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase (e.g., development/app/app.py). For consistency, should this be 50?

targetValue: '50'

gemini-code-assist · 2025-12-05T02:58:57Z

benchmarks/scenarios/autoscaling/deepseek-llm-7b-chat/apa.yaml

      path: metrics
-      targetMetric: gpu_cache_usage_perc
+      targetMetric: kv_cache_usage_perc
      targetValue: '0.5'


The targetValue is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase (e.g., development/app/app.py). For consistency, should this be 50?

targetValue: '50'

gemini-code-assist · 2025-12-05T02:58:57Z

samples/autoscaling/apa.yaml

      path: metrics
-      targetMetric: gpu_cache_usage_perc
+      targetMetric: kv_cache_usage_perc
      targetValue: '0.5'


The targetValue is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase (e.g., development/app/app.py). For consistency, should this be 50?

targetValue: '50'

gemini-code-assist · 2025-12-05T02:58:57Z

samples/autoscaling/kpa.yaml

      path: metrics
-      targetMetric: gpu_cache_usage_perc
+      targetMetric: kv_cache_usage_perc
      targetValue: '0.5'


The targetValue is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase (e.g., development/app/app.py). For consistency, should this be 50?

targetValue: '50'

gemini-code-assist · 2025-12-05T02:58:57Z

test/integration/controller/podautoscaler_test.go

 						ScaleTargetRefWithKind("Deployment", "apps/v1", "cooldown-deployment").
 						MetricSource(wrapper.MakeMetricSourcePod(
-							autoscalingv1alpha1.HTTP, "8080", "/metrics", "gpu_cache_usage_perc", "0.5")).
+							autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "0.5")).


The target value is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase. For consistency, should this be 50?

Suggested change

autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "0.5")).

autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "50")).

gemini-code-assist · 2025-12-05T02:58:57Z

test/integration/controller/podautoscaler_test.go

 						ScaleTargetRefWithKind("Deployment", "apps/v1", "delay-deployment").
 						MetricSource(wrapper.MakeMetricSourcePod(
-							autoscalingv1alpha1.HTTP, "8080", "/metrics", "gpu_cache_usage_perc", "0.5")).
+							autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "0.5")).


The target value is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase. For consistency, should this be 50?

Suggested change

autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "0.5")).

autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "50")).

gemini-code-assist · 2025-12-05T02:58:57Z

test/integration/controller/podautoscaler_test.go

 						ScaleTargetRefWithKind("Deployment", "apps/v1", "annotations-deployment").
 						MetricSource(wrapper.MakeMetricSourcePod(
-							autoscalingv1alpha1.HTTP, "8080", "/metrics", "gpu_cache_usage_perc", "0.5")).
+							autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "0.5")).


The target value is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase. For consistency, should this be 50?

Suggested change

autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "0.5")).

autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "50")).

gemini-code-assist · 2025-12-05T02:58:57Z

test/integration/webhook/podautoscaler_webhook_test.go

 					ScaleTargetRefWithKind("Deployment", "apps/v1", "test").
 					MetricSource(wrapper.MakeMetricSourcePod(autoscalingapi.HTTP,
-						"8080", "/metrics", "gpu_cache_usage_perc", "0.5")).
+						"8080", "/metrics", "kv_cache_usage_perc", "0.5")).


The target value is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase. For consistency, should this be 50?

Suggested change

"8080", "/metrics", "kv_cache_usage_perc", "0.5")).

"8080", "/metrics", "kv_cache_usage_perc", "50")).

gemini-code-assist · 2025-12-05T02:58:57Z

test/integration/webhook/podautoscaler_webhook_test.go

 					ScaleTargetRefWithKind("Deployment", "apps/v1", "test-ss").
 					MetricSource(wrapper.MakeMetricSourceExternal(autoscalingapi.HTTP,
-						"monitoring.example.com", "/metrics", "gpu_cache_usage_perc", "0.5")).
+						"monitoring.example.com", "/metrics", "kv_cache_usage_perc", "0.5")).


The target value is 0.5, which suggests a ratio. However, the metric kv_cache_usage_perc seems to be a percentage (0-100) in other parts of the codebase. For consistency, should this be 50?

Suggested change

"monitoring.example.com", "/metrics", "kv_cache_usage_perc", "0.5")).

"monitoring.example.com", "/metrics", "kv_cache_usage_perc", "50")).

googs1025 · 2025-12-05T03:27:35Z

part of: #1815

googs1025 · 2025-12-05T03:29:38Z

I’ve become aware of this issue recently and will put it upstream to the sglang/vLLM maintainers.
Any changes to metrics should be made with caution, since downstream components—whether gateway layers or autoscaling—depend on them.

Jeffwan · 2025-12-05T04:00:07Z

I think right now, we lack of mechanisim to make control plane compatible with engine changes. this is a design problem

googs1025 · 2025-12-05T05:19:29Z

I think right now, we lack of mechanisim to make control plane compatible with engine changes. this is a design problem

some thought: https://github.com/vllm-project/aibrix/pull/1814/files#r2591433626

haitwang-cloud · 2025-12-05T06:04:59Z

@googs1025 @Jeffwan Thanks so much for commenting! I wholeheartedly agree that it's of utmost importance to request the upstream (vLLM/SGLang) to maintain metric compatibility. It's super crucial for all LLM users.

Jeffwan · 2025-12-05T06:18:16Z

could i know which version introduced kv_cache_usage_perc ?

Jeffwan · 2025-12-05T06:19:08Z

@googs1025 we should consider to allow user to annotate version on pod and then support engine[version[metrics]] map if version is given. WDYT?

googs1025 · 2025-12-05T08:09:45Z

This may be unrelated to this PR; we can discuss it in this issue #1815.
Additionally, I made some compatibility in this PR #1814 , but we can still be more flexible for user to set

haitwang-cloud · 2025-12-05T08:36:27Z

could i know which version introduced kv_cache_usage_perc ?

At least from V.10.*, cause this PR vllm-project/vllm#18354 was merged in May 2025

haitwang-cloud · 2025-12-05T08:56:28Z

This may be unrelated to this PR; we can discuss it in this issue #1815. Additionally, I made some compatibility in this PR #1814 , but we can still be more flexible for user to set

How about adding a TODO list in the roadmap for the next version to deal with this?

googs1025 · 2025-12-05T09:25:40Z

pkg/metrics/engine_fetcher_test.go

 		MetricSource: PodRawMetrics,
 		MetricType:   MetricType{Raw: Gauge},
 		EngineMetricsNameMapping: map[string]string{
-			"vllm":   "vllm_gpu_cache_usage_perc",


plz remove this part. We need a fallback mechanism; if we make this change, the value won't be available in versions prior to v0.10.0. This will be supported in this pr #1814

gemini-code-assist bot reviewed Dec 5, 2025

View reviewed changes

googs1025 requested a review from Jeffwan December 5, 2025 03:29

googs1025 reviewed Dec 5, 2025

View reviewed changes

	autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "0.5")).
	autoscalingv1alpha1.HTTP, "8080", "/metrics", "kv_cache_usage_perc", "50")).

	"8080", "/metrics", "kv_cache_usage_perc", "0.5")).
	"8080", "/metrics", "kv_cache_usage_perc", "50")).

	"monitoring.example.com", "/metrics", "kv_cache_usage_perc", "0.5")).
	"monitoring.example.com", "/metrics", "kv_cache_usage_perc", "50")).

[Docs]: update metric names for clarity and consistency #1822

Are you sure you want to change the base?

[Docs]: update metric names for clarity and consistency #1822

Conversation

haitwang-cloud commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Related Issues

Pull Request Title Format

Submission Checklist

Uh oh!

gemini-code-assist bot commented Dec 5, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

googs1025 commented Dec 5, 2025

Uh oh!

googs1025 commented Dec 5, 2025

Uh oh!

Jeffwan commented Dec 5, 2025

Uh oh!

googs1025 commented Dec 5, 2025

Uh oh!

haitwang-cloud commented Dec 5, 2025

Uh oh!

Jeffwan commented Dec 5, 2025

Uh oh!

Jeffwan commented Dec 5, 2025

Uh oh!

googs1025 commented Dec 5, 2025

Uh oh!

haitwang-cloud commented Dec 5, 2025

Uh oh!

haitwang-cloud commented Dec 5, 2025

Uh oh!

googs1025 Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

haitwang-cloud commented Dec 5, 2025 •

edited

Loading