Optimize routing for multimodal inputs with image/video URLs in the request

[rahulgurnani]

What type of PR is this?

Prefixing the text with image/video URL in prefix cache scorer, leads to better performance i.e. higher throughput and lower response latency.

This is an interim change until #2665 is ideated/finalised.

What this PR does / why we need it:
Prefixing with URL in prefix cache scorer to index on encoder cache, specifically valuable in case of multimodal inputs with URLs.
This is an interim change until #2665 is ideated/finalised.

Benchmarking results

By adding url as a prefix, the throughput/TTFT seem to improve like of the order of 40%. These numbers are comparing the changes in this PR with EPP on main. The same_url_multi_text.py takes like 14.41 seconds on main and it takes 9.06 seconds on this PR.

Which issue(s) this PR fixes:

This PR is related to #2665 in a way they support each other

Does this PR introduce a user-facing change?:

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	4e272b7
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/69c27f60368e650008ca9b9d
😎 Deploy Preview	https://deploy-preview-2666--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rahulgurnani
Once this PR has been reviewed and has the lgtm label, please assign ahg-g for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rahulgurnani]

/retest

[capri-xiyue]

I'm wondering did you just append the image url before text? If so, probably this will cause issues when you have multiple image urls has the same prefix like gs://image-repo/xxxx

[k8s-ci-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.