You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/tutorials/demo.md
+6-10Lines changed: 6 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,10 @@
1
1
# vllm with wva autoscaler
2
2
3
3
4
-
Notes:
4
+
Notes:
5
5
1. Experiments on OpenShift Cluster with H100 GPUs.
6
6
2. To setup `vLLM` on `Openshift`, refer to [vllm-samples.md](vllm-samples.md).
7
-
3. We use `guidellm` as the load generator. Refer to [guidellm-sample.md](guidellm-sample.md) for a quick tutorial to create your guidellm image that will be used in a `Job` resource.
7
+
3. We use `guidellm` as the load generator. Refer to [guidellm-sample.md](guidellm-sample.md) for a quick tutorial to create your guidellm image that will be used in a `Job` resource.
8
8
3. The WVA autoscaler is assumed to be deployed in `workload-variant-autoscaler-system` namespace.
9
9
10
10
@@ -96,19 +96,19 @@ spec:
96
96
- "--data"
97
97
- "prompt_tokens=128,output_tokens=512"
98
98
- "--output-path"
99
-
- "/tmp/benchmarks.json"
99
+
- "/tmp/benchmarks.json"
100
100
restartPolicy: Never
101
101
backoffLimit: 4
102
102
```
103
103
104
104
In each job, fill in `image: <image-repo>:<tag>` with your `guidellm` image repo and tag. The `<rate>` and `max-seconds` are set as follows.
105
105
106
-
- In `guidellm-job-1.yaml`, we set `<rate>` and `<max-seconds>` to `8` and `1800` respectively. By doing this, we force `guidellm` client to send requests at rate `8` requests per second (480 req/min) for `30` minutes.
106
+
- In `guidellm-job-1.yaml`, we set `<rate>` and `<max-seconds>` to `8` and `1800` respectively. By doing this, we force `guidellm` client to send requests at rate `8` requests per second (480 req/min) for `30` minutes.
107
107
- In `guidellm-job-2.yaml`, we set `<rate>` and `<max-seconds>` to `8` and `1200` respectively. We start this job after a couple of minutes of starting `guidellm-job-1`. When both jobs are running, we are effectively sending requests at rate `8+8 = 16` requests per second (960 req/min).
108
108
- In `guidellm-job-3.yaml`, we set `<rate>` and `<max-seconds>` to `8` and `720` respectively. We start this job after a couple of minutes of starting `guidellm-job-2`. When all the three jobs are running, we are effectively sending requests at rate `8+8+8 = 24` requests per second (1440 req/min) for 12 minutes.
109
109
- With this setup, `guidellm-job-3` will complete first, bringing the effective request rate back to `16` req/sec. This is followed by the completion of `guidellm-job-2`, which will bring down rate to `8` req/sec. Finally, `guidellm-job-1` completes, after which no further requests are sent.
110
110
111
-
**Dynamic Load Generation Summary:**
111
+
**Dynamic Load Generation Summary:**
112
112
- Step 1: `oc apply -f guidellm-job-1.yaml`. Wait about 5 minutes before continuing to step 2.
113
113
- Step 2: `oc apply -f guidellm-job-2.yaml`. Wait about 5 minutes before continuing to step 3.
114
114
- Step 3: `oc apply -f guidellm-job-3.yaml`
@@ -118,8 +118,4 @@ In each job, fill in `image: <image-repo>:<tag>` with your `guidellm` image repo
118
118
## WVA Performance
119
119
The following figure shows the behaviour observed from the controller logs.
0 commit comments