Description
Over time there has been concerns regarding performance/efficiency relating to inputting form data (the frequency of patch requests). By default an app sends PATCH requests on React onchange (meaning on input and blur events) with a 400ms debounce. In this analysis we want to find out if we should make changes to configuration or their defaults, and wether or not it presents a scalability problem.
In scope
- Is frequent form data patching a scalability problem
- Should we change default configuration/make other recommendations
- Should we extend configuration options
Analysis
Concerns
- Datamodel XML is updated as a whole document in Azure Blob Storage (regardless of patch granularity)
- There are a lot of requests and traffic generated for a single patch request (e.g. authorization requests)
The continuous patching presents some UX advantages and enables
- Continuous validation based on backend code
- Continuous data processing (updating related form data fields based on incoming patches)
- Loss of data more unlikely due to frequently storing the current state
If we were to stop storing form data as frequently we would have to find alternative ways to offer the same UX, likely making some behavior changes and/or introduce new concepts.
Performance
I've extracted some performance data from Application Insights.
These are formdata patch requests for brg, showing the 99th percentile latency numbers for the last 24hrs:
These are all Azure blob storage requests for brg, showing the 99th percentile latency numbers for the last 24hrs:
I was unable to specifically extract latency numbers for Azure blob storage requests relating to PatchFormData operations since this telemetry exists in different workspaces technically. The performance tab in App Insights is able to show both but I can't seem to be able to query/join them both from "Logs"/kql.
For the PatchFormData requests the top dependencies seem to be:
So the Azure blob storage requests are apparantly not big parts of the e2e latency for form data patching.
If we ignore that the metrics for Azure blob storage requests include requests that come from operations other than PatchFormData, we could say that for this particular time window the blob updating accounts for about 7% for the e2e latency.
If we zoom in on a single slow patch request (well within 99th but still on the slower end of the spectrum at 500ms):
We see that in this case the initial authorization request was the biggest latency contributor (the one originating from the app) at ~306ms.
Note that we authorize the operation for a second time within Storage during the PUT rqeuest, but this takes only ~4ms
Infra/vendor SLA and limits
Azure Storage accounts have the following relevant limits
- 40 000 requests per second for the whole Storage Account
- 60 Gbps maximum ingress for the whole Storage Account
- 200 Gbps maximum egress for the whole Storage Account
- 5PiB maximum storage capacity for the whole Storage Account
All of these limits are defaults and can be extended. There doesn't seem to be an SLA/perormance target for latency.
For each patch request we seem to make 4 individual blob storage requests (2 reads and 2 writes presumably). If we assume that each person makes a patch request once every 2 seconds, we can assume that ignoring any other concurrent operations (frontend doing reads, processors loading additional dataelements etc etc...) on the same SA we could tolerate 20 000 people per serviceowner concurrently.
Sources:
Conclusion
--
Description
Over time there has been concerns regarding performance/efficiency relating to inputting form data (the frequency of patch requests). By default an app sends PATCH requests on React
onchange(meaning oninputandblurevents) with a 400ms debounce. In this analysis we want to find out if we should make changes to configuration or their defaults, and wether or not it presents a scalability problem.In scope
Analysis
Concerns
The continuous patching presents some UX advantages and enables
If we were to stop storing form data as frequently we would have to find alternative ways to offer the same UX, likely making some behavior changes and/or introduce new concepts.
Performance
I've extracted some performance data from Application Insights.
These are formdata patch requests for brg, showing the 99th percentile latency numbers for the last 24hrs:
These are all Azure blob storage requests for brg, showing the 99th percentile latency numbers for the last 24hrs:
I was unable to specifically extract latency numbers for Azure blob storage requests relating to
PatchFormDataoperations since this telemetry exists in different workspaces technically. The performance tab in App Insights is able to show both but I can't seem to be able to query/join them both from "Logs"/kql.For the
PatchFormDatarequests the top dependencies seem to be:So the Azure blob storage requests are apparantly not big parts of the e2e latency for form data patching.
If we ignore that the metrics for Azure blob storage requests include requests that come from operations other than
PatchFormData, we could say that for this particular time window the blob updating accounts for about 7% for the e2e latency.If we zoom in on a single slow patch request (well within 99th but still on the slower end of the spectrum at 500ms):
We see that in this case the initial authorization request was the biggest latency contributor (the one originating from the app) at ~306ms.
Note that we authorize the operation for a second time within Storage during the PUT rqeuest, but this takes only ~4ms
Infra/vendor SLA and limits
Azure Storage accounts have the following relevant limits
All of these limits are defaults and can be extended. There doesn't seem to be an SLA/perormance target for latency.
For each patch request we seem to make 4 individual blob storage requests (2 reads and 2 writes presumably). If we assume that each person makes a patch request once every 2 seconds, we can assume that ignoring any other concurrent operations (frontend doing reads, processors loading additional dataelements etc etc...) on the same SA we could tolerate 20 000 people per serviceowner concurrently.
Sources:
Conclusion
--