Commit 89a1643
![autofix-ci[bot]](https://avatars.githubusercontent.com/in/243519?v=4&size=40){kind=avatar}
feat(job_queue): export cancel_stats and active_jobs to OpenTelemetry (#13123)
* feat: implement Redis job queue and fakeredis support
- Refactored the job queue service to support Redis-backed management for cross-worker scaling.
- Added environment variables for configuration:
- `LANGFLOW_JOB_QUEUE_TYPE=redis`
- `LANGFLOW_REDIS_QUEUE_DB=1`
- Updated job ownership methods to be asynchronous for improved concurrency handling.
- Enhanced Redis cache service with namespacing via key prefixes.
- Introduced `fakeredis` for in-memory Redis simulation in testin>
- Added comprehensive unit tests for Redis job queue components.
* fix: run experimental warning for RedisCache usage only once
- Introduced a mechanism to emit a one-time warning for the RedisCache experimental feature during server runtime.
- The warning is logged only if no other worker has already emitted it, ensuring clarity for users regarding the experimental status of RedisCache.
- The implementation includes a temporary file check to prevent multiple warnings across different processes.
* docs: document environment variables for worker management
- Added documentation for LANGFLOW_GUNICORN_PRELOAD to explain preloading for better performance.
- Detailed the use of LANGFLOW_JOB_QUEUE_TYPE for specifying backends (e.g., Redis).
- Included LANGFLOW_REDIS_QUEUE_DB to define the database index for job queues.
- Updated the "High-Load Environments" guide with these optimal configurations.
* docs: updated 'High-load and multi-worker environments' section
* feat: enhance RedisJobQueueService with consumer wrapper management
- Introduced a caching mechanism for Redis stream consumers to optimize job data retrieval.
- Added methods to manage consumer wrappers, ensuring they are reused across sequential polls.
- Implemented cleanup logic to cancel and clear consumer wrappers during job cleanup and service stop.
- Expanded unit tests to verify consumer wrapper reuse and cleanup behavior.
* fix: ensure Redis keys are deleted during job cleanup even on cancellation
- Updated the cleanup_job method in RedisJobQueueService to guarantee Redis keys are removed even if the job cleanup is interrupted by a CancelledError.
- Added a new unit test to verify that Redis keys are deleted correctly when cleanup is called during task cancellation.
* fix: manage connection check task in RedisJobQueueService
- Added handling for the connection check task in the stop method to ensure it is properly cancelled and awaited if still running.
- This change improves resource management and prevents potential issues during service shutdown.
* fix: handle unpublished sentinel requeue on cancellation in RedisJobQueueService
- Updated the job processing logic to ensure that if a job is cancelled during the xadd operation, the unpublished sentinel is requeued instead of being dropped.
- Introduced a new unit test to verify this behavior, ensuring robustness in job handling during cancellations.
* fix: improve Redis job queue service with enhanced configuration and cleanup
- Added atexit cleanup to remove stale temporary files for RedisCache.
- Refactored Redis job queue service to use shared constants for stream prefixes, improving maintainability.
- Updated type hints for better clarity and consistency in RedisQueueWrapper and RedisJobQueueService.
- Enhanced error handling with configurable backoff for transient read failures.
* fix: enhance Redis job queue service with maxlen configuration for xadd
- Updated the xadd method in RedisJobQueueService to include maxlen and approximate parameters, improving stream management and preventing excessive memory usage.
* fix: enhance job ownership retrieval in RedisJobQueueService
- Updated the get_job_owner method to refresh the Redis key TTL on successful lookups, ensuring long-running jobs maintain their ownership anchor.
- Improved code clarity by extracting the owner key into a variable and adding detailed docstring explanations for better understanding of the TTL management.
* fix: improve job ownership handling in cleanup_job method
- Enhanced the cleanup_job method in RedisJobQueueService to accurately capture job ownership before deleting Redis keys, preventing potential data corruption in multi-worker scenarios.
- Added comments for clarity on ownership logic and its implications during job cleanup.
* fix: optimize TTL management in RedisJobQueueService
- Introduced periodic TTL refresh logic in the _bridge_to_redis method to enhance Redis stream management, reducing round-trips and improving throughput.
- Added constants for TTL refresh events and seconds to maintain clarity and configurability.
- Updated event handling to ensure TTL is refreshed appropriately based on event count and time elapsed.
* fix: improve event handling in flow response management
- Removed unnecessary error handling for missing event tasks in get_flow_events_response, allowing for smoother operation when no task exists.
- Updated create_flow_response to handle optional event_task parameter, ensuring proper cleanup during disconnections.
- Added unit tests to verify behavior when event tasks are missing, enhancing robustness in streaming scenarios.
* fix: enhance RedisQueueWrapper with startup grace period and stream observation
- Added a startup grace period to prevent premature end-of-stream signals when the producer has not yet issued its first XADD.
- Introduced a flag to track whether the stream has been observed, improving the handling of early polling scenarios.
- Updated logic to ensure proper handling of stream existence checks and logging for better debugging during job processing.
* fix: implement cleanup for old cross-worker job queues in RedisJobQueueService
- Added a new method to clean up done cross-worker consumer wrappers that are not owned by the current worker, ensuring proper resource management.
- Enhanced the existing cleanup logic to prevent memory leaks by explicitly pruning stale entries from the consumer wrappers dictionary.
- Improved logging to provide better visibility into the cleanup process for cross-worker jobs.
* fix: enhance job handling in JobQueueService with guarded task execution
- Updated the start_job method to accept a Coroutine type for task_coro, ensuring type safety.
- Introduced a new _guarded_task method to wrap job coroutines, guaranteeing that unhandled exceptions emit an error event and write a sentinel to the Redis Stream, improving reliability in job processing.
- Enhanced documentation to clarify the behavior of the new task handling mechanism and its implications for cross-worker consumers.
* fix: improve client disconnection handling in create_flow_response
- Added logging for scenarios where a client disconnects without an associated event_task, clarifying that the producer will continue running until the build completes.
- Documented the limitation regarding cross-worker passive disconnects and the need for a Redis side-channel for proper cancellation, enhancing observability in the logs.
* fix: enhance RedisQueueWrapper to manage initial read state and buffer behavior
- Introduced a flag to track the completion of the first XREAD call, ensuring proper buffer management during the initial read phase.
- Updated the empty method to reflect the state of the buffer accurately, preventing premature exits from the drain loop until the first read is complete.
- Improved documentation to clarify the behavior of the new flag and its impact on job processing.
* fix: clarify RedisQueueWrapper behavior in tests for buffer state and no-op operations
- Updated test documentation to explain the behavior of the empty() method in relation to the first XREAD completion, ensuring accurate understanding of buffer state during job processing.
- Adjusted assertions in tests to reflect the intended behavior of the RedisQueueWrapper, specifically regarding the no-op nature of put_nowait and its impact on the internal buffer.
* fix: update dependencies in pyproject.toml and uv.lock
- Added fakeredis dependency with a minimum version of 2.0.0 to both pyproject.toml and uv.lock files.
- Ensured proper formatting and comments in pyproject.toml for clarity on onnxruntime version constraints.
* feat(job_queue): cross-worker cancel via Redis pub/sub + configurable startup grace
- Add LANGFLOW_REDIS_QUEUE_STARTUP_GRACE_S (default 30.0) so deployments
with slow producer-worker cold-start can bump the wrapper's early-poll
grace period without editing code.
- Add LANGFLOW_REDIS_QUEUE_CANCEL_CHANNEL_ENABLED (default True) wiring a
per-job Redis pub/sub channel (langflow:cancel:<job_id>) so
POST /build/{job_id}/cancel works cross-worker. The producer subscribes
when the job starts; any worker can publish via
RedisJobQueueService.signal_cancel and the subscriber cancels the local
build task on receipt.
- cancel_flow_build now falls back to signal_cancel when event_task is
None on this worker, replacing the previous no-op for cross-worker
cancellation.
- Subscriber tasks are tracked in _cancel_subscribers and torn down in
cleanup_job/stop alongside the bridge tasks.
- Tests: 3 new unit tests covering cross-worker signal propagation, the
disabled-flag no-op, and the wrapper's per-instance startup_grace_s
override.
* fix(job_queue): close three cross-worker cancel corner cases
Discovered by a deeper local stress pass over the prototype:
1. **Signal-before-subscribe race**: publish from worker A would be lost if
worker B's subscriber had not finished SUBSCRIBE yet. signal_cancel now
also sets a short-lived langflow:cancel-marker:<job_id> key, and the
subscriber checks it immediately after subscribing — so a cancel that
raced the SUBSCRIBE still fires.
2. **Restart subscriber leak**: start_job called twice for the same job_id
silently overwrote the dict entry and orphaned the previous subscriber
(and its Redis pubsub connection). Now cancels the previous subscriber
before storing the new one.
3. **Slow end-of-stream after cross-worker cancel**: cancelling the local
task did not unblock cross-worker consumers — the bridge sat waiting on
local_queue.get() until periodic cleanup ran ~5 min later. The
subscriber now puts a sentinel on the local queue after task.cancel()
so the bridge flushes a clean end-of-stream marker to Redis. Measured
8ms from signal_cancel to consumer end-of-stream against real Redis.
Tests: 3 new unit tests covering each corner case; 9-scenario local
stress harness against real Redis confirms all green.
* refactor(job_queue): address self-review of cross-worker cancel
Addresses the eight points from the PR review of the previous commits:
1. Configurable marker TTL — new LANGFLOW_REDIS_QUEUE_CANCEL_MARKER_TTL
setting (default 60s) replaces the hardcoded constant.
2. Connection-pool footprint is O(1) in active jobs — replaced the per-job
pub/sub subscriber with a single PSUBSCRIBE dispatcher per worker
listening on langflow:cancel:*.
3. Prompt stream cleanup after cancel — the dispatcher now waits for the
bridge to drain the sentinel before triggering cleanup_job, so the
Redis stream + owner keys are deleted within milliseconds instead of
waiting for the 5-minute periodic cleanup grace.
4. signal_cancel raises on Redis failure — publish errors propagate to the
caller instead of silently returning 0. The cancel HTTP endpoint
catches and returns False so the client can retry.
5. Auth note in signal_cancel docstring — explicit note that callers must
verify authorization; the HTTP cancel endpoint already does via
_verify_job_ownership before calling through.
6. Structured cancel logging — INFO-level logs on publish, marker hit,
owned dispatch, foreign dispatch. _cancel_stats counters expose
{published, marker_hit, dispatched_owned, dispatched_foreign,
publish_errors} for ops/metrics.
7. redis-py version compatibility — _close_pubsub falls back from
aclose() to close(), handles both sync and async return values.
8. Fire-and-forget tasks hold strong references — _background_tasks set
keeps marker checks and post-cancel cleanups from being GC'd before
they run; each task self-removes via add_done_callback. stop() drains
the set on shutdown.
Tests: 24 passing (1 skipped due to timing); deep stress harness verified
6 scenarios against real Redis.
* feat(job_queue): propagate client disconnect via signal_cancel for cross-worker builds
Closes the remaining cross-worker passive-disconnect gap. Previously, when a
client closed its streaming connection on a non-owner worker, the producer
worker kept emitting events until the build completed naturally. Now the
disconnect handler in create_flow_response publishes a cross-worker cancel
via signal_cancel so the owning worker stops promptly.
- create_flow_response accepts optional queue_service + job_id kwargs and
uses them in on_disconnect for the cross-worker case (event_task is None).
Both are keyword-only and default to None, preserving the single-worker
contract.
- get_flow_events_response wires both through.
- New unit test test_cross_worker_disconnect_publishes_signal_cancel covers
the full pubsub propagation path with two services sharing a fake Redis.
- Tested end-to-end against real Redis with a two-worker harness: 10/10
scenarios pass, including the new disconnect propagation case.
* feat(job_queue): production-harden cross-worker cancel for multi-worker setups
Five complementary improvements that make the Redis-backed cancel/lifecycle
path safe to leave running unattended under real ops pressure.
A. Dispatcher reconnect loop with exponential backoff. _run_cancel_dispatcher
used to exit silently on any Redis-side error (broker restart, network
blip, listen() hiccup), leaving the worker permanently blind to
cross-worker cancels until process restart. The loop now reconnects with
capped backoff (max 30s), and a dispatcher_reconnects counter exposes the
event for monitoring.
B. Outer timeout on _post_cancel_cleanup. The background cleanup task could
hang indefinitely if cleanup_job stalled (Redis pathology during DELETE);
wrap in asyncio.wait_for and let periodic cleanup retry instead.
C. Public metrics_snapshot() on JobQueueService (memory + Redis backends)
plus GET /monitor/job_queue endpoint behind get_current_active_user.
Surfaces backend, active_jobs, bridge_count, consumer_wrapper_count,
background_task_count, cancel_dispatcher_running, and the full
cancel_stats counter set.
D. Deterministic rewrite of test_redis_service_signal_cancel_flushes_sentinel_to_consumer.
The previous version was skipped roughly half the time due to a fakeredis
timing race; the new version no-ops cleanup_job for the duration of the
assertion so the bridge XADD ordering can be observed reliably. No more
skipped tests in the suite (33 pass, 0 skip).
E. Polling-mode stale-client watchdog. Polling has no persistent connection
for on_disconnect, so abandoned polling builds previously ran to natural
completion even after the client gave up. New flow:
* touch_activity(job_id) writes langflow:activity:<job_id> on every poll
and on streaming-response open; consume_and_yield refreshes it every
10s while the connection is held.
* _run_polling_watchdog scans owned jobs every
LANGFLOW_REDIS_QUEUE_POLLING_WATCHDOG_INTERVAL_S (default 15s) and
publishes signal_cancel for jobs whose activity is older than
LANGFLOW_REDIS_QUEUE_POLLING_STALE_THRESHOLD_S (default 90s).
* Streaming clients are protected by the heartbeat refresh; threshold <=
0 disables the watchdog entirely.
* cleanup_job folds the activity key into the existing DEL round-trip
so successful builds clean up after themselves.
End-to-end coverage: 33 unit tests pass (up from 25, with the previous skip
now passing deterministically); a two-worker real-Redis harness exercises 12
scenarios including dispatcher reconnect, post-cancel timeout, watchdog
reclamation, and metrics_snapshot schema completeness.
* fix(job_queue): address PR review + Codex adversarial review findings
Acts on every must-fix finding from a multi-agent review pass (code-reviewer,
pr-test-analyzer, silent-failure-hunter, comment-analyzer) plus the Codex
adversarial review. No behavior is left to chance under realistic operational
conditions.
Critical fixes:
- Streaming heartbeat now runs as an INDEPENDENT task for the lifetime of the
response, not coupled to event yield cadence. A quiet build (long graph
step, slow LLM, no tokens for a while) previously had no heartbeat path, so
the polling watchdog could reclaim a live streaming client after the
threshold. The new heartbeat fires every streaming_activity_refresh_s
regardless of queue activity, and is cancelled cleanly on both
on_disconnect AND natural stream end.
- Watchdog start-grace window. A new in-memory _job_start_times[job_id]
timestamp is set in start_job() BEFORE super().start_job() so a job can
never appear in self._queues without a corresponding start time. When the
watchdog scans and the Redis activity key is missing, it skips the kill
until (now - start_ts) >= threshold. Without this, a slow background
touch_activity (event-loop pressure, Redis blip) could nuke a brand-new
build on the very first watchdog tick — the prior code's comment promised
the guard but didn't implement it.
- touch_activity errors are now observable. Replaces a blanket
contextlib.suppress(Exception) with an explicit try/except that bumps
activity_touch_errors and emits a debug log. CancelledError is no longer
swallowed. Operators get a signal when the heartbeat is silently failing.
- Watchdog UnboundLocalError closed. Initializes last = 0.0 before the
raw-is-None branch so a malformed activity value (parse failure) can't
crash the entire watchdog task. Adds activity_parse_errors and
activity_get_errors counters.
- Dispatcher except is split: ConnectionError/TimeoutError/OSError stay at
awarning (transient Redis), every other Exception goes to aerror with
exc_info=True AND increments dispatcher_internal_errors. Code bugs in
_handle_cancel no longer look identical to Redis disconnects.
- _post_cancel_cleanup narrows the bridge-wait suppress to TimeoutError
only, with a dedicated awarning naming the consequence ("cross-worker
consumers may see late end-of-stream"). Real bridge failures bubble up
via a separate Exception arm with awarning instead of being silent.
- Polling watchdog uses local _handle_cancel for owned jobs instead of
round-tripping through pubsub. Faster (no Redis RTT), dispatcher-
independent (works during a reconnect backoff), and keeps the
cancel_stats["published"] counter honest as a count of *external* cancels.
- /monitor/job_queue gated to get_current_active_superuser instead of
get_current_active_user — the snapshot exposes process-wide tenant
workload + cancel activity, which matters in multi-tenant deployments.
Docstring corrections:
- Removed the "cross-worker cancel is a known limitation" remnant from
RedisJobQueueService.get_queue_data — the limitation was closed in the
prior commit, and the new docstring tells callers exactly how cross-worker
cancel travels (signal_cancel → dispatcher).
- touch_activity docstring rewritten. The previous text described a
"TTL-based reasoning" fallback that didn't exist; the new text correctly
explains the 4x-threshold TTL and how it interacts with the watchdog's
start-time grace window.
New tests (5 added, total 38 pass):
- test_polling_watchdog_grants_start_grace_window: brand-new job with
deleted activity key survives until the threshold passes, then dies.
- test_polling_watchdog_skips_malformed_activity_value: malformed value
bumps activity_parse_errors and does NOT kill the job.
- test_streaming_heartbeat_runs_independent_of_event_yield: quiet streaming
build is not reclaimed; heartbeat task is cancelled on disconnect.
- test_concurrent_cancels_from_multiple_workers_are_idempotent: two workers
publish cancel simultaneously, no crashes, single observable cancel.
- test_dispatcher_internal_error_logged_at_error_level: bug in
_handle_cancel bumps dispatcher_internal_errors and dispatcher_reconnects;
the dispatcher task is still running afterwards.
Plus: test_metrics_snapshot_exposes_cancel_stats_and_counters now pins the
full cancel_stats key set (using set equality), so adding an increment
without registering the key — or vice versa — fails the test instead of
producing a silent KeyError in production.
End-to-end coverage: 38 unit tests pass; the two-worker real-Redis harness
passes all 12 scenarios on commit HEAD.
* fix: harden redis job queue cancellation
* feat(job_queue): export cancel_stats and active_jobs to OpenTelemetry
Wire the new Redis job queue counters into the existing langflow OTel
instrumentation so they flow to the Prometheus exporter when
LANGFLOW_PROMETHEUS_ENABLED=true. Until now cancel_stats lived only in
the JSON /monitor/job_queue snapshot — fine for ad-hoc inspection,
useless for ongoing ops dashboards or alerting.
Two new metrics are registered:
* langflow_job_queue_cancel_events_total (Counter, label: event_type) —
one Counter per distinct event_type covering all eleven cancel_stats
keys (published, marker_hit, dispatched_owned, dispatched_foreign,
publish_errors, dispatcher_reconnects, dispatcher_internal_errors,
polling_watchdog_kills, activity_touch_errors, activity_get_errors,
activity_parse_errors).
* langflow_job_queue_active_jobs (UpDownCounter, label: backend) —
bumped +1 in JobQueueService.create_queue and -1 in cleanup_job, so
both the in-memory ("memory") and Redis ("redis") backends report.
A single _bump_cancel_stat helper on RedisJobQueueService is now the
sole mutation point for the cancel_stats dict, guaranteeing the JSON
snapshot and Prometheus stay in lockstep.
The OTel emit path is best-effort: a cached lazy resolver fetches the
telemetry singleton once, and contextlib.suppress wraps the emit so a
broken or unavailable telemetry layer never propagates into the queue
hot path.
Tests cover dict + counter parity, active_jobs delta on create/cleanup,
silent failure when telemetry is unavailable, and that every
cancel_stats key routes through the helper.
* [autofix.ci] apply automated fixes
* test: update telemetry metric expectations
* fix: export job queue metrics to Prometheus
---------
Co-authored-by: Arek Mateusiak <arek@a4bits.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Hare <ericrhare@gmail.com>1 parent b9d207e commit 89a1643
4 files changed
Lines changed: 251 additions & 21 deletions
File tree
- src/backend
- base/langflow/services
- job_queue
- telemetry
- tests/unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
129 | 129 | | |
130 | 130 | | |
131 | 131 | | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
132 | 135 | | |
133 | 136 | | |
134 | 137 | | |
| |||
142 | 145 | | |
143 | 146 | | |
144 | 147 | | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
145 | 186 | | |
146 | 187 | | |
147 | 188 | | |
| |||
241 | 282 | | |
242 | 283 | | |
243 | 284 | | |
| 285 | + | |
244 | 286 | | |
245 | 287 | | |
246 | 288 | | |
| |||
436 | 478 | | |
437 | 479 | | |
438 | 480 | | |
439 | | - | |
| 481 | + | |
| 482 | + | |
440 | 483 | | |
441 | 484 | | |
442 | 485 | | |
| |||
810 | 853 | | |
811 | 854 | | |
812 | 855 | | |
| 856 | + | |
| 857 | + | |
813 | 858 | | |
814 | 859 | | |
815 | 860 | | |
| |||
880 | 925 | | |
881 | 926 | | |
882 | 927 | | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
883 | 938 | | |
884 | 939 | | |
885 | 940 | | |
| |||
1208 | 1263 | | |
1209 | 1264 | | |
1210 | 1265 | | |
1211 | | - | |
| 1266 | + | |
1212 | 1267 | | |
1213 | 1268 | | |
1214 | 1269 | | |
| |||
1248 | 1303 | | |
1249 | 1304 | | |
1250 | 1305 | | |
1251 | | - | |
| 1306 | + | |
1252 | 1307 | | |
1253 | 1308 | | |
1254 | 1309 | | |
| |||
1306 | 1361 | | |
1307 | 1362 | | |
1308 | 1363 | | |
1309 | | - | |
| 1364 | + | |
1310 | 1365 | | |
1311 | 1366 | | |
1312 | 1367 | | |
| |||
1324 | 1379 | | |
1325 | 1380 | | |
1326 | 1381 | | |
1327 | | - | |
| 1382 | + | |
1328 | 1383 | | |
1329 | 1384 | | |
1330 | 1385 | | |
| |||
1335 | 1390 | | |
1336 | 1391 | | |
1337 | 1392 | | |
1338 | | - | |
| 1393 | + | |
1339 | 1394 | | |
1340 | 1395 | | |
1341 | 1396 | | |
| |||
1383 | 1438 | | |
1384 | 1439 | | |
1385 | 1440 | | |
1386 | | - | |
| 1441 | + | |
1387 | 1442 | | |
1388 | 1443 | | |
1389 | 1444 | | |
| |||
1393 | 1448 | | |
1394 | 1449 | | |
1395 | 1450 | | |
1396 | | - | |
| 1451 | + | |
1397 | 1452 | | |
1398 | 1453 | | |
1399 | 1454 | | |
1400 | 1455 | | |
1401 | 1456 | | |
1402 | 1457 | | |
1403 | | - | |
1404 | | - | |
| 1458 | + | |
| 1459 | + | |
1405 | 1460 | | |
1406 | 1461 | | |
1407 | 1462 | | |
| |||
1418 | 1473 | | |
1419 | 1474 | | |
1420 | 1475 | | |
1421 | | - | |
| 1476 | + | |
1422 | 1477 | | |
1423 | 1478 | | |
1424 | 1479 | | |
| |||
1458 | 1513 | | |
1459 | 1514 | | |
1460 | 1515 | | |
1461 | | - | |
| 1516 | + | |
1462 | 1517 | | |
1463 | 1518 | | |
1464 | | - | |
| 1519 | + | |
1465 | 1520 | | |
1466 | 1521 | | |
1467 | 1522 | | |
| |||
1569 | 1624 | | |
1570 | 1625 | | |
1571 | 1626 | | |
1572 | | - | |
| 1627 | + | |
1573 | 1628 | | |
1574 | | - | |
| 1629 | + | |
1575 | 1630 | | |
1576 | 1631 | | |
1577 | 1632 | | |
| |||
Lines changed: 23 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
143 | 162 | | |
144 | 163 | | |
145 | 164 | | |
| |||
154 | 173 | | |
155 | 174 | | |
156 | 175 | | |
157 | | - | |
158 | | - | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
159 | 180 | | |
160 | 181 | | |
161 | 182 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2904 | 2904 | | |
2905 | 2905 | | |
2906 | 2906 | | |
| 2907 | + | |
| 2908 | + | |
| 2909 | + | |
| 2910 | + | |
| 2911 | + | |
| 2912 | + | |
| 2913 | + | |
| 2914 | + | |
| 2915 | + | |
| 2916 | + | |
| 2917 | + | |
| 2918 | + | |
| 2919 | + | |
| 2920 | + | |
| 2921 | + | |
| 2922 | + | |
| 2923 | + | |
| 2924 | + | |
| 2925 | + | |
| 2926 | + | |
| 2927 | + | |
| 2928 | + | |
| 2929 | + | |
| 2930 | + | |
| 2931 | + | |
| 2932 | + | |
| 2933 | + | |
| 2934 | + | |
| 2935 | + | |
| 2936 | + | |
| 2937 | + | |
| 2938 | + | |
| 2939 | + | |
| 2940 | + | |
| 2941 | + | |
| 2942 | + | |
| 2943 | + | |
| 2944 | + | |
| 2945 | + | |
| 2946 | + | |
| 2947 | + | |
| 2948 | + | |
| 2949 | + | |
| 2950 | + | |
| 2951 | + | |
| 2952 | + | |
| 2953 | + | |
| 2954 | + | |
| 2955 | + | |
| 2956 | + | |
| 2957 | + | |
| 2958 | + | |
| 2959 | + | |
| 2960 | + | |
| 2961 | + | |
| 2962 | + | |
| 2963 | + | |
| 2964 | + | |
| 2965 | + | |
| 2966 | + | |
| 2967 | + | |
| 2968 | + | |
| 2969 | + | |
| 2970 | + | |
| 2971 | + | |
| 2972 | + | |
| 2973 | + | |
| 2974 | + | |
| 2975 | + | |
| 2976 | + | |
| 2977 | + | |
| 2978 | + | |
| 2979 | + | |
| 2980 | + | |
| 2981 | + | |
| 2982 | + | |
| 2983 | + | |
| 2984 | + | |
| 2985 | + | |
| 2986 | + | |
| 2987 | + | |
| 2988 | + | |
| 2989 | + | |
| 2990 | + | |
| 2991 | + | |
| 2992 | + | |
| 2993 | + | |
| 2994 | + | |
| 2995 | + | |
| 2996 | + | |
| 2997 | + | |
| 2998 | + | |
| 2999 | + | |
| 3000 | + | |
| 3001 | + | |
| 3002 | + | |
| 3003 | + | |
| 3004 | + | |
| 3005 | + | |
| 3006 | + | |
| 3007 | + | |
| 3008 | + | |
| 3009 | + | |
| 3010 | + | |
| 3011 | + | |
| 3012 | + | |
| 3013 | + | |
| 3014 | + | |
| 3015 | + | |
| 3016 | + | |
| 3017 | + | |
| 3018 | + | |
| 3019 | + | |
| 3020 | + | |
| 3021 | + | |
| 3022 | + | |
0 commit comments