Skip to content

[receiver/sqlserver] Fix query sample dropping sessions blocked on schema locks - #49984

Merged
ChrsMark merged 7 commits into
open-telemetry:mainfrom
splunk-shanu:sqlserver-schema-lock-blocking-fix
Aug 5, 2026
Merged

[receiver/sqlserver] Fix query sample dropping sessions blocked on schema locks#49984
ChrsMark merged 7 commits into
open-telemetry:mainfrom
splunk-shanu:sqlserver-schema-lock-blocking-fix

Conversation

@splunk-shanu

@splunk-shanu splunk-shanu commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Description

The query sample template used CROSS APPLY sys.dm_exec_sql_text(r.plan_handle). For a session waiting on LCK_M_SCH_S (Schema Stability) or LCK_M_SCH_M (Schema Modification), sys.dm_exec_sql_text returns zero rows — because the plan-cache lookup itself needs schema stability that the wait is holding up. CROSS APPLY then eliminates the outer row entirely, so blocked sessions were silently omitted from db.server.query_sample events. Users lost visibility into all schema-related blocking (index rebuilds, ALTER TABLE, statistics updates during load).

This PR:

  • Changes CROSS APPLY to OUTER APPLY on sys.dm_exec_sql_text(r.plan_handle) so rows are preserved when the TVF returns nothing.
  • Adds OUTER APPLY sys.dm_exec_input_buffer(r.session_id, r.request_id) as a fallback source for statement_text. dm_exec_input_buffer reads directly from the client connection's TDS buffer and still works when plan_handle is unresolvable.
  • Wraps statement_text extraction in COALESCE(SUBSTRING(...), ib.event_info, '') so statement-level granularity (via statement_start_offset / statement_end_offset) is preserved for healthy sessions and only falls back to whole-batch text from input_buffer when the plan-cache lookup fails.
  • Makes the WHERE clause NULL-safe for both o.TEXT and ib.event_info so encrypted stored procedures and other zero-row scenarios are no longer silently dropped.

The same class of bug also affected encrypted procs (WITH ENCRYPTION), plans aged out of cache during long blocks, and pre-parse sessions — all fixed by the same change.

sys.dm_exec_input_buffer requires the same VIEW SERVER STATE permission the receiver already needs.

Link to tracking issue

Fixes #49983

Testing

  • Live-reproduced against SQL Server 2022 with two scenarios:
    • Session B waiting on LCK_M_SCH_S blocked by Session A holding LCK_M_SCH_M → confirmed row now emitted.
    • Session B waiting on LCK_M_SCH_M blocked by Session A holding LCK_M_SCH_S (via long-running SELECT with HOLDLOCK) → confirmed row now emitted.
    • Healthy multi-statement batch → confirmed statement_text still returns the specific waiting statement (e.g. WAITFOR DELAY '00:05:00'), not the whole batch — statement-level granularity preserved.
  • Updated existing golden test testQuerySampleQuery.txt to match the new query.
  • Added new regression test TestQuerySampleQueryDetectsSchemaLockBlocking that asserts each invariant needed to prevent this class of bug:
    • must not use CROSS APPLY sys.dm_exec_sql_text
    • must use OUTER APPLY sys.dm_exec_sql_text(r.plan_handle)
    • must reference sys.dm_exec_input_buffer
    • WHERE must be NULL-safe for both o.TEXT and ib.event_info
    • statement_text must use a COALESCE(...) chain with ib.event_info fallback
  • Verified the regression test fails against the pre-fix template with a clear diagnostic message pointing at the offending construct.
  • Full go test ./... in receiver/sqlserverreceiver passes.

Documentation

No user-facing documentation changes. documentation.md unchanged (no metric/attribute additions).

Authorship

  • I, a human, wrote this pull request description myself.

…hema locks

The query sample template used `CROSS APPLY sys.dm_exec_sql_text(r.plan_handle)`.
For a session waiting on `LCK_M_SCH_S` or `LCK_M_SCH_M`, `sys.dm_exec_sql_text`
returns zero rows because the plan-cache lookup itself needs schema stability
that the wait is holding up. `CROSS APPLY` then drops the outer row, so
blocked sessions were silently omitted from `db.server.query_sample` events.

Fix:
- Change `CROSS APPLY` to `OUTER APPLY` on `sys.dm_exec_sql_text(r.plan_handle)`
  so rows are preserved when the TVF returns nothing.
- Add `OUTER APPLY sys.dm_exec_input_buffer(r.session_id, r.request_id)` as a
  fallback source for statement_text. `dm_exec_input_buffer` reads directly
  from the connection buffer and works even when plan_handle is unresolvable.
- Wrap statement_text extraction in `COALESCE(SUBSTRING(...), ib.event_info, '')`
  so statement-level granularity is preserved for healthy sessions and only
  falls back to whole-batch text when plan_handle lookup fails.
- Make the `WHERE` clause NULL-safe for both `o.TEXT` and `ib.event_info`.

Also adds a targeted regression test `TestQuerySampleQueryDetectsSchemaLockBlocking`
that asserts each invariant needed to prevent this class of bug from recurring.

Fixes open-telemetry#49983

Assisted-by: Claude Sonnet 4.6
@splunk-shanu
splunk-shanu requested review from a team and crobert-1 as code owners July 31, 2026 08:13
@github-actions github-actions Bot added receiver/sqlserver Run Windows Enable running windows test on a PR labels Jul 31, 2026
@splunk-shanu
splunk-shanu marked this pull request as draft July 31, 2026 08:14
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Jul 31, 2026

Copy link
Copy Markdown

Pull request dashboard status

Merged · refreshed 2026-08-05 06:45 UTC

Status above doesn't look right?
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@sv-splunk sv-splunk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@splunk-shanu
splunk-shanu marked this pull request as ready for review August 3, 2026 11:47
@crobert-1 crobert-1 added ready to merge Code review completed; ready to merge by maintainers receiver/sqlserver and removed receiver/sqlserver labels Aug 4, 2026
@github-actions
github-actions Bot requested review from crobert-1 and sv-splunk August 4, 2026 23:29
@singhvibhanshu

Copy link
Copy Markdown
Member

/rerun

@ChrsMark ChrsMark left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, approved by code-owner. Will merge.

@ChrsMark
ChrsMark merged commit d408a35 into open-telemetry:main Aug 5, 2026
166 of 169 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready to merge Code review completed; ready to merge by maintainers receiver/sqlserver Run Windows Enable running windows test on a PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[receiver/sqlserver] Blocking sessions on schema locks (LCK_M_SCH_S / LCK_M_SCH_M) not reported

5 participants