Skip to content

Commit f3ffe43

Browse files
committed
[Conformance] Report expected client terminal failures quietly
## Motivation and Context Negative auth scenarios such as `auth/resource-mismatch` and `auth/scope-retry-limit` are designed to drive the client into a terminal failure: the SDK correctly refuses a mismatched protected-resource metadata `resource` and stops after the scope step-up retry limit. The conformance harness asserts on the observed protocol exchange and already marks these scenarios as passed, but the client script had no top-level rescue, so each run leaked a raw Ruby backtrace to stderr inside a noisy "Client exited with code 1" block. This change wraps the connect and scenario dispatch in a rescue that reports the expected `AuthorizationError`, `RequestHandlerError`, and `Faraday::Error` outcomes as a concise one-line reason on stderr and exits with status 0. The harness echoes the client's output only when the exit status is non-zero and does not consult the exit status for its verdict, so the expected failures no longer produce any output block at all; the one-line reason remains visible in manual runs and in the harness `--output-dir` stderr artifact. Any unexpected error still raises with a full backtrace and a failing exit status so genuine SDK bugs remain visible. ## How Has This Been Tested? Ran `bundle exec rake conformance`: the server suite passes 40/40 and the client suite passes 280/280 with no "Client exited with code" or "Stderr:" blocks in the output. Exercised `auth/resource-mismatch` and `auth/scope-retry-limit` individually via `Conformance::ClientRunner`; both pass with clean output. ## Breaking Changes None. Only the conformance client script output changes; no library code is affected.
1 parent f9fd78e commit f3ffe43

1 file changed

Lines changed: 61 additions & 49 deletions

File tree

conformance/client.rb

Lines changed: 61 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -163,55 +163,67 @@ def build_provider_for(scenario, context)
163163
transport = MCP::Client::HTTP.new(url: server_url, oauth: oauth)
164164
client = MCP::Client.new(transport: transport)
165165
capabilities = scenario == "elicitation-sep1034-client-defaults" ? { elicitation: {} } : {}
166-
client.connect(
167-
client_info: { name: "ruby-sdk-conformance-client", version: MCP::VERSION },
168-
capabilities: capabilities,
169-
)
170-
171-
case scenario
172-
when "initialize"
173-
client.tools
174-
when "tools_call"
175-
tools = client.tools
176-
add_numbers = tools.find { |t| t.name == "add_numbers" }
177-
abort("Tool add_numbers not found") unless add_numbers
178-
client.call_tool(tool: add_numbers, arguments: { a: 1, b: 2 })
179-
when "sse-retry"
180-
# SEP-1699: the server closes the tools/call SSE stream right after a priming event.
181-
# The transport waits the server's `retry:` interval, reconnects with a GET carrying `Last-Event-ID`,
182-
# and receives the tool result on the resumed stream; the harness verifies the reconnect,
183-
# its timing, and the header.
184-
tools = client.tools
185-
test_reconnection = tools.find { |t| t.name == "test_reconnection" }
186-
abort("Tool test_reconnection not found") unless test_reconnection
187-
client.call_tool(tool: test_reconnection, arguments: {})
188-
when "elicitation-sep1034-client-defaults"
189-
# SEP-1034: the server sends `elicitation/create` on the tools/call SSE stream with every property declaring
190-
# a `default` and none required. The handler accepts and fills the omitted fields from those defaults.
191-
client.on_elicitation do |params|
192-
{ action: "accept", content: MCP::Client::Elicitation.apply_defaults(params["requestedSchema"] || {}) }
193-
end
166+
# The conformance harness asserts on the observed protocol exchange, not on the client's exit status,
167+
# and echoes the client's output only when the exit status is non-zero. Several negative auth scenarios
168+
# (e.g. `auth/resource-mismatch`, `auth/scope-retry-limit`) are expected to drive the client into
169+
# a terminal failure: the SDK correctly refuses a mismatched resource or stops after the scope step-up
170+
# retry limit. Report such an expected failure as a one-line reason and exit cleanly so the run stays
171+
# free of noise, while letting any unexpected error (a real SDK bug) still raise with a full backtrace
172+
# and a failing exit status.
173+
begin
174+
client.connect(
175+
client_info: { name: "ruby-sdk-conformance-client", version: MCP::VERSION },
176+
capabilities: capabilities,
177+
)
178+
179+
case scenario
180+
when "initialize"
181+
client.tools
182+
when "tools_call"
183+
tools = client.tools
184+
add_numbers = tools.find { |t| t.name == "add_numbers" }
185+
abort("Tool add_numbers not found") unless add_numbers
186+
client.call_tool(tool: add_numbers, arguments: { a: 1, b: 2 })
187+
when "sse-retry"
188+
# SEP-1699: the server closes the tools/call SSE stream right after a priming event.
189+
# The transport waits the server's `retry:` interval, reconnects with a GET carrying `Last-Event-ID`,
190+
# and receives the tool result on the resumed stream; the harness verifies the reconnect,
191+
# its timing, and the header.
192+
tools = client.tools
193+
test_reconnection = tools.find { |t| t.name == "test_reconnection" }
194+
abort("Tool test_reconnection not found") unless test_reconnection
195+
client.call_tool(tool: test_reconnection, arguments: {})
196+
when "elicitation-sep1034-client-defaults"
197+
# SEP-1034: the server sends `elicitation/create` on the tools/call SSE stream with every property declaring
198+
# a `default` and none required. The handler accepts and fills the omitted fields from those defaults.
199+
client.on_elicitation do |params|
200+
{ action: "accept", content: MCP::Client::Elicitation.apply_defaults(params["requestedSchema"] || {}) }
201+
end
194202

195-
tools = client.tools
196-
defaults_tool = tools.find { |t| t.name == "test_client_elicitation_defaults" }
197-
198-
abort("Tool test_client_elicitation_defaults not found") unless defaults_tool
199-
200-
client.call_tool(tool: defaults_tool, arguments: {})
201-
when %r|\Aauth/|
202-
# Auth-only scenarios: the protocol-level checks (PRM/AS metadata, DCR, PKCE, token usage)
203-
# are observed by the conformance server during `connect` and the subsequent request below.
204-
# Listing tools forces a second authenticated MCP request so the bearer token usage check fires.
205-
tools = client.tools
206-
207-
# `auth/scope-step-up` only fires its escalation 403 on `tools/call`, not `tools/list`,
208-
# so the client must actually invoke a tool to drive the second authorization request
209-
# the scenario asserts on.
210-
if scenario == "auth/scope-step-up"
211-
tool = tools.find { |t| t.name == "test-tool" } || tools.first
212-
abort("No tool exposed by conformance server for #{scenario}") unless tool
213-
client.call_tool(tool: tool, arguments: {})
203+
tools = client.tools
204+
defaults_tool = tools.find { |t| t.name == "test_client_elicitation_defaults" }
205+
206+
abort("Tool test_client_elicitation_defaults not found") unless defaults_tool
207+
208+
client.call_tool(tool: defaults_tool, arguments: {})
209+
when %r|\Aauth/|
210+
# Auth-only scenarios: the protocol-level checks (PRM/AS metadata, DCR, PKCE, token usage)
211+
# are observed by the conformance server during `connect` and the subsequent request below.
212+
# Listing tools forces a second authenticated MCP request so the bearer token usage check fires.
213+
tools = client.tools
214+
215+
# `auth/scope-step-up` only fires its escalation 403 on `tools/call`, not `tools/list`,
216+
# so the client must actually invoke a tool to drive the second authorization request
217+
# the scenario asserts on.
218+
if scenario == "auth/scope-step-up"
219+
tool = tools.find { |t| t.name == "test-tool" } || tools.first
220+
abort("No tool exposed by conformance server for #{scenario}") unless tool
221+
client.call_tool(tool: tool, arguments: {})
222+
end
223+
else
224+
abort("Unknown or unsupported scenario: #{scenario}")
214225
end
215-
else
216-
abort("Unknown or unsupported scenario: #{scenario}")
226+
rescue MCP::Client::OAuth::Flow::AuthorizationError, MCP::Client::RequestHandlerError, Faraday::Error => e
227+
warn("#{scenario}: client ended in the expected terminal failure (#{e.class}: #{e.message})")
228+
exit
217229
end

0 commit comments

Comments
 (0)