fix(arcade-core,arcade-mcp-server): Nested-object schemas across all tool-schema converters (#860)

## Summary

Arcade renders a tool's `ValueSchema` into JSON Schema in three
independent places, and they had drifted into the same class of bug:
nested object subschemas (especially `list[<TypedDict>]` items) were
emitted incompletely, producing OpenAI-strict-invalid schemas that the
OpenAI Responses API rejects with a hard 400. Because OpenAI rejects the
whole tool list when one schema is invalid, every eval/tool batch
containing that tool fails.

This PR fixes all three converter paths and the shared root cause they
all read from, so correct schemas come from one corrected source rather
than per-converter patches.

Resolves:
https://linear.app/arcadedev/issue/TOO-990/arcade-core-tool-schema-converters-drop-nested-object-fields-for

## What was wrong (three layers)

1. **`arcade_core/converters/openai.py`** (OpenAI strict) and
**`anthropic.py`** (standard) built array `items` from the inner scalar
type alone, emitting bare `{"type": "object"}` and dropping object
fields. OpenAI strict also requires `additionalProperties: false` +
every key in `required` on each object.
2. **`arcade_mcp_server/convert.py`** (MCP `inputSchema`) preserved
nested properties/required but never set `additionalProperties: false`
on nested objects or array-of-object items (only the top-level wrapper
did). This is the path consumers monkey-patch when feeding MCP tool
schemas to OpenAI strict mode.
3. **`arcade_core/catalog.py::extract_properties`** extracted
`required_keys`/`nullable` for **TypedDict** params but not for
**Pydantic** models, so every converter received imprecise data for
Pydantic object params (no `required`, no nullability).

## Changes

- **OpenAI + Anthropic converters** (commit 1): expand every object
recursively at any depth (top-level param, nested object,
array-of-object items). OpenAI strict closes every object
(`additionalProperties: false`), lists all keys in `required`, and
renders optionals as nullable unions; Anthropic uses standard JSON
Schema (only required keys).
- **MCP converter** (commit 2): set `additionalProperties: false` on
objects with a known shape, at every depth. Freeform `dict` params (no
properties) stay open, since closing them would reject all keys.
- **Root cause** (commit 2): `extract_properties` now populates
`required_keys` for Pydantic models via `FieldInfo.is_required()` and
marks `nullable` on `Optional` fields, matching the existing TypedDict
handling. All three converters get accurate metadata from one place.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes shared schema extraction and all three tool-schema emitters,
which can alter API/MCP validation behavior for nested
Pydantic/TypedDict params; mitigated by extensive new tests but still
user-visible for existing tools.
> 
> **Overview**
> Fixes **incomplete nested object JSON Schema** for tool inputs so
OpenAI strict, Anthropic, and MCP paths stay aligned and OpenAI no
longer 400s on invalid tool lists.
> 
> **`arcade_core/catalog`:** Pydantic params now get **`required_keys`**
and **`nullable`** like TypedDict; known shapes use **`({}, [])`**
instead of collapsing to `None`, so empty/zero-field models differ from
freeform **`dict`**. List and **`ValueSchema`** conversion propagate
nested shapes when **`properties` / `inner_properties` is not `None`**.
> 
> **OpenAI & Anthropic converters:** Shared **`_build_object_schema`**
recursively expands **`json`** and **`list[object]`** items (no more
bare **`{"type": "object"}`**). OpenAI strict closes objects, lists all
keys in **`required`**, and uses **`null`** unions (including enums);
Anthropic lists only real required keys and unions **`null`** on
nullable nested fields.
> 
> **MCP `convert`:** Known-shape objects and array-of-object items get
**`additionalProperties: false`** at every depth; unknown **`dict`**
shapes stay open.
> 
> Bumps **`arcade-core` 4.7.3** and **`arcade-mcp-server` 1.22.2** with
broad converter/catalog tests plus a cross-dialect consistency test.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
46dee1d. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Author: EricGustin

libs/arcade-core/arcade_core/catalog.py

@@ -775,12 +775,13 @@ def get_wire_type_info(_type: type) -> WireTypeInfo:
         inner_info = get_wire_type_info(inner_type)
         inner_wire_type = cast(InnerWireType, inner_info.wire_type)
 
-        # If inner type has properties (it's a complex object), propagate them
-        if inner_info.properties:
+        # If inner type has a known object shape (possibly empty), propagate it. A known-empty
+        # shape ({}) is distinct from None (unknown shape) so converters can close it.
+        if inner_info.properties is not None:
             inner_properties = inner_info.properties
             inner_required_keys = inner_info.required_keys
         # If inner type is array (nested arrays), propagate inner_properties
-        elif inner_info.inner_properties:
+        elif inner_info.inner_properties is not None:
             inner_properties = inner_info.inner_properties
             inner_required_keys = inner_info.inner_required_keys
     else:
@@ -860,30 +861,42 @@ def extract_properties(
     """
     Extract properties from TypedDict, Pydantic models, or other structured types.
 
-    Returns (properties, required_keys). required_keys is a sorted list of required
-    property names for TypedDict types, or None for other types.
+    Returns (properties, required_keys). For a known structured type (TypedDict or
+    Pydantic model) properties is a dict of its fields, empty when the type declares no
+    fields, and required_keys is a sorted list of required property names, empty when every
+    field is optional. A known-empty shape (no fields) therefore yields ``({}, [])``, which
+    converters can distinguish from a type with no known shape (e.g. a freeform ``dict``),
+    which yields ``(None, None)`` to signal that both shape and requiredness are unknown.
     """
     properties = {}
 
     # Handle Pydantic BaseModel
     if isinstance(type_to_check, type) and issubclass(type_to_check, BaseModel):
+        pydantic_required_keys: list[str] = []
         for field_name, field_info in type_to_check.model_fields.items():
             # Get the field type
             field_type = field_info.annotation
             if field_type is None:
                 continue
 
             # Handle Optional types (Union[T, None])
-            if is_strict_optional(field_type):
+            is_nullable = is_strict_optional(field_type)
+            if is_nullable:
                 # Extract the non-None type from Optional
                 field_type = next(arg for arg in get_args(field_type) if arg is not type(None))
 
             # Get wire type info recursively
             # field_type cannot be None here due to the check above
             wire_info = get_wire_type_info(field_type)
+            if is_nullable:
+                wire_info.nullable = True
             properties[field_name] = wire_info
 
-        return (properties or None, None)
+            # A Pydantic field is required when it has no default (default or factory).
+            if field_info.is_required():
+                pydantic_required_keys.append(field_name)
+
+        return (properties, sorted(pydantic_required_keys))
 
     # Handle TypedDict
     elif is_typeddict(type_to_check):
@@ -909,9 +922,8 @@ def extract_properties(
 
             properties[field_name] = wire_info
 
-        req = sorted(getattr(type_to_check, "__required_keys__", frozenset()))
-        required_keys = req or None  # normalize empty → None
-        return (properties or None, required_keys)
+        required_keys = sorted(getattr(type_to_check, "__required_keys__", frozenset()))
+        return (properties, required_keys)
 
     # Handle regular dict with type annotations (e.g., dict[str, Any])
     elif get_origin(type_to_check) is dict:
@@ -925,17 +937,18 @@ def wire_type_info_to_value_schema(wire_info: WireTypeInfo) -> ValueSchema:
     """
     Convert WireTypeInfo to ValueSchema, including nested properties.
     """
-    # Convert nested properties if they exist
+    # Convert nested properties if they exist. A known-empty object shape ({}) is preserved
+    # as {} rather than collapsed to None so converters can emit it as a closed object.
     properties = None
-    if wire_info.properties:
+    if wire_info.properties is not None:
         properties = {
             name: wire_type_info_to_value_schema(nested_info)
             for name, nested_info in wire_info.properties.items()
         }
 
     # Convert inner properties for array items
     inner_properties = None
-    if wire_info.inner_properties:
+    if wire_info.inner_properties is not None:
         inner_properties = {
             name: wire_type_info_to_value_schema(nested_info)
             for name, nested_info in wire_info.inner_properties.items()

libs/arcade-core/arcade_core/converters/anthropic.py

@@ -17,16 +17,16 @@
 class AnthropicInputSchemaProperty(TypedDict, total=False):
     """Type definition for a property within Anthropic input schema."""
 
-    type: str
-    """The JSON Schema type for this property."""
+    type: str | list[str]
+    """The JSON Schema type(s) for this property. A list expresses a union (e.g. ["string", "null"])."""
 
     description: str
     """Description of the property."""
 
     enum: list[Any]
     """Allowed values for enum properties."""
 
-    items: dict[str, Any]
+    items: "AnthropicInputSchemaProperty"
     """Schema for array items when type is 'array'."""
 
     properties: dict[str, "AnthropicInputSchemaProperty"]
@@ -112,6 +112,49 @@ def _create_tool_schema(
     return tool
 
 
+def _add_null_to_type(schema: AnthropicInputSchemaProperty) -> None:
+    """Union a field's type with "null" so a null value validates.
+
+    A nested field can be required yet nullable (``str | None`` with no default): it must
+    stay required while still accepting null. When the field is an enum, ``None`` is
+    appended to the enum too, since ``type`` and ``enum`` are independent assertions.
+    """
+    field_type = schema.get("type")
+    if isinstance(field_type, str):
+        schema["type"] = [field_type, "null"]
+    elif isinstance(field_type, list) and "null" not in field_type:
+        schema["type"] = [*field_type, "null"]
+
+    enum_values = schema.get("enum")
+    if isinstance(enum_values, list) and None not in enum_values:
+        schema["enum"] = [*enum_values, None]
+
+
+def _build_object_schema(
+    properties: dict[str, ValueSchema],
+    required_keys: list[str] | None,
+) -> AnthropicInputSchemaProperty:
+    """Build an object schema from a structured type's fields.
+
+    Anthropic uses standard JSON Schema: only the actually-required fields appear in
+    ``required`` and there is no ``additionalProperties`` constraint. A nullable field
+    unions ``null`` into its type so a null value validates even when the field is required.
+    """
+    field_schemas: dict[str, AnthropicInputSchemaProperty] = {}
+    for name, field_value_schema in properties.items():
+        field_schema = _convert_value_schema_to_json_schema(field_value_schema)
+        if field_value_schema.description:
+            field_schema["description"] = field_value_schema.description
+        if field_value_schema.nullable:
+            _add_null_to_type(field_schema)
+        field_schemas[name] = field_schema
+
+    schema: AnthropicInputSchemaProperty = {"type": "object", "properties": field_schemas}
+    if required_keys:
+        schema["required"] = list(required_keys)
+    return schema
+
+
 def _convert_value_schema_to_json_schema(
     value_schema: ValueSchema,
 ) -> AnthropicInputSchemaProperty:
@@ -125,28 +168,30 @@ def _convert_value_schema_to_json_schema(
         "array": "array",
     }
 
-    schema: AnthropicInputSchemaProperty = {"type": type_mapping[value_schema.val_type]}
+    schema: AnthropicInputSchemaProperty
 
     if value_schema.val_type == "array" and value_schema.inner_val_type:
-        items_schema: dict[str, Any] = {"type": type_mapping[value_schema.inner_val_type]}
-
-        # For arrays, enum should be applied to the items, not the array itself
-        if value_schema.enum:
-            items_schema["enum"] = value_schema.enum
-
-        schema["items"] = items_schema
-    else:
-        # Handle enum for non-array types
-        if value_schema.enum:
-            schema["enum"] = value_schema.enum
-
-    # Handle object properties
-    if value_schema.val_type == "json" and value_schema.properties:
-        schema["properties"] = {
-            name: _convert_value_schema_to_json_schema(nested_schema)
-            for name, nested_schema in value_schema.properties.items()
-        }
-
+        schema = {"type": "array"}
+        if value_schema.inner_val_type == "json" and value_schema.inner_properties is not None:
+            schema["items"] = _build_object_schema(
+                value_schema.inner_properties, value_schema.inner_required_keys
+            )
+        else:
+            items_schema: AnthropicInputSchemaProperty = {
+                "type": type_mapping[value_schema.inner_val_type]
+            }
+            # For arrays of scalars, enum applies to the items, not the array itself
+            if value_schema.enum:
+                items_schema["enum"] = value_schema.enum
+            schema["items"] = items_schema
+        return schema
+
+    if value_schema.val_type == "json" and value_schema.properties is not None:
+        return _build_object_schema(value_schema.properties, value_schema.required_keys)
+
+    schema = {"type": type_mapping[value_schema.val_type]}
+    if value_schema.enum:
+        schema["enum"] = value_schema.enum
     return schema
 
 

libs/arcade-core/arcade_core/converters/openai.py

@@ -25,7 +25,7 @@ class OpenAIFunctionParameterProperty(TypedDict, total=False):
     enum: list[Any]
     """Allowed values for enum properties."""
 
-    items: dict[str, Any]
+    items: "OpenAIFunctionParameterProperty"
     """Schema for array items when type is 'array'."""
 
     properties: dict[str, "OpenAIFunctionParameterProperty"]
@@ -136,6 +136,62 @@ def _create_tool_schema(
     return tool
 
 
+def _add_null_to_type(schema: OpenAIFunctionParameterProperty) -> None:
+    """Union a property's type with "null" so it may be omitted in strict mode.
+
+    Strict mode lists every property in ``required``; an optional property is then
+    expressed by allowing ``null`` rather than by leaving it out of ``required``. When the
+    property is an enum, ``None`` is appended to the enum too: ``type`` and ``enum`` are
+    independent assertions, so a null value must satisfy both.
+    """
+    param_type = schema.get("type")
+    if isinstance(param_type, str):
+        schema["type"] = [param_type, "null"]
+    elif isinstance(param_type, list) and "null" not in param_type:
+        schema["type"] = [*param_type, "null"]
+
+    enum_values = schema.get("enum")
+    if isinstance(enum_values, list) and None not in enum_values:
+        schema["enum"] = [*enum_values, None]
+
+
+def _build_object_schema(
+    properties: dict[str, ValueSchema],
+    required_keys: list[str] | None,
+) -> OpenAIFunctionParameterProperty:
+    """Build a strict-mode object schema from a structured type's fields.
+
+    Strict mode requires ``additionalProperties: false`` on every object and every
+    property listed in ``required``. Fields that are not actually required are unioned
+    with ``null`` so the model may omit them. When ``required_keys`` is a list (a known
+    shape, possibly empty) a field is required iff it appears there; when it is ``None``
+    (an unknown shape) the field's ``nullable`` flag decides.
+    """
+    required_set = set(required_keys or [])
+    field_schemas: dict[str, OpenAIFunctionParameterProperty] = {}
+
+    for name, field_value_schema in properties.items():
+        field_schema = _convert_value_schema_to_json_schema(field_value_schema)
+        if field_value_schema.description:
+            field_schema["description"] = field_value_schema.description
+
+        if required_keys is None:
+            is_required = not field_value_schema.nullable
+        else:
+            is_required = name in required_set
+        if field_value_schema.nullable or not is_required:
+            _add_null_to_type(field_schema)
+
+        field_schemas[name] = field_schema
+
+    return {
+        "type": "object",
+        "properties": field_schemas,
+        "required": list(properties.keys()),
+        "additionalProperties": False,
+    }
+
+
 def _convert_value_schema_to_json_schema(
     value_schema: ValueSchema,
 ) -> OpenAIFunctionParameterProperty:
@@ -149,28 +205,30 @@ def _convert_value_schema_to_json_schema(
         "array": "array",
     }
 
-    schema: OpenAIFunctionParameterProperty = {"type": type_mapping[value_schema.val_type]}
+    schema: OpenAIFunctionParameterProperty
 
     if value_schema.val_type == "array" and value_schema.inner_val_type:
-        items_schema: dict[str, Any] = {"type": type_mapping[value_schema.inner_val_type]}
-
-        # For arrays, enum should be applied to the items, not the array itself
-        if value_schema.enum:
-            items_schema["enum"] = value_schema.enum
-
-        schema["items"] = items_schema
-    else:
-        # Handle enum for non-array types
-        if value_schema.enum:
-            schema["enum"] = value_schema.enum
-
-    # Handle object properties
-    if value_schema.val_type == "json" and value_schema.properties:
-        schema["properties"] = {
-            name: _convert_value_schema_to_json_schema(nested_schema)
-            for name, nested_schema in value_schema.properties.items()
-        }
-
+        schema = {"type": "array"}
+        if value_schema.inner_val_type == "json" and value_schema.inner_properties is not None:
+            schema["items"] = _build_object_schema(
+                value_schema.inner_properties, value_schema.inner_required_keys
+            )
+        else:
+            items_schema: OpenAIFunctionParameterProperty = {
+                "type": type_mapping[value_schema.inner_val_type]
+            }
+            # For arrays of scalars, enum applies to the items, not the array itself
+            if value_schema.enum:
+                items_schema["enum"] = value_schema.enum
+            schema["items"] = items_schema
+        return schema
+
+    if value_schema.val_type == "json" and value_schema.properties is not None:
+        return _build_object_schema(value_schema.properties, value_schema.required_keys)
+
+    schema = {"type": type_mapping[value_schema.val_type]}
+    if value_schema.enum:
+        schema["enum"] = value_schema.enum
     return schema
 
 
@@ -192,21 +250,15 @@ def _convert_input_parameters_to_json_schema(
     for parameter in parameters:
         param_schema = _convert_value_schema_to_json_schema(parameter.value_schema)
 
-        # For optional parameters in strict mode, we need to add "null" as a type option
+        # In strict mode every parameter is listed in `required`; optional ones are
+        # expressed by allowing "null" rather than by omission.
         if not parameter.required:
-            param_type = param_schema.get("type")
-            if isinstance(param_type, str):
-                # Convert single type to union with null
-                param_schema["type"] = [param_type, "null"]
-            elif isinstance(param_type, list) and "null" not in param_type:
-                param_schema["type"] = [*param_type, "null"]
+            _add_null_to_type(param_schema)
 
         if parameter.description:
             param_schema["description"] = parameter.description
         properties[parameter.name] = param_schema
 
-        # In strict mode, all parameters (including optional ones) go in required array
-        # Optional parameters are handled by adding "null" to their type
         required.append(parameter.name)
 
     json_schema: OpenAIFunctionParameters = {

libs/arcade-core/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "arcade-core"
-version = "4.7.2"
+version = "4.7.3"
 description = "Arcade Core - Core library for Arcade platform"
 readme = "README.md"
 license = { text = "MIT" }