Skip to content

nemo_fabric.runtime.md

Latest commit

 

History

History
209 lines (108 loc) · 4.83 KB

nemo_fabric.runtime.md

File metadata and controls

209 lines (108 loc) · 4.83 KB
Raw
title Runtime
slug /reference/api/python-library-reference/runtime
description Drive stateful multi-turn execution through the Runtime API.

module nemo_fabric.runtime

Runtime lifecycle support for the NVIDIA NeMo Fabric Python SDK.

class RuntimeStatus

Lifecycle state of a runtime.

ACTIVE accepts invocations, STOPPED has released its runtime, and FAILED records a lifecycle failure that prevents further invocations but still permits cleanup.

Inheritance

Direct bases: str, Enum.

Values

The enum defines the following values:

Name Value
ACTIVE active
STOPPED stopped
FAILED failed

class Runtime

One logical, stateful harness execution.

Create runtimes with Fabric.start_runtime() rather than calling the constructor. A runtime serializes invocations and preserves adapter-owned harness state across turns. Use it as an asynchronous context manager to stop the runtime on exit.

Runtime-scoped overrides are recursively merged with invocation overrides; invocation values win.

property handle

Return a detached snapshot of the runtime handle.

property invocations

Return copied request, runtime, and invocation IDs for completed turns.

property messages

Return a deep copy of the latest harness-provided message history.

property runtime_id

Return the unique identifier for this started runtime lifecycle.

property status

Return the current ACTIVE, STOPPED, or FAILED state.

property supports_openai_streaming

Return whether the selected adapter implements native OpenAI streaming.

property supports_streaming

Return whether NVIDIA NeMo Relay ATOF streaming is enabled.

method invoke

async def invoke(*, input: Any = None, request: RunRequest | None = None) -> RunResult

Run one turn on this runtime.

input and request are mutually exclusive. Runtime overrides are merged below invocation overrides from RunRequest. Concurrent turns on the same runtime are rejected.

Args:

  • input: JSON-compatible turn input.
  • request: Complete validated RunRequest.

Returns: The normalized RunResult for this turn.

Raises:

  • FabricConfigError: If request fields conflict or are not JSON-compatible.
  • FabricStateError: If the runtime is not active, is stopping, or is already running a turn.
  • FabricNativeUnavailableError: If the native extension is missing.
  • FabricRuntimeError: If native invocation fails before returning a normalized result.

method invoke_openai_stream

def invoke_openai_stream(
    *,
    input: Any = None,
    request: RunRequest | None = None,
) -> OpenAIInvokeStream

Start one turn and stream native OpenAI chat-completion chunks.

The returned stream yields chat.completion.chunk mappings. Await stream.result() for the separate normalized terminal result.

Raises:

  • FabricCapabilityError: If the selected adapter does not advertise native OpenAI streaming.
  • FabricConfigError: If request fields conflict or are not JSON-compatible.
  • FabricStateError: If another turn or stream is active.

method invoke_stream

def invoke_stream(
    *,
    input: Any = None,
    request: RunRequest | None = None,
) -> InvokeStream

Start one turn and stream raw NeMo Relay ATOF records as they arrive.

input and request are mutually exclusive. The returned InvokeStream yields raw ATOF dictionaries. Await stream.result() for the terminal normalized RunResult.

Raises:

  • FabricCapabilityError: If the runtime was not started with NeMo Relay enabled and streaming=True.
  • FabricConfigError: If request fields conflict or are not JSON-compatible.
  • FabricStateError: If another turn or stream is active.

method stop

async def stop() -> None

Destroy an idle runtime exactly once.

Repeated calls after a successful stop are no-ops. A failed runtime may still be stopped so its resources are released.

Raises:

  • FabricStateError: If the runtime is already stopping or has an invocation in flight.
  • FabricNativeUnavailableError: If the native extension is missing.
  • FabricRuntimeError: If native runtime shutdown fails.

This file was automatically generated via lazydocs.