Interpreting System and Platform Errors

When a remote workflow run fails before task code begins executing, or fails with a connectivity timeout to the control plane, the error does not originate from user logic. flyte-sdk distinguishes infrastructure and transport failures from user code exceptions through a dedicated error taxonomy defined in errors.py.

Understanding how platform and system errors are classified, raised, and translated helps diagnose configuration issues, network timeouts, and lifecycle states in Flyte and Union deployments.

---

The BaseRuntimeError Hierarchy

All runtime execution errors in flyte-sdk inherit from BaseRuntimeError (which subclasses Python's standard RuntimeError), except ActionNotFoundError which inherits directly from RuntimeError.

RuntimeError
├── BaseRuntimeError
│   ├── InitializationError
│   ├── LogsNotYetAvailableError
│   ├── RuntimeSystemError
│   │   └── UnionRpcError
│   ├── RuntimeUserError
│   └── RuntimeUnknownError
└── ActionNotFoundError

Inspecting Error Attributes

Instances of BaseRuntimeError provide structured attributes identifying the category and cause of failure:

• code (str): A machine-readable string describing the failure condition (for example, "ClientNotInitializedError", "SystemUnavailableError", or "LogsNotYetAvailable").
• kind (Literal["system", "unknown", "user"]): The error origin category.
  • "system": Platform, infrastructure, or network failures.
  • "user": Invalid inputs, out-of-memory errors, task timeouts, or missing configurations.
  • "unknown": Unclassified execution failures.
• root_cause_message (str): The primary error message passed to the underlying RuntimeError.
• worker (str | None): The worker or pod identifier where the failure occurred, if available.

import flyte
from flyte.errors import BaseRuntimeError, RuntimeSystemError

try:
    # Trigger a remote action or run inspection
    run = flyte.get_run("my-run-id")
    run.wait()
except RuntimeSystemError as e:
    print(f"Platform failure [{e.kind}]: {e.code}")
    print(f"Message: {e}")
    if e.worker:
        print(f"Occurred on worker: {e.worker}")
except BaseRuntimeError as e:
    print(f"Runtime error code: {e.code}, kind: {e.kind}")

---

Client and Context Setup Failures: InitializationError

InitializationError is raised when SDK components or remote operations are invoked before establishing the required connection or execution context.

Common Symptoms and Error Codes

Error Code	Trigger Condition	Common Location
ClientNotInitializedError	Calling remote API methods (such as get_client(), Logs.tail(), or run.wait()) before calling flyte.init().	_initialize.py:get_client, _initialize.py:ensure_client
StorageNotInitializedError	Invoking functions decorated with @requires_storage when storage backends are unconfigured.	_initialize.py:requires_storage
NotInitConfiguredError	Accessing operations guarded by @requires_initialization without global config.	_initialize.py:requires_initialization
ConfigFileNotFoundError	Specifying a config_file path in flyte.init(config_file=...) that does not exist.	_initialize.py:init
MissingEndpointOrApiKeyError	Initializing remote mode without supplying either endpoint or api_key.	_initialize.py:init
EndpointUnavailable	Remote data endpoints return gRPC status UNAVAILABLE when fetching signed upload/download URLs.	remote/_data.py:get_signed_urls

Resolving Initialization Failures

To avoid ClientNotInitializedError or MissingEndpointOrApiKeyError, initialize flyte-sdk with a valid endpoint or API key prior to calling remote services:

import flyte

# Explicit initialization with endpoint and credentials
flyte.init(
    endpoint="dns:///flyte.example.com:443",
    project="my-project",
    domain="development",
)

# Or initialize using a Union/Flyte configuration file
flyte.init(config_file="~/.flyte/config.yaml")

Decorators such as @requires_initialization, @requires_storage, and @requires_upload_location in _initialize.py automatically guard SDK operations by verifying _get_init_config() before execution:

# Internal check inside flyte-sdk
def ensure_client():
    if _get_init_config() is None or _get_init_config().client is None:
        raise InitializationError(
            "ClientNotInitializedError",
            "user",
            "Client has not been initialized. Call flyte.init() with a valid endpoint"
            " or api-key before using this function.",
        )

Signature Note: Unlike RuntimeSystemError, InitializationError does not override __init__. When instantiating it directly, pass all four BaseRuntimeError parameters: code, kind, root_cause_message, and optional worker.

---

Platform and Network Failures: RuntimeSystemError & UnionRpcError

RuntimeSystemError represents execution failures caused by the underlying system infrastructure rather than user task logic. It sets kind="system" by default. UnionRpcError subclasses RuntimeSystemError specifically for RPC transport failures communicating with control plane services.

class RuntimeSystemError(BaseRuntimeError):
    def __init__(self, code: str, message: str, worker: str | None = None):
        super().__init__(code, "system", message, worker)


class UnionRpcError(RuntimeSystemError):
    """Raised when communication with the Union server fails."""

Transport and Service Outages

When remote controllers and client methods communicate with gRPC services (such as run creation in _run.py or data transfers in remote/_data.py), transport errors from grpc.aio.AioRpcError are caught and mapped:

• grpc.StatusCode.UNAVAILABLE: Converted into RuntimeSystemError("SystemUnavailableError", "Flyte system is currently unavailable...").
• grpc.StatusCode.NOT_FOUND / PERMISSION_DENIED: Converted into RuntimeSystemError("NotFound", ...) or RuntimeSystemError("PermissionDenied", ...) when querying data signatures or remote objects.
• Unhandled gRPC transport codes: Converted into RuntimeSystemError("RunCreationError", f"Failed to create run: {e.details()}") or RuntimeSystemError(e.code().value, ...).

from flyte.errors import RuntimeSystemError, UnionRpcError

try:
    # Trigger remote execution
    run = my_task.run(x=10)
except RuntimeSystemError as e:
    if e.code == "SystemUnavailableError":
        # Handle backend outage or unreachable gRPC endpoint
        print("Control plane unreachable. Verify network connectivity and endpoint configuration.")
    elif e.code == "PermissionDenied":
        print("Check role authorization and API key permissions.")
    else:
        raise

Internal Engine Error Mapping

In _internal/runtime/convert.py, execution errors from engine protocol buffers (execution_pb2.ExecutionError) are mapped back to native Python errors:

# Protobuf execution error conversion in convert.py
if err.kind == execution_pb2.ExecutionError.SYSTEM:
    return flyte.errors.RuntimeSystemError(
        code=user_code,
        message=err.message,
        worker=err.worker,
    )

Internal system errors also arise during task execution lifecycle checks:

• "BadContext": Controller or task context was not initialized when action tracking was triggered (_task.py, _internal/controllers/remote/_controller.py).
• "InformerWatchFailure": Background controller watch loop encountered an unrecoverable worker thread failure (_internal/controllers/remote/_core.py).
• "UploadFailed": Remote controller failed to upload serialized action outputs or artifacts to blob storage.

---

Operational and Lifecycle Errors

Polling Logs Before Pod Start: LogsNotYetAvailableError

When tailing execution logs via Logs.tail() in remote/_logs.py, flyte-sdk queries the remote TailLogs gRPC service. When a container is still provisioning or waiting for pod initialization, the service returns grpc.StatusCode.NOT_FOUND.

Logs.tail retries this request (configured by retry=5 by default, polling every 2 seconds). If the retries are exhausted without finding an active log stream, flyte-sdk raises LogsNotYetAvailableError:

class LogsNotYetAvailableError(BaseRuntimeError):
    def __init__(self, message: str):
        super().__init__("LogsNotYetAvailable", "system", message, None)

Handling LogsNotYetAvailableError during log tailing:

from flyte.errors import LogsNotYetAvailableError
from flyte.remote import Logs

try:
    # Stream logs for an action attempt
    async for line in Logs.tail(action_id=action_id, attempt=1, retry=3):
        print(line.message)
except LogsNotYetAvailableError:
    print("Execution container is still starting up; logs are not yet available.")

Nonexistent Action Queries: ActionNotFoundError

ActionNotFoundError is raised when referencing or querying outputs for an action identifier that does not exist in the execution context or local trace datastore:

class ActionNotFoundError(RuntimeError):
    """
    This error is raised when the user tries to access an action that does not exist.
    """

Because ActionNotFoundError inherits directly from RuntimeError rather than BaseRuntimeError, catch it as a standard RuntimeError or by its specific class:

from flyte.errors import ActionNotFoundError

try:
    outputs = controller.get_action_outputs(interface, func, *args, **kwargs)
except ActionNotFoundError:
    print("Requested action trace or output was not found in the controller store.")

---

Summary of Error Handling Strategies

Error Class	Origin	Key Attributes	Typical Remediation
InitializationError	Client/SDK	kind="user" or "system", code="ClientNotInitializedError"	Call flyte.init(endpoint=...) with valid endpoint/credentials before remote calls.
RuntimeSystemError	Platform / Infrastructure	kind="system", code="SystemUnavailableError", worker	Check service status, network firewall/proxy rules, or controller logs.
UnionRpcError	Transport	kind="system", code	Inspect gRPC channel connectivity and retry policies.
LogsNotYetAvailableError	Log Service	kind="system", code="LogsNotYetAvailable"	Wait for pod initialization before tailing logs or increase retry iterations.
ActionNotFoundError	Execution Engine	Inherits from RuntimeError (no code / kind)	Verify the action name, sub-action path, and execution attempt.