Skip to main content

Understanding Union's Error Hierarchy

Understanding flyte-sdk's Error Hierarchy​

When a task fails in flyte-sdk, quickly identifying the root cause—whether it's an issue with your code, the system, or an unknown factor—is crucial for efficient debugging. The flyte-sdk provides a structured error hierarchy to categorize these failures, allowing for precise error handling and clearer diagnostics.

At the core of this system is the BaseRuntimeError class, defined in errors.py. This class serves as the foundational parent for all runtime errors within flyte-sdk, inheriting from Python's built-in RuntimeError.

class BaseRuntimeError(RuntimeError):
    """
    Base class for all Union runtime errors. These errors are raised when the underlying task execution fails, either
    because of a user error, system error or an unknown error.
    """

    def __init__(self, code: str, kind: ErrorKind, root_cause_message: str, worker: str | None = None):
        super().__init__(root_cause_message)
        self.code = code
        self.kind = kind
        self.worker = worker

Every BaseRuntimeError instance carries essential attributes:

  • code: A string identifier for the specific error type.
  • kind: An ErrorKind enum value (e.g., "user", "system", "unknown") that categorizes the error's origin.
  • root_cause_message: A detailed message explaining the error.
  • worker: An optional identifier for the worker that encountered the error.

This common structure allows you to catch any flyte-sdk runtime error generically, as demonstrated in the CLI's error handling in src/flyte/cli/_common.py:

try:
    # Some operation that might raise a BaseRuntimeError
    pass
except flyte.errors.BaseRuntimeError as e:
    raise click.ClickException(f"{e.kind} failure, {e.code}. {e}") from e

Categorizing Failures: User, System, and Unknown Errors​

To provide more granular insight into the nature of a failure, BaseRuntimeError is subclassed into more specific error types, each indicating a different source of the problem.

RuntimeUserError​

When a task fails due to an issue within your application code, flyte-sdk raises a RuntimeUserError. This error explicitly signals that the problem lies with the logic or data provided by the user.

class RuntimeUserError(BaseRuntimeError):
    """
    This error is raised when the underlying task execution fails because of an error in the user's code.
    """

    def __init__(self, code: str, message: str, worker: str | None = None):
        super().__init__(code, "user", message, worker)

Notice how RuntimeUserError initializes its base class with kind="user". A common scenario where this occurs is when any unhandled Python Exception is caught during task execution and wrapped into a RuntimeUserError. This ensures that all unhandled exceptions from user code are consistently categorized. For example, in src/flyte/_task.py:

try:
    # User task execution logic
    pass
except RuntimeSystemError:
    raise
except RuntimeUserError:
    raise
except Exception as e:
    raise RuntimeUserError(type(e).__name__, str(e)) from e

RuntimeSystemError​

RuntimeSystemError indicates a problem within the flyte-sdk system itself or a critical environmental misconfiguration. These errors are distinct from user errors and typically require attention to the flyte-sdk deployment or internal components.

class RuntimeSystemError(BaseRuntimeError):
    """
    This error is raised when the underlying task execution fails because of a system error. This could be a bug in the
    Union system or a bug in the user's code.
    """

    def __init__(self, code: str, message: str, worker: str | None = None):
        super().__init__(code, "system", message, worker)

This class initializes its base with kind="system". An example of a RuntimeSystemError being raised is when a critical component, such as the task controller, is not properly initialized, as seen in src/flyte/_task.py:

def some_function_context():
    controller = get_controller()
    if not controller:
        raise RuntimeSystemError("BadContext", "Controller is not initialized.")

RuntimeUnknownError​

In situations where the origin of an error cannot be definitively classified as either a user or system issue, flyte-sdk uses RuntimeUnknownError. This acts as a fallback for uncategorized or ambiguous failures.

class RuntimeUnknownError(BaseRuntimeError):
    """
    This error is raised when the underlying task execution fails because of an unknown error.
    """

    def __init__(self, code: str, message: str, worker: str | None = None):
        super().__init__(code, "unknown", message, worker)

This error sets its base kind to "unknown". While less specific, it ensures that no error goes unclassified within the flyte-sdk's runtime error framework.

Bridging External Errors: The Conversion Process​

flyte-sdk often interacts with external execution environments that report errors in their own formats, such as flyteidl.core.tasks_pb2.ExecutionError. The src/flyte/_internal/runtime/convert.py module is responsible for translating these external error representations into the appropriate BaseRuntimeError subclasses.

This conversion logic inspects the kind and code of the external error and instantiates the corresponding flyte-sdk error class. For instance, an execution_pb2.ExecutionError with a SYSTEM kind will be converted into a RuntimeSystemError:

def convert_execution_error(err, user_code):
    match err.kind:
        case execution_pb2.ExecutionError.UNKNOWN:
            return flyte.errors.RuntimeUnknownError(code=user_code, message=err.message, worker=err.worker)
        case execution_pb2.ExecutionError.USER:
            if "OOM" in err.code.upper():
                return flyte.errors.OOMError(code=user_code, message=err.message, worker=err.worker)
            elif "Interrupted" in err.code:
                return flyte.errors.TaskInterruptedError(code=user_code, message=err.message, worker=err.worker)
            elif "PrimaryContainerNotFound" in err.code:
                return flyte.errors.PrimaryContainerNotFoundError(
                    code=user_code, message=err.message, worker=err.worker
                )
            elif "RetriesExhausted" in err.code:
                return flyte.errors.RetriesExhaustedError(code=user_code, message=err.message, worker=err.worker)
            elif "Unknown" in err.code:
                return flyte.errors.RuntimeUnknownError(code=user_code, message=err.message, worker=err.worker)
            elif "InvalidImageName" in err.code:
                return flyte.errors.InvalidImageNameError(code=user_code, message=err.message, worker=err.worker)
            elif "ImagePullBackOff" in err.code:
                return flyte.errors.ImagePullBackOffError(code=user_code, message=err.message, worker=err.worker)
            return flyte.errors.RuntimeUserError(code=user_code, message=err.message, worker=err.worker)
        case execution_pb2.ExecutionError.SYSTEM:
            return flyte.errors.RuntimeSystemError(code=user_code, message=err.message, worker=err.worker)

This conversion mechanism is vital for maintaining a consistent error reporting structure within flyte-sdk, regardless of the underlying execution environment.

Handling Errors in the Task Runner​

The structured error hierarchy also simplifies error handling within the task execution lifecycle. The task runner, for instance, can specifically catch and log these categorized errors to provide clear diagnostics, as shown in src/flyte/_internal/runtime/taskrunner.py:

try:
    # Task execution logic
    pass
except RuntimeSystemError as e:
    logger.exception(f"Task failed with error: {e}")
    return {}, e
except RuntimeUnknownError as e:
    logger.exception(f"Task failed with error: {e}")
    return {}, e
except RuntimeUserError as e:
    logger.exception(f"Task failed with error: {e}")
    return {}, e

By categorizing errors with BaseRuntimeError and its subclasses, flyte-sdk provides a robust and transparent mechanism for understanding and debugging task failures, enabling developers to quickly pinpoint and resolve issues whether they originate from user code, the system, or an unknown source.