Handling Aborts and Interruptions

Two distinct stopping signals

A stopped Flyte action can reach the runtime through two different paths, and RunAbortedError and TaskInterruptedError are not interchangeable. The remote controller uses RunAbortedError for a completed action whose phase is PHASE_ABORTED; it uses TaskInterruptedError when an execution error is classified from a server error code containing Interrupted.

Both exceptions are runtime user errors:

import flyte

aborted = flyte.errors.RunAbortedError("the run was stopped")
interrupted = flyte.errors.TaskInterruptedError(
    code="TaskInterrupted",
    message="the underlying task execution was interrupted",
    worker="worker-1",
)

BaseRuntimeError in errors.py supplies the shared RuntimeError behavior and metadata. Its constructor stores the root-cause message as the normal exception message, and exposes code, kind, and worker attributes. RuntimeUserError fixes kind to "user" and accepts (code, message, worker=None), which is the inherited constructor used by TaskInterruptedError.

Run-aborted actions

When a nested remote action returns run_definition_pb2.PHASE_ABORTED, the remote controller raises RunAbortedError with both the child action and the current action in its message. This check occurs before the timeout and ordinary failure branches in _internal/controllers/remote/_controller.py:

        # If the action is aborted, we should abort the controller as well
        if n.phase == run_definition_pb2.PHASE_ABORTED:
            logger.warning(f"Action {n.action_id.name} was aborted, aborting current Action{current_action_id.name}")
            raise flyte.errors.RunAbortedError(
                f"Action {n.action_id.name} was aborted, aborting current Action {current_action_id.name}"
            )

Construct this exception with one argument, the message:

raise flyte.errors.RunAbortedError("Action child was aborted, aborting current Action parent")

Its constructor hard-codes the error code to RunAbortedError and passes the user classification through RuntimeUserError. In the resulting object, str(error) is the supplied message and .kind is "user". The implementation also leaves .worker set to "user", because RunAbortedError.__init__ passes that literal as the third argument to RuntimeUserError.

The remote controller treats an aborted phase separately from client-side cancellation. If waiting for submit_action raises asyncio.CancelledError, it calls cancel_action(action) and re-raises the cancellation:

        except asyncio.CancelledError:
            # If the action is cancelled, we need to cancel the action on the server as well
            logger.info(f"Action {action.action_id.name} cancelled, cancelling on server")
            await self.cancel_action(action)
            raise

Thus, cancellation of the waiting operation does not become RunAbortedError, while a server notification with PHASE_ABORTED does.

Interrupted task execution

TaskInterruptedError has no own __init__ method. If you construct it directly, provide the inherited RuntimeUserError arguments: code, message, and optionally worker.

In normal runtime operation, _internal/runtime/convert.py creates it while translating a protobuf execution_pb2.ExecutionError:

def convert_error_to_native(err: execution_pb2.ExecutionError | Exception | Error) -> Exception | None:
    if not err:
        return None

    if isinstance(err, Exception):
        return err

    if isinstance(err, Error):
        err = err.err

    user_code, server_code = _clean_error_code(err.code)
    match err.kind:
        case execution_pb2.ExecutionError.UNKNOWN:
            return flyte.errors.RuntimeUnknownError(code=user_code, message=err.message, worker=err.worker)
        case execution_pb2.ExecutionError.USER:
            if "OOM" in err.code.upper():
                return flyte.errors.OOMError(code=user_code, message=err.message, worker=err.worker)
            elif "Interrupted" in err.code:
                return flyte.errors.TaskInterruptedError(code=user_code, message=err.message, worker=err.worker)

The conversion requires both conditions:

• err.kind must be execution_pb2.ExecutionError.USER.
• The original err.code must contain the case-sensitive substring "Interrupted".

The converter passes the cleaned code, server message, and worker to the native exception. A user-kind error that does not match one of the specialized checks falls through to RuntimeUserError instead. In particular, "interrupted" with a lowercase i does not select TaskInterruptedError; the interruption check is not case-insensitive.

Because the class inherits its constructor, this is valid:

error = flyte.errors.TaskInterruptedError(
    code="TaskInterrupted",
    message="execution interrupted",
    worker="worker-1",
)

This is not the same constructor shape as RunAbortedError; omitting code raises Python's TypeError for the inherited signature:

# Do not use this constructor shape for TaskInterruptedError.
error = flyte.errors.TaskInterruptedError("execution interrupted")

Propagation through controllers and task execution

The conversion path is shared by local and remote execution. For local dispatch, _internal/controllers/_local_controller.py converts the returned error and raises the resulting native exception:

        if err:
            exc = convert.convert_error_to_native(err)
            if exc:
                raise exc
            else:
                raise flyte.errors.RuntimeSystemError("BadError", "Unknown error")

The remote controller's ordinary action-failure path similarly calls convert_error_to_native after obtaining the action error, so a server-reported interruption can propagate as TaskInterruptedError. A phase-based PHASE_ABORTED result does not use that path: the remote phase check raises RunAbortedError first.

At the task-runtime boundary, _internal/runtime/taskrunner.py catches both classes through their RuntimeUserError superclass. It returns an empty output mapping and the exception rather than converting it to a generic exception:

    except RuntimeUserError as e:
        logger.exception(f"Task failed with error: {e}")
        return {}, e

The finally block runs for either error and finalizes the parent action:

    finally:
        logger.info(f"Parent task finalized {tctx.action}")
        # reconstruct run id here
        await controller.finalize_parent_action(tctx.action)

The resulting flow is:

remote PHASE_ABORTED ───────────────▶ RunAbortedError
server USER + code containing        ▶ TaskInterruptedError
  exact "Interrupted"
                                      │
local/remote controller raises       ▼
                                      run_task catches RuntimeUserError
                                      ├─ returns ({}, exception)
                                      └─ finalizes parent action

Task execution also re-raises RuntimeUserError subclasses unchanged, so these typed errors retain their identity as they pass through task invocation.

Choosing the relevant error

Condition observed by the runtime	Result in flyte-sdk	Handling path
The remote wait is cancelled with asyncio.CancelledError	asyncio.CancelledError is re-raised	The remote controller calls cancel_action(action) first.
A submitted child action reports PHASE_ABORTED	RunAbortedError	The remote controller raises it before timeout or failure conversion.
A server execution error has user kind and its code contains exact "Interrupted"	TaskInterruptedError	convert_error_to_native preserves the cleaned code, message, and worker.
A submitted child action reports PHASE_TIMED_OUT	TaskTimeoutError	The remote controller handles this in the timeout branch, separately from aborts.

When you handle failures at the run_task boundary, account for the return shape: user runtime failures are returned as ({}, exception), and parent-action finalization still occurs. Do not assume that RunAbortedError or TaskInterruptedError must escape directly from run_task.

Implementation caveats

• Use RunAbortedError(message) for the phase-based abort type; its code is fixed by the class.
• Use the inherited (code, message, worker=None) signature for TaskInterruptedError; the converter normally supplies these values from ExecutionError metadata.
• Preserve the capitalization of the server error code if you depend on automatic interruption classification: the source checks "Interrupted" in err.code.
• Do not interpret the two exception types as equivalent: one describes an aborted action phase, while the other describes an execution error classified as interrupted.
• RunAbortedError has the unusual .worker == "user" value because of its constructor's positional call into RuntimeUserError.
• The checkout contains production integration code for these paths but no dedicated test_*.py, *_test.py, or example files for these exceptions.