Handling Type Transformation Errors

What TypeTransformerFailedError means

A task can have the right Python return annotation but still fail when its value is turned into a Flyte Literal: a File return value must be a File object, and a File literal received during deserialization must describe a single-part blob. flyte.types.TypeTransformerFailedError is the public exception for these transformer-level failures.

from flyte.types import TypeTransformerFailedError

flyte.types re-exports the class from types._type_engine. The class has no methods or custom state:

class TypeTransformerFailedError(TypeError, AssertionError, ValueError): ...

It is therefore catchable as any of those three built-in exception categories. It does not perform conversion; TypeEngine selects a transformer and the transformer raises it when checks or conversion logic fail.

The conversion pipeline

Flyte's type system represents runtime values as protobuf-based Literal objects and Python annotations as LiteralType definitions. The relevant flow is:

Python annotation -> TypeEngine.to_literal_type -> transformer.get_literal_type
Python value + LiteralType -> TypeEngine.to_literal
  -> transformer.assert_type (when enabled) -> transformer.to_literal -> Literal
Literal + expected Python type -> TypeEngine.to_python_value
  -> unwrap offloaded literal -> transformer.to_python_value -> Python value

TypeEngine.to_literal_type(python_type) resolves a transformer and calls get_literal_type. TypeEngine.to_literal resolves it again, invokes assert_type when type_assertions_enabled is true, awaits to_literal, and normalizes remote URIs:

transformer = cls.get_transformer(python_type)

if transformer.type_assertions_enabled:
    transformer.assert_type(python_type, python_val)

lv = await transformer.to_literal(python_val, python_type, expected)
modify_literal_uris(lv)
return lv

This ordering matters: a value can fail before serialization starts. A custom transformer can disable the engine-level assertion by constructing TypeTransformer with enable_type_assertions=False, while retaining control in its own conversion methods.

In the reverse direction, TypeEngine.to_python_value first calls unwrap_offloaded_literal when offloaded_metadata is present. That method downloads and loads the referenced protobuf before the transformer runs, so storage/download failures occur before a transformer-specific error.

Native value and declared-type mismatches

Compare the runtime value, declared Python type, and selected LiteralType. The base TypeTransformer.assert_type rejects a value that is not an instance of a declared non-generic type:

if not hasattr(t, "__origin__") and not isinstance(v, t):
    raise TypeTransformerFailedError(f"Expected value of type {t} but got '{v}' of type {type(v)}")

Concrete transformers can be stricter. SimpleTransformer.to_literal requires exact type equality, not merely a compatible subclass:

if type(python_val) is not self._type:
    raise TypeTransformerFailedError(
        f"Expected value of type {self._type} but got '{python_val}' of type {type(python_val)}"
    )

Lists and dictionaries

ListTransformer supports a univariate generic list such as typing.List[int]. Its serialization path requires exactly a list and recursively converts each element using the declared subtype:

if type(python_val) is not list:
    raise TypeTransformerFailedError("Expected a list")

t = self.get_sub_type(python_type)
lit_list = [TypeEngine.to_literal(x, t, expected.collection_type) for x in python_val]

A list annotation without a usable element type fails while its literal type is built, and deserialization requires the literal to contain a collection.

DictTransformer maps typed Dict[str, T] values to Flyte maps and requires string keys. Other dictionaries use a MessagePack-backed struct representation. The runtime check is:

if type(python_val) is not dict:
    raise TypeTransformerFailedError("Expected a dict")

A non-string map key raises ValueError("Flyte MapType expects all keys to be strings"), rather than the marker exception. A map literal without usable subtype hints, or with a destination key type other than str, raises a type-mismatch TypeError during deserialization.

Literal shape, blobs, and binary formats

Deserialization failures often mean that the protobuf shape does not match the expected Python type. FileTransformer requires a File and emits a SINGLE blob; it rejects missing blobs and multipart blobs when reading:

if not isinstance(python_val, File):
    raise TypeTransformerFailedError(f"Expected File object, received {type(python_val)}")

if not lv.scalar.HasField("blob"):
    raise TypeTransformerFailedError(f"Expected blob literal, received {lv}")
if not lv.scalar.blob.metadata.type.dimensionality == types_pb2.BlobType.BlobDimensionality.SINGLE:
    raise TypeTransformerFailedError(
        f"Expected single part blob, received {lv.scalar.blob.metadata.type.dimensionality}"
    )

DirTransformer has the parallel contract: it requires a Dir instance and a MULTIPART blob. A valid blob with the opposite dimensionality is still a transformation failure.

MessagePack-backed transformers accept the binary tag msgpack; another tag follows the base implementation's failure path:

if binary_idl_object.tag == MESSAGEPACK:
    # Decode the MessagePack bytes.
    ...
else:
    raise TypeTransformerFailedError(f"Unsupported binary format `{binary_idl_object.tag}`")

The same unsupported-format behavior appears in SimpleTransformer, the dataclass transformer, and the Pydantic transformer. Check the literal's tag before investigating the decoded payload.

Structured dataset handling has a specific native-value diagnostic. Its transformer extracts column and format annotations, determines the dataset format, and then checks a declared DataFrame type:

if issubclass(python_type, DataFrame) and not isinstance(python_val, DataFrame):
    raise TypeTransformerFailedError(
        f"Expected a DataFrame instance, but got {type(python_val)} instead."
        f" Did you forget to wrap your dataframe in a DataFrame instance?"
    )

Use the Flyte DataFrame wrapper when the declared type requires it; passing the underlying dataframe object reaches this check.

Serialization failures and dictionary pickle fallback

DictTransformer.dict_to_binary_literal first calls MessagePackEncoder(python_type).encode(v). On TypeError, the default is a TypeTransformerFailedError containing the value and encoder message. Its core behavior is:

try:
    encoder = MessagePackEncoder(python_type)
    msgpack_bytes = encoder.encode(v)
    return Literal(scalar=Scalar(binary=Binary(value=msgpack_bytes, tag=MESSAGEPACK)))
except TypeError as e:
    if allow_pickle:
        remote_path = await FlytePickle.to_pickle(v)
        return Literal(
            scalar=Scalar(
                generic=_json_format.Parse(json.dumps({"pickle_file": remote_path}), struct_pb2.Struct())
            ),
            metadata={"format": "pickle"},
        )
    raise TypeTransformerFailedError(f"Cannot convert dictionary to Flyte Literal: {e}")

Pickle is not enabled implicitly. DictTransformer.is_pickle reads allow_pickle from an OrderedDict entry in Annotated metadata; its default is False. When enabled, the dictionary is written through FlytePickle.to_pickle, and deserialization recognizes metadata={"format": "pickle"} and calls FlytePickle.from_pickle.

This fallback is specific to dictionary struct serialization. The flyte.types package documents FlytePickle as a way to serialize arbitrary Python objects, while noting that pickled values are not human-readable, cannot be represented in the UI, and may be inefficient for large datasets.

Dataclass serialization has a separate boundary. DataclassTransformer.to_literal raises TypeTransformerFailedError when the value is not a dataclass. If Mashumaro cannot encode a custom type because it lacks the required serialization support, the transformer raises NotImplementedError with guidance to inherit from mashumaro.types.SerializableType and implement _serialize and _deserialize. Not every serialization problem is this marker exception.

Runtime error translation and added context

At the task output boundary, _internal.runtime.convert.convert_from_native_to_outputs converts each returned value with its output name and declared type. It translates this exception into flyte.errors.RuntimeDataValidationError:

for (output_name, python_type), v in zip(interface.outputs.items(), o):
    try:
        lit = await TypeEngine.to_literal(v, python_type, TypeEngine.to_literal_type(python_type))
        named.append(run_definition_pb2.NamedLiteral(name=output_name, value=lit))
    except TypeTransformerFailedError as e:
        raise flyte.errors.RuntimeDataValidationError(output_name, e, task_name)

The transformer message remains the cause while the runtime error identifies the output and task. This handler catches TypeTransformerFailedError; a plain ValueError, TypeError, or NotImplementedError follows its own path.

For task inputs, TypeEngine.literal_map_to_kwargs schedules TypeEngine.to_python_value for each literal. If one fails, it wraps the exception with the literal, expected Python type, and original exception. In source this is an except Exception block that raises a new TypeTransformerFailedError beginning with Error converting input: and including Literal value, Expected Python type, and Exception. This explains an outer message around a nested transformer diagnostic.

The method requires either python_types or literal_types; omitting both raises ValueError before conversion. TypeEngine.dict_to_literal_map differs: after gathering conversions, it re-raises exceptions recognized as TypeError as a plain TypeError containing the variable name, type, and received value. Do not require every dictionary-to-literal-map failure to remain this exception.

Distinguish nearby exceptions

• TypeTransformerFailedError: transformer assertions, incompatible literal shapes, unsupported binary formats, failed dictionary MessagePack conversion, invalid enum values, and failed conversion with no successful union variant.
• TypeError: UnionTransformer uses it when multiple variants match ambiguously, or when no variant can be converted during deserialization. dict_to_literal_map also deliberately reclassifies TypeError failures.
• ValueError: unsupported literal-type inference and invalid map-key/type configuration can fail outside the marker-exception path.
• NotImplementedError: abstract transformer methods use it for unimplemented conversions, and dataclass MessagePack capability failures use it for missing SerializableType support.
• Storage/download exceptions: offloaded literals are fetched before the selected transformer runs.

Diagnostic checklist

Message or symptom	Inspect
Expected value of type ... or Expected ... object	Compare the runtime value with the annotation. For SimpleTransformer, check exact type equality; for File, Dir, and DataFrame, check the required wrapper.
Expected a list or Expected a dict	Confirm the exact container and usable generic element/value hints.
Expected blob literal / Expected single part blob / Expected multipart	Inspect scalar shape and dimensionality. File requires SINGLE; Dir requires MULTIPART.
Unsupported binary format	Check that the binary tag is msgpack.
Cannot convert ... to Flyte Literal	Inspect the MessagePack encoder error. For dictionary struct serialization, opt into annotated allow_pickle only when that representation is appropriate.
Error converting input	Read the literal, expected type, and nested exception added by literal_map_to_kwargs.
Ambiguous choice of variant for union type	This is plain TypeError; make the union variants distinguishable or avoid a value matching multiple variants.
Dataclass error mentioning SerializableType	This is NotImplementedError; implement the Mashumaro serialization hooks required by the custom type.

List and map element conversions run in batches controlled by _TYPE_ENGINE_COROS_BATCH_SIZE, initialized from the _F_TE_MAX_COROS environment variable with a default of 10. Changing _F_TE_MAX_COROS changes concurrency batch size, not the type checks or serialization formats that produce TypeTransformerFailedError.