Controlling Task Caching with Hashing

File identity and task caching

A File or Dir is an IO reference first: its path points to a local or remote object, while its optional hash records a cache/content identity supplied by the caller or produced during a file upload. A path alone is not a content hash. If no hash is present, Flyte's runtime falls back to the deterministic serialization of the literal, which contains the serialized IO reference rather than a digest of the bytes in the referenced object.

The cache-key boundary is generate_inputs_repr_for_literal in _internal/runtime/convert.py. For a literal with a hash, the function returns the UTF-8 bytes of that hash instead of serializing the complete literal:

if literal.hash:
    return literal.hash.encode("utf-8")

For collections and maps, the same rule is applied to nested literals; otherwise, scalar and non-hashed literals are serialized deterministically. Consequently, setting File.hash or Dir.hash before TypeEngine converts the value to a Flyte literal changes the representation consumed by task input cache-key generation. FileTransformer and DirTransformer carry the object hash into the corresponding literal; File.hash_method itself is not transported.

Choosing an identity for a file

File has both a public hash and an excluded hash_method field:

path: str
name: Optional[str] = None
format: str = ""
hash: Optional[str] = None
hash_method: Annotated[Optional[HashMethod], Field(default=None, exclude=True), SkipJsonSchema()] = None

The distinction matters. hash is the value that can be serialized into a Flyte literal. hash_method is a live accumulator used while bytes are being transferred or written; after the operation, the resulting string is assigned to hash where the relevant code path supports it.

Existing remote data: provide a known cache key

File.from_existing_remote creates a reference without reading the file or hashing its contents. Its optional file_cache_key is stored directly as hash:

@classmethod
def from_existing_remote(cls, remote_path: str, file_cache_key: Optional[str] = None) -> File[T]:
    return cls(path=remote_path, hash=file_cache_key)

For example, this creates a reference whose cache identity is the supplied value:

remote_file = File.from_existing_remote(
    "s3://my-bucket/data.csv",
    file_cache_key="dataset-version-42",
)

The same method is used by the CLI's FileParamType. The CLI validates a local path when it is not remote, then calls File.from_existing_remote(value); it does not calculate a content hash. Thus, CLI conversion produces a File reference but leaves hash unset unless the caller constructs or modifies the reference with a cache key elsewhere.

Upload a local file while accumulating its bytes

File.from_local accepts either a string or a HashMethod. A string means the identity is already known. A HashMethod supplies the accumulator used to compute the identity during the upload.

The concrete standard-library adapter is HashlibAccumulator:

from hashlib import sha256

from flyte.io._hashing_io import HashlibAccumulator

accumulator = HashlibAccumulator(sha256())

It forwards each memoryview to the wrapped hashlib object and returns hexdigest(). The equivalent factory form is available when the algorithm name is supplied as a string:

from flyte.io._hashing_io import HashlibAccumulator

accumulator = HashlibAccumulator.from_hash_name("sha256")

Passing that accumulator to the real upload API causes the remote upload path to be streamed through AsyncHashingReader; after the upload, File.from_local stores src_wrapper.result() in the new File.hash:

from flyte.io import File
from flyte.io._hashing_io import HashlibAccumulator

remote_file = await File.from_local(
    "/tmp/data.csv",
    "s3://bucket/data.csv",
    hash_method=HashlibAccumulator.from_hash_name("sha256"),
)

File.from_local first raises ValueError if the local path does not exist. For a remote destination and an ordinary accumulator, it opens the source, wraps it in AsyncHashingReader, and passes that reader to storage.put_stream. The reader hashes the bytes returned by its read operations, so the resulting value represents the stream actually consumed by the upload.

A string follows a different path:

remote_file = await File.from_local(
    "/tmp/data.csv",
    "s3://bucket/data.csv",
    hash_method="dataset-version-42",
)

The string is placed directly in hash; it is not interpreted as a hashlib algorithm and no file-content calculation is performed.

Use a known value without reading for hashing

PrecomputedValue implements HashMethod, but its update method does nothing and result() always returns the value passed to its constructor:

from flyte.io._hashing_io import PrecomputedValue

known_identity = PrecomputedValue("dataset-version-42")

File.from_local explicitly recognizes this type. For a remote destination, it uploads normally with storage.put, skips AsyncHashingReader, and obtains the stored hash from known_identity.result():

from flyte.io import File
from flyte.io._hashing_io import PrecomputedValue

remote_file = await File.from_local(
    "/tmp/data.csv",
    "s3://bucket/data.csv",
    hash_method=PrecomputedValue("dataset-version-42"),
)

This is appropriate only when the caller already knows the identity represented by the value. PrecomputedValue does not verify the uploaded bytes; an incorrect value makes different data appear to have the same cache identity.

Streaming output files

File.new_remote creates a reference with a random remote path from the initialized raw-data context. It accepts either a string or a HashMethod:

@classmethod
@requires_initialization
def new_remote(cls, hash_method: Optional[HashMethod | str] = None) -> File[T]:
    ctx = internal_ctx()
    known_cache_key = hash_method if isinstance(hash_method, str) else None
    method = hash_method if isinstance(hash_method, HashMethod) else None

    return cls(path=ctx.raw_data.get_random_remote_path(), hash=known_cache_key, hash_method=method)

The documented streaming pattern writes binary data to the returned reference:

@env.task
async def my_task() -> File[DataFrame]:
    df = pd.DataFrame(...)
    file = File.new_remote()
    async with file.open("wb") as f:
        df.to_csv(f)
    return file

A HashMethod can be supplied to new_remote when the applicable open path supports hashing:

file = File.new_remote(HashlibAccumulator.from_hash_name("sha256"))

However, the open implementation does not wrap every kind of file handle. Hashing occurs only in its synchronous-fsspec fallback, when self.hash_method exists and self.hash is still None:

with fs.open(self.path, mode) as file_handle:
    if self.hash_method and self.hash is None:
        fh = HashingWriter(file_handle, accumulator=self.hash_method)
        yield fh
        self.hash = fh.result()
        fh.close()
    else:
        yield file_handle
        file_handle.close()

Local files use aiofiles in open, and an AsyncFileSystem handle is yielded directly. Those branches do not install this HashingWriter. The synchronous remote fallback also requires binary mode and raises ValueError when the mode does not contain "b".

Directory identity is caller-supplied

Dir has hash, but unlike File it has no hash_method:

path: str
name: Optional[str] = None
format: str = ""
hash: Optional[str] = None

Directory construction therefore accepts a string cache key, not a streaming accumulator. Dir.from_local uploads recursively and stores dir_cache_key:

remote_dir = await Dir[DataFrame].from_local(
    "/tmp/data_dir/",
    "s3://bucket/data/",
    dir_cache_key="directory-version-42",
)

Dir.from_existing_remote performs no IO and creates the same kind of reference:

remote_dir = Dir.from_existing_remote(
    "s3://bucket/data/",
    dir_cache_key="directory-version-42",
)

Neither constructor computes a directory content digest. If a caller wants content-derived identity, the caller must compute it and pass the resulting string as dir_cache_key. A directory walk does not change this: walk and walk_sync yield newly constructed File[T](path=...) objects, and those child files have no hash. list_files and list_files_sync simply collect those references.

The directory API also has distinct IO limitations. Dir.from_local_sync raises NotImplementedError, and Dir.download_sync can copy an already-local directory but raises NotImplementedError for remote paths. Dir.from_local does not explicitly validate that its local input is a directory before calling storage.put(..., recursive=True).

Hashing the exact stream

HashMethod is a runtime-checkable protocol rather than a required hashing algorithm:

@runtime_checkable
class HashMethod(Protocol):
    def update(self, data: memoryview, /) -> None: ...
    def result(self) -> str: ...

    def reset(self) -> None: ...

The writers and readers require update and result; the comment in io/_hashing_io.py identifies reset as optional convenience rather than a requirement of the wrappers. HashlibAccumulator is the built-in hashlib adapter, while callers may provide another object satisfying the protocol.

HashingWriter calls update before forwarding each write. It hashes the exact value passed to the wrapper: bytes-like values are viewed directly, and strings are encoded using the underlying handle's encoding when available, otherwise UTF-8, for hashing only. The original string is still passed unchanged to the underlying handle. Because accumulation precedes the underlying write, a failed write can leave the accumulator containing bytes that were not persisted.

HashingReader and AsyncHashingReader update the accumulator only for data returned by operations actually performed (read, readline, readlines, or iteration). They do not hash unread data. The async reader supports an underlying async iterator and has a readline-based fallback. These wrappers proxy most other file-handle attributes, but they do not turn an arbitrary File.open call into an automatically hashed read.

Hash lifecycle

Stage	`File` behavior	`Dir` behavior
Reference creation	`hash` is absent unless a string cache key, `file_cache_key`, or `PrecomputedValue` result is supplied.	`hash` is absent unless `dir_cache_key` is supplied.
Transfer or write	`from_local` can accumulate uploaded bytes with `AsyncHashingReader`; the supported synchronous-fsspec `open` fallback can accumulate writes with `HashingWriter`.	`from_local` uploads recursively but does not accumulate a directory hash.
Assignment	`File.from_local` stores the string result in `hash`; the supported `open` fallback assigns `fh.result()` after the context yields.	The supplied `dir_cache_key` is stored directly in `hash`.
Literal conversion	`FileTransformer` includes `File.hash` in the Flyte literal. `hash_method` is excluded from serialization and JSON schema.	`DirTransformer` includes the optional directory hash in the multipart literal.
Cache-key representation	`generate_inputs_repr_for_literal` uses `literal.hash.encode("utf-8")` when present.	The same runtime rule applies to a directory literal.

Initialization and integration constraints

Generating an output path without a destination requires Flyte initialization. Both File.new_remote and File.from_local are decorated with requires_initialization; when a destination is omitted, they use internal_ctx().raw_data.get_random_remote_path(). Remote transfers also require the configured Flyte storage backend and credentials because File and Dir resolve fsspec filesystems from their paths.

Container output conversion illustrates the default behavior when no explicit identity is provided. extras/_container.py converts a generated file with await File.from_local(output_path) and a generated directory with await Dir.from_local(output_path). Those calls do not pass hash_method or dir_cache_key, so these conversion paths do not request content-derived hashes.

The CLI conversion has the complementary behavior: DirParamType validates local directories and returns Dir(path=value), while FileParamType validates local files and returns File.from_existing_remote(value). Neither conversion calculates content identity.

The source still marks synchronous file download as a TODO, and the IO transformers contain TODOs around the final synchronous story and generic format handling. Code that needs cache identity should therefore set or compute the hash explicitly rather than infer that upload, traversal, download, or opening a reference always hashes its contents.

File identity and task caching​

Choosing an identity for a file​

Existing remote data: provide a known cache key​

Upload a local file while accumulating its bytes​

Use a known value without reading for hashing​

Streaming output files​

Directory identity is caller-supplied​

Hashing the exact stream​

Hash lifecycle​

Initialization and integration constraints​