Skip to main content

Reading and Writing File Content

The flyte-sdk provides the io.File class as a generic abstraction for handling files, whether they reside on a local filesystem or in remote object storage like S3. It allows you to specify the expected format of the file using a generic type T, enabling Flyte to understand and manage data types effectively. The File class offers both synchronous and asynchronous interfaces for common file operations.

Creating File Objects​

To interact with files using flyte-sdk, you first need to create an instance of io.File. There are several ways to instantiate a File object, depending on your use case.

From an Existing Path​

You can create a File object by directly providing its path. This path can point to a local file or a remote storage location.

from flyte.io import File
from pandas import DataFrame

# For a remote file
csv_file_remote = File[DataFrame](path="s3://my-bucket/data.csv")

# For a local file
local_file = File[str](path="/tmp/my_local_file.txt")

Creating a New Remote File​

When you need to generate a new file as an output of a Flyte task and stream its content directly to remote storage, use the File.new_remote() class method. This method creates a File object with a generated remote path.

from flyte.io import File
from pandas import DataFrame
from flytekit import task

@task
async def create_and_write_remote_file() -> File[DataFrame]:
    df = DataFrame({"col1": [1, 2], "col2": ["A", "B"]})
    csv_file = File[DataFrame].new_remote()
    async with csv_file.open(mode="w") as f:
        df.to_csv(f, index=False)
    return csv_file

Referencing an Existing Remote File​

If you have a file already present in remote storage and want to create a File object that points to it without uploading, use File.from_existing_remote().

from flyte.io import File
from pandas import DataFrame
from flytekit import task

@task
async def use_existing_remote_file() -> File[DataFrame]:
    # This creates a File object pointing to the S3 path, no data is copied yet.
    csv_file = File[DataFrame].from_existing_remote("s3://my-bucket/existing.csv")
    async with csv_file.open() as f:
        # You can now read content from the existing remote file
        df = DataFrame.from_csv(f)
    return csv_file

Uploading a Local File to Remote Storage​

To create a File object from a local file and automatically upload it to the configured remote store, use File.from_local(). You can optionally specify a remote_destination.

import os
from pathlib import Path
from flyte.io import File
from pandas import DataFrame

async def upload_local_file():
    local_path = Path("/tmp/data.csv")
    # Create a dummy local file for demonstration
    with open(local_path, "w") as f:
        f.write("col1,col2\n1,A\n2,B")

    # Upload to a generated remote path
    remote_file_auto = await File[DataFrame].from_local(local_path)
    print(f"Uploaded to: \{remote_file_auto.path\}")

    # Upload to a specific remote path
    remote_file_specific = await File[DataFrame].from_local(local_path,