Configuring Data Storage

Configure storage through initialization

To make Flyte remote data paths use explicit cloud credentials or an emulator endpoint, create a frozen S3, GCS, or ABFS configuration and pass it to flyte.init(storage=...):

import flyte
from flyte.storage import S3

flyte.init(
    endpoint="flyte.example.com",
    storage=S3(
        endpoint="https://s3.example.com",
        access_key_id="access-key",
        secret_access_key="secret-key",
    ),
)

Storage is the provider-independent base dataclass. It supplies retries=3, backoff=datetime.timedelta(seconds=5), enable_debug=False, and attach_execution_metadata=True. S3, GCS, and ABFS are frozen dataclasses, so construct a new object rather than modifying one after initialization.

Storage selection is driven by the URI protocol. get_underlying_filesystem() obtains the protocol from the path, asks get_configured_fsspec_kwargs() for provider settings, merges operation-specific keyword arguments, and calls fsspec.filesystem(). The initialized object is used only when its type matches the URI: S3 for s3://, GCS for gs://, and ABFS for either abfs:// or abfss://.

Configure from environment variables

Each provider has an auto() constructor. Unset variables are omitted by flyte.config.set_if_exists rather than inserted as None.

from flyte.storage import ABFS, GCS, S3

s3 = S3.auto()
gcs = GCS.auto()
abfs = ABFS.auto()

S3.auto() reads the provider variables below and also calls Storage._auto_as_kwargs():

Setting	Environment variable
S3 endpoint	FLYTE_AWS_ENDPOINT
S3 access key	FLYTE_AWS_ACCESS_KEY_ID
S3 secret key	FLYTE_AWS_SECRET_ACCESS_KEY
Retry count	UNION_STORAGE_RETRIES
Initial backoff	UNION_STORAGE_BACKOFF_SECONDS
Debug flag	UNION_STORAGE_DEBUG

ABFS.auto() reads AZURE_STORAGE_ACCOUNT_NAME, AZURE_STORAGE_ACCOUNT_KEY, AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET. GCS.auto() reads only GCP_GSUTIL_PARALLELISM; it does not call the base auto helper, so the common retry, backoff, and debug variables are not included in an automatically created GCS object.

Environment values are read as strings. The auto constructors pass those values through without explicit conversion, including values assigned to annotated int, datetime.timedelta, and bool fields. Construct a provider explicitly when typed values are required:

import datetime
from flyte.storage import S3

storage = S3(
    retries=5,
    backoff=datetime.timedelta(seconds=10),
    enable_debug=True,
    endpoint="https://s3.example.com",
    access_key_id="access-key",
    secret_access_key="secret-key",
)

Configure Amazon S3 and S3-compatible endpoints

S3 adds endpoint, access_key_id, and secret_access_key to the common Storage fields. The dataclass field is named endpoint, but get_fsspec_kwargs() emits it downstream as endpoint_url:

from flyte.storage import S3

config = S3(
    endpoint="http://localhost:4566",
    access_key_id="minio",
    secret_access_key="miniostorage",
)

filesystem_kwargs = config.get_fsspec_kwargs()

The resulting provider configuration places credentials and the endpoint in config under access_key_id, secret_access_key, and endpoint_url. Per-operation overrides use the downstream names, so override an endpoint with endpoint_url, not endpoint:

filesystem_kwargs = config.get_fsspec_kwargs(
    endpoint_url="https://s3.example.com",
    retries=5,
)

S3 always adds these filesystem options:

• client_options: {"timeout": "99999s", "allow_http": True}
• retry_config: a max_retries value, exponential backoff with base 2, the configured initial backoff, a 16-second maximum backoff, and a three-minute retry timeout

For local development, S3.for_sandbox() supplies the built-in endpoint and credentials:

import flyte
from flyte.storage import S3

flyte.init(
    endpoint="flyte.example.com",
    storage=S3.for_sandbox(),
)

S3.for_sandbox() uses http://localhost:4566, minio, and miniostorage. Its HTTP-enabled client options and hard-coded local credentials are appropriate for a local emulator, not production AWS access.

For anonymous S3 access, pass anonymous=True to get_fsspec_kwargs(). S3 removes the generic argument and instead places skip_signature=True in its provider config:

anonymous_kwargs = S3().get_fsspec_kwargs(anonymous=True)

Configure Google Cloud Storage

GCS exposes one Flyte-specific field, gsutil_parallelism, defaulting to False:

import flyte
from flyte.storage import GCS

flyte.init(
    endpoint="flyte.example.com",
    storage=GCS(gsutil_parallelism=True),
)

GCS.get_fsspec_kwargs() removes anonymous and returns the remaining caller-supplied keyword arguments unchanged. It does not add credentials, retry configuration, client options, or an explicit anonymous flag. Supply GCS credentials through the underlying Google/fsspec credential mechanism or as direct filesystem keyword arguments; GCS itself does not translate credential fields.

The automatic setting is GCP_GSUTIL_PARALLELISM:

from flyte.storage import GCS

config = GCS.auto()

Configure Azure Blob Storage

ABFS supports account-key and service-principal fields:

import flyte
from flyte.storage import ABFS

flyte.init(
    endpoint="flyte.example.com",
    storage=ABFS(
        account_name="storageaccount",
        account_key="account-key",
    ),
)

For service-principal authentication, construct the same class with tenant_id, client_id, and client_secret:

from flyte.storage import ABFS

config = ABFS(
    account_name="storageaccount",
    tenant_id="tenant-id",
    client_id="client-id",
    client_secret="client-secret",
)

ABFS.auto() maps these environment variables to the corresponding fields:

Field	Environment variable
account_name	AZURE_STORAGE_ACCOUNT_NAME
account_key	AZURE_STORAGE_ACCOUNT_KEY
tenant_id	AZURE_TENANT_ID
client_id	AZURE_CLIENT_ID
client_secret	AZURE_CLIENT_SECRET

The abfs:// and abfss:// protocols both select ABFS. Its get_fsspec_kwargs() puts configured values in config, adds skip_signature=True when anonymous=True, and always adds client_options with timeout="99999s" and allow_http="true".

Use the configured storage from high-level APIs

After initialization, high-level I/O selects the same filesystem configuration from each destination URI. For example, File.from_local() uploads a non-local destination through the storage layer:

from flyte.io import File

async def upload_file():
    remote_file = await File.from_local("/tmp/data.csv", "s3://bucket/data.csv")
    return remote_file

The implementation detects the destination protocol. For a remote protocol it calls storage.put() (or storage.put_stream() when hashing is requested); for a file destination it retains its local-file path/copy behavior.

Streaming writes can target an explicit provider URI:

import flyte.storage as storage

async def write_data():
    return await storage.put_stream(
        b"hello",
        to_path="s3://my_bucket/my_file.txt",
    )

put_stream() calls get_underlying_filesystem(path=to_path), so the s3:// URI selects the initialized S3 configuration or, when no matching initialized configuration is available, S3.auto(). The same path is used for gs://, abfs://, and abfss:// operations. Runtime inputs, outputs, errors, reports, code bundles, and directory/file operations use these storage-layer APIs rather than maintaining separate provider credentials.

Troubleshoot configuration selection

Initialize before storage operations

Operations that resolve storage without a usable protocol-specific fallback require initialization. get_storage() raises InitializationError when no storage configuration has been initialized, and guarded high-level operations instruct callers to call flyte.init(...). Initialize Flyte before using remote File or storage operations.

A protocol can still trigger automatic configuration: for s3, gs, abfs, and abfss, the storage layer catches an uninitialized-storage error and uses the matching provider's auto() method. This fallback does not apply to the no-protocol branch, which calls get_storage() directly.

Match the configuration class to the URI

An initialized configuration of the wrong provider type is ignored for a protocol. For example, an initialized GCS does not configure an s3:// filesystem; the S3 branch uses S3.auto() instead. This can select environment credentials instead of the object passed to flyte.init.

Distinguish anonymous options by provider

S3 and ABFS represent anonymous access as config["skip_signature"] = True; they do not forward a generic anonymous=True option. GCS removes anonymous but does not add skip_signature, so anonymous GCS behavior must come from caller-supplied options or the underlying GCS credential behavior.

Account for emulator HTTP settings

S3's client_options sets allow_http to the Boolean True; ABFS uses the string value "true". Both provider conversions also set a very long client timeout. These values are emitted for every filesystem construction from those configurations, including production configurations, so review them when selecting endpoints.

Remember the obstore integration boundary

The storage layer registers asynchronous obstore filesystems for S3, GCS, ABFS, and ABFSS. Its stream operations contain private obstore workarounds, including store construction and asynchronous reader/writer paths; the source comments note that these paths can break if obstore changes its API. Provider configuration should therefore flow through flyte.storage and its filesystem helpers rather than bypassing the configured storage layer.