Skip to content

HfFileSystem is incompatible with fsspec's config #3996

@martindurant

Description

@martindurant

Describe the bug

fsspec allows for confg to be defined (from files or environment variables) https://filesystem-spec.readthedocs.io/en/latest/features.html#configuration . Because HF's fsspec implementation uses it's own version of fsspec.spec._Cached, the config is not applied.

(I realise that HF has a bunch of other env vars it uses)

I suggest either:

  • add the line kwargs = apply_config(cls, kwargs) to hf_file_system._Cached.__call__
  • upstream the changes in _Cached to fsspec and retire the class.

I prefer the second option, if possible.

Reproduction

No response

Logs

System info

- huggingface_hub version: 1.8.0
- Platform: macOS-15.7.3-arm64-arm-64bit
- Python version: 3.12.11
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /Users/mdurant/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: osxkeychain
- Installation method: unknown
- httpx: 0.28.1
- hf_xet: 1.4.2
- gradio: N/A
- tensorboard: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /Users/mdurant/.cache/huggingface/hub
- HF_ASSETS_CACHE: /Users/mdurant/.cache/huggingface/assets
- HF_TOKEN_PATH: /Users/mdurant/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /Users/mdurant/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_DISABLE_XET: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
- HF_XET_HIGH_PERFORMANCE: False

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions