Default cache type

[martindurant]

When this repo was originally written, "bytes" was the only in-memory cache type available, and so has been the default. s3fs switched to "readahead" some time ago, as it often has better performance, particularly for sequential reading (e.g., pd.read_csv).

Recently, gcsfs has acquired a "prefetcher" mechanism, which can dramatically improve throughput. It can in theory work with any cache type, but is best suited to readahead. It isn't certain yet how exactly prefetcher will be surfaced for use in other backends (it might yet become its own cache type), but this seems like a good time to reconsider the default cache type for adlfs.

[kyleknap]

Thanks for the heads up @martindurant! I have not dug into the caching mechanics of fsspec, but in reading the difference between bytes and readahead, I do agree that readahead seems like a better default for most scenarios. I'd probably want us to do some perf testing before making a change to make sure we are happy with changing the default. A couple of questions that I did have on this:

1. I'm assuminggcsfs is using readahead by default already? I think if both s3fs and gcsfs are already doing readahead that is already a compelling reason to change at a consistency level.
2. Do you have any recommendations on downstream dependency interfaces to test against (e.g., pd.read_csv) for understanding the perf difference if we made the switch? Mainly trying to understand for the popular downstream libraries what is the most common sequentially read sizes (e.g., 4 KiB) we should be targeting?

On the prefetcher, that is really interesting to learn about. I do see how it would be useful. Is there any issues to follow on fsspec core tracking it being made available upstream and to learn more about it?

[martindurant]

Yes, gcsfs uses "readahead". The google team wishes to enable the prefetcher improvements for maybe any cache type, but readahead is the most obvious and should work the best.

All development ont he prefetcher is happening in gcsfs for now, and the team wish to go through exhaustive testing before upstreaming anything. I am hoping it should be possible to grab by importing gcsfs directory (even if you don't intend to use it), but that's not possible yet either. There is no issue covering this at filesystem_spec yet.

[martindurant]

On the downstream libraries - well, it could be many, and that's the problem! Certainly CSV indeed any text format that is sequential will work well with readahead and eventually prefetching. Others like parquet fetch absolute ranges (perhaps concurrently) and won't be affected at all. Yet others may get worse, e.g., HFD5 loves to alternate reading small (~bytes) sections from near the start of the file and large blocks from the rest of the file.

[kyleknap]

Gotcha. That makes sense to me on both comments.

In regards, to testing read patterns, we'll plan to test a variety of these read types to see what the impact looks like in making the change. I do agree the small random reads, especially if there are minimal/no large read blocks, seems like the most likely pattern to be negatively impacted.