Partition filter KO in ZIPFilesystem if collection at root

[robin-cls]

If a zcollection is supported by a zip file system, and is located at the root, the first part of the partitioning (year for year/month/day for example) will break the partitions filtering.

An example to reproduce the problem

• Create a zcollection on the Local file system
• Zip it to a file with a top-level folder -> partition filters work properly
• Zip it to a file without a top-level folder (zcollection is at the root) -> partition filters is KO in some cases

The code to reproduce (in IPython 3.12)

from __future__ import annotations

from typing import Iterator
import datetime
import pprint

import dask.distributed as dist
import fsspec.implementations.zip
import fsspec.implementations.local
import numpy

import zcollection as zc
import zcollection.tests.data as zc_data

zc.__version__

'2024.2.0'

# Create collection
zds = next(zc_data.create_test_dataset_with_fillvalue())

cluster = dist.LocalCluster(processes=False)
client = dist.Client(cluster)
partition_handler = zc.partitioning.Date(('time', ), resolution='M')

collection = zc.create_collection(
    'time',
    zds,
    partition_handler,
    'my_collection',
    filesystem=fsspec.implementations.local.LocalFileSystem())
collection.insert(zds)
list(collection.partitions())

['/home/my_collection/year=2000/month=01',
'/home/my_collection/year=2000/month=02',
'/home/my_collection/year=2000/month=03',
'/home/my_collection/year=2000/month=04',
'/home/my_collection/year=2000/month=05',
'/home/my_collection/year=2000/month=06']

!zip -q -r my_collection.zip my_collection/
%cd my_collection
!zip -q -r ../my_collection2.zip .
%cd -

fs1 = fsspec.implementations.zip.ZipFileSystem('my_collection.zip')
zcoll = zc.open_collection('/my_collection', filesystem=fs1)
list(zcoll.partitions(filters='year == 2000 and month >= 3'))

['my_collection/year=2000/month=03',
'my_collection/year=2000/month=04',
'my_collection/year=2000/month=05',
'my_collection/year=2000/month=06']

# Filter over months works properly
fs2 = fsspec.implementations.zip.ZipFileSystem('my_collection2.zip')
zcoll = zc.open_collection('/', filesystem=fs2)
list(zcoll.partitions(filters='month >= 3'))

['year=2000/month=03',
'year=2000/month=04',
'year=2000/month=05',
'year=2000/month=06']

# Fitler over years is KO
fs2 = fsspec.implementations.zip.ZipFileSystem('my_collection2.zip')
zcoll = zc.open_collection('/', filesystem=fs2)
list(zcoll.partitions(filters='year == 2000 and month >= 3'))

[]

Why this case matters

Because there are a lot of files in a zcollection (see #29 ), we might want to zip old zcollections (read-only, few reads so performance is not important) to retrieve inodes for the file system. Zipping the collection without a top-level folder can be quite natural, but will break the filtering functionnality.

My guess

Partition filtering tries to detect relative and absolute paths. In case the base dir is the root, the distinction cannot be done.