Monthly datetime ingestion regex inconsistency from VEDA-docs

[kyle-lesinger]

Summary

veda-docs states that for ingesting monthly data, you can use the following format YYYY-MM_monthly.tif or YYYYMM.tif --- the important thing is the number regex parsing.

I tried to ingest this file 202409_Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_2024-08_monthly.tif but the start and end datetime were for Jan 2024 instead of August 2024. ISSUE: YYYY-MM is not an acceptable format

After debugging in veda-data-airflow it was observed that this pattern YYYY-MM is incompatible, despite veda-docs saying that it is allowable. See below for veda-data-airflow regex patterns.

Additional investigation info from veda-data-airflow

Root cause: filename date format doesn't match any multi-component pattern

Per-item datetimes come from parsing the filename, not from the collection's temporal_extent (that only sets the collection-level extent). The parser is in regex.py:37-91 and is invoked from stac.py:84-86.

Tracing your filename

202409_Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_2024-08_monthly.tif

extract_dates tries 7 regex strategies in order, breaks on first match. All require a [_.-] separator immediately before the digits:

#	Pattern	Format	Match?
1	(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})	ISO	no
2	(\d{8}T\d{6})	yyyymmddThhmmss	no
3	(\d{4}_\d{2}_\d{2})	yyyy_mm_dd	no
4	(\d{4}-\d{2}-\d{2})	yyyy-mm-dd	no — your 2024-08 is only yyyy-mm
5	(\d{8})	yyyymmdd	no — no _<8digits> anywhere
6	(\d{6})	yyyymm	no — 202409 is at filename start with no preceding _/-/.
7	(\d{4})	yyyy	matches _2024 inside _2024-08

So single_datetime = datetime(2024, 1, 1). Then regex.py:18-21 applies datetime_range: "month" to Jan 1 → start=2024-01-01, end=2024-01-31.

Your items are getting January 2024, not August 2024.

Fixes (pick one)

1. Override in the discovery config — set explicit dates in the discovery_items entry; these short-circuit filename parsing per stac.py:72-78:

   "start_datetime": "2024-08-01T00:00:00Z",
   "end_datetime":   "2024-08-31T23:59:59Z"

   (Drop datetime_range if using explicit start/end.) This won't work if you want per-item dates derived from each file — only if all discovered files share the same date.
2. Add a yyyy-mm strategy to DATE_REGEX_STRATEGIES in regex.py:43-51, e.g. (r"[_\.\-](\d{4}-\d{2})(?!-\d)", "%Y-%m") placed above the bare-year fallback. This is the most general fix if you have many files with _yyyy-mm_ naming.

Note

The leading 202409 (event YYYYMM) in the filename isn't reachable by any strategy because it sits at position 0 with no preceding separator — so even if you wanted that date, the regex couldn't grab it.

[anayeaye]

@kyle-lesinger I agree that the docs are a little confusing and our datetime from filename parsing could be improved. For this case why not adjust that filename which is confusing anyway (dates seem to be in random places in the filename and are in two different formats YYYYMM and YYYY-MM)?

My preference would be to update the filename pattern to a single date with datetime_range="month" or to use the datetime range pattern as filename_YYYYMM_YYYYMM.tif AND to add a task to our backlog to improve the veda docs and airflow parsing methods.

refs

(the links in this issue weren't working for me but I think these are the intended references)
https://docs.openveda.cloud/user-guide/content-curation/dataset-ingestion/file-preparation.html#name-your-files-correctly

veda-data-airflow/dags/veda_data_pipeline/utils/build_stac/utils/regex.py

Lines 37 to 91 in fe98ccf

	def extract_dates(
	filename: str, datetime_range: events.INTERVAL
	) -> Union[Tuple[datetime, datetime, None], Tuple[None, None, datetime]]:
	"""
	Extracts start & end or single date string from filename.
	"""
	DATE_REGEX_STRATEGIES = [
	(r"[_\.\-](\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})", "%Y-%m-%dT%H:%M:%S"),
	(r"[_\.\-](\d{8}T\d{6})", "%Y%m%dT%H%M%S"),
	(r"[_\.\-](\d{4}_\d{2}_\d{2})", "%Y_%m_%d"),
	(r"[_\.\-](\d{4}-\d{2}-\d{2})", "%Y-%m-%d"),
	(r"[_\.\-](\d{8})", "%Y%m%d"),
	(r"[_\.\-](\d{6})", "%Y%m"),
	(r"[_\.\-](\d{4})", "%Y"),
	]
	# Find dates in filename
	dates = []
	for pattern, dateformat in DATE_REGEX_STRATEGIES:
	dates_found = re.compile(pattern).findall(filename)
	if not dates_found:
	continue
	for date_str in dates_found:
	dates.append(datetime.strptime(date_str, dateformat))
	break
	num_dates_found = len(dates)
	# No dates found
	if not num_dates_found:
	raise Exception(
	f"No dates provided in {filename=}. "
	"At least one date in format yyyy-mm-dd is required."
	)
	# Many dates found
	if num_dates_found > 1:
	dates.sort()
	start_datetime, *_, end_datetime = dates
	return start_datetime, end_datetime, None
	# Single date found
	single_datetime = dates[0]
	# Convert single date to range
	if datetime_range:
	start_datetime, end_datetime = DATETIME_RANGE_METHODS[datetime_range](
	single_datetime
	)
	return start_datetime, end_datetime, None
	# Return single date
	return None, None, single_datetime

A little test for a workable filenaming solution (and a way to prove that many of the example veda docs filenames patterns are not matched in regex.py)

import re
from datetime import datetime
from dateutil.relativedelta import relativedelta

DATE_REGEX_STRATEGIES = [
    (r"[_\.\-](\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})", "%Y-%m-%dT%H:%M:%S"),
    (r"[_\.\-](\d{8}T\d{6})", "%Y%m%dT%H%M%S"),
    (r"[_\.\-](\d{4}_\d{2}_\d{2})", "%Y_%m_%d"),
    (r"[_\.\-](\d{4}-\d{2}-\d{2})", "%Y-%m-%d"),
    (r"[_\.\-](\d{8})", "%Y%m%d"),
    (r"[_\.\-](\d{6})", "%Y%m"),
    (r"[_\.\-](\d{4})", "%Y"),
]

filenames = [
    "202409_Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_2024-08_monthly.tif",
    "Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_202408_202409.tif",
    "Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_202408.tif"
]

def _calculate_month_range(datetime_obj: datetime):
    start_datetime = datetime_obj.replace(day=1)
    end_datetime = datetime_obj + relativedelta(day=31)
    return start_datetime, end_datetime

    
datetime_range = "month"

for filename in filenames:
    dates = []
    dates_found = []
    datestrings = []
    num_dates_found = 0
    for pattern, dateformat in DATE_REGEX_STRATEGIES:
        dates_found = re.compile(pattern).findall(filename)
        if not dates_found:
            continue

        for date_str in dates_found:
            dates.append(datetime.strptime(date_str, dateformat))

        break

    num_dates_found = len(dates)

    # Many dates found
    if num_dates_found > 1:
        dates.sort()
        start_datetime, *_, end_datetime = dates
        dates= [start_datetime, end_datetime]

    else:
        # Single date found
        single_datetime = dates[0]
    
        # Convert single date to range
        start_datetime, end_datetime = _calculate_month_range(single_datetime)
        dates= [start_datetime, end_datetime]
            
    datestrings = [d.isoformat() for d in dates]
    print(f"{filename=}\n{dates_found=}\n{datestrings=}\n" )

Yields:

filename='202409_Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_2024-08_monthly.tif'
dates_found=['2024']
datestrings=['2024-01-01T00:00:00', '2024-01-31T00:00:00']

filename='Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_202408_202409.tif'
dates_found=['202408', '202409']
datestrings=['2024-08-01T00:00:00', '2024-09-01T00:00:00']

filename='Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_202408.tif'
dates_found=['202408']
datestrings=['2024-08-01T00:00:00', '2024-08-31T00:00:00']

[kyle-lesinger]

@kyle-lesinger I agree that the docs are a little confusing and our datetime from filename parsing could be improved. For this case why not adjust that filename which is confusing anyway (dates seem to be in random places in the filename and are in two different formats YYYYMM and YYYY-MM)?

@anayeaye Yes this is do-able, the only issue is that I've already re-named all of the Disasters files to the YYYY-MM format. Additionally, I've documented it in several locations instructing Disasters that they should follow this format. So it would be easier lift for me to not have to rename them, but I understand that your team may decide to not implement this. Please discuss it and provide me with an update. Thank you!

[anayeaye]

@kyle-lesinger it looks like @sandrahoang686 already got a start on this so maybe YYYY-MM will be accounted for sooner.

The renamed files have both date strategies and out of order dates though? 202409_Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_2024-08_monthly might hit other problems in ingest because 2024-09-01 > 2024-08-01 (I know we validate that start is less than end for the temporal range but the item ingest may not test this). Also I do not know if it would impact temporal searches if it were ingested with that reverse range. Maybe the example in this ticket was just a rare file that has that pattern.

[kyle-lesinger]

The renamed files have both date strategies and out of order dates though? 202409_Hurricane_Helene_finalBMHD_VNP46A3_MonthlyComposite_2024-08_monthly might hit other problems in ingest because 2024-09-01 > 2024-08-01

@anayeaye we are slowly removing the EVENT names from the filenames e.g., 202409_Hurricane_Helene. So in the future, there will be no conflict with Disasters. Additionally, in the veda-data-airflow code, it will not use the first datetime value IF it is the very first character as a digit.

But if you want me to explore other ways that it might impact then please let me know. Otherwise it looks like the YYYY-MM is the only fix that was needed and is being worked on. Thank you!