Search_for_wildcards function updated to add @Dateto@ functionality (#510)

* {Search for wildcards function updated}

* Refactoring changes asked by the maintainer which include centralisation of code my making functions in validation.py and using in search_with_tags feature in folders file

* Fabrication of the code moved functions in folders.py

* Update datashuttle/utils/folders.py

Co-authored-by: Joe Ziminski <55797454+JoeZiminski@users.noreply.github.com>

* Update datashuttle/utils/folders.py

Co-authored-by: Joe Ziminski <55797454+JoeZiminski@users.noreply.github.com>

* Update datashuttle/utils/folders.py

Co-authored-by: Joe Ziminski <55797454+JoeZiminski@users.noreply.github.com>

* Update datashuttle/utils/folders.py

Co-authored-by: Joe Ziminski <55797454+JoeZiminski@users.noreply.github.com>

* Update datashuttle/utils/validation.py

Co-authored-by: Joe Ziminski <55797454+JoeZiminski@users.noreply.github.com>

* Refactor: Clean up docstrings in folders.py and canonical_tags.py as per review

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix linting.

* Properly ignore mypy on tests.

* Updated test cases to use basemodel and also added more test cases to cover all edges

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update datashuttle/utils/folders.py

Co-authored-by: Joe Ziminski <55797454+JoeZiminski@users.noreply.github.com>

* Cover DATETO etc. cases in check_and_format_names so they are not validated.

* Introduce get_datetime_to_search_regexp function.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Introduce get_datetime_to_search_regexp function.

* Remove 'test_simple_wildcard_first'.

* Update 'test_datetime_range_transfer'.

* Use parameterisation for download test.

* Update error message check.

* Add 'upload_or_download' for all key tests.

* Extend test_subject_level_date_range.

* Update test_edge_case_exact_boundary_dates.

* Adjust date test.

* Small tidy ups.

* Troubleshooting.

* Support datetime directly as sub ses values @datetime@.

* Add tests for new sub-@Date@ ses-@Date@ behaviour.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Extend tests to new sub case + some refactoring.

* Fix docstring for linter.

* Handle sub case for datetime, date, time.

* Fix tests.

* Update tests/test_date_search_range.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update datashuttle/utils/formatting.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/test_date_search_range.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update datashuttle/utils/folders.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update datashuttle/utils/folders.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/test_date_search_range.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/test_date_search_range.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/test_date_search_range.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert copilot commit

* Fix outdated api call in tests.

---------

Co-authored-by: Joe Ziminski <55797454+JoeZiminski@users.noreply.github.com>
Co-authored-by: JoeZiminski <joseph.j.ziminski@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Author: 4 people

datashuttle/configs/canonical_tags.py

@@ -9,5 +9,24 @@ def tags(tag_name: str) -> str:
         "datetime": "@DATETIME@",
         "to": "@TO@",
         "*": "@*@",
+        "DATETO": "@DATETO@",
+        "TIMETO": "@TIMETO@",
+        "DATETIMETO": "@DATETIMETO@",
     }
     return tags[tag_name]
+
+
+def get_datetime_formats() -> dict:
+    """Get all datetime format strings.
+
+    Returns
+    -------
+    dict
+        A dictionary containing format strings for datetime, time, and date
+
+    """
+    return {
+        "datetime": "%Y%m%dT%H%M%S",
+        "time": "%H%M%S",
+        "date": "%Y%m%d",
+    }

datashuttle/utils/data_transfer.py

@@ -488,7 +488,7 @@ def get_processed_names(
             processed_names = formatting.check_and_format_names(
                 names_checked, prefix, bypass_validation=True
             )
-            processed_names = folders.search_for_wildcards(
+            processed_names = folders.search_with_tags(
                 self.__cfg,
                 self.__base_folder,
                 self.__local_or_central,