Describe the bug
The csv parser does not create the check_s3_folder_exists task correctly. There is a typo with the task name, which will result in the task not actually being skipped in Airflow:
|
"check_s3_folder_exists_task": check_s3_folder_exists_task, |
To Reproduce
Steps to reproduce the behavior:
- Submit a csv or excel job with "force_cloud_sync" set to True
- Observe that the "check_s3_folder_exists" task is not skipped in Airflow.
Expected behavior
A csv with force_cloud_sync set to True should create an upload job with a check_s3_folder_exists Skip Task.
Additional context
I believe the final_check_s3_folder_exist task can also be removed. This task has been removed from the airflow DAG.
Describe the bug
The csv parser does not create the
check_s3_folder_existstask correctly. There is a typo with the task name, which will result in the task not actually being skipped in Airflow:aind-data-transfer-service/src/aind_data_transfer_service/configs/csv_handler.py
Line 96 in 4804af5
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A csv with force_cloud_sync set to True should create an upload job with a
check_s3_folder_existsSkip Task.Additional context
I believe the
final_check_s3_folder_existtask can also be removed. This task has been removed from the airflow DAG.