Restart failed upload jobs from 2026-02-19

[helen-m-lin]

Describe the bug
Many upload jobs failed after 7PM on 2026-02-19. Some error messages say "Disk Quota Exceeded".

To Reproduce
Steps to reproduce the behavior:

1. Go to http://aind-data-transfer-service/jobs and the Airflow dashboard
2. Filter by failed jobs from 2026-02-19
3. Observe failures and related Airflow logs
4. Open related log files written to VAST.
5. See error

Expected behavior
VAST seems operational today. The failed jobs can be restarted:

• Create Service Now ticket for Disk Quota Exceeded error
• Notify users of failures on the Scientific Computing channel
• Reach out to users to see if they want us to restart the jobs
• Delete staging folders for jobs that failed during compression task
• Restart jobs in Airflow

Additional context
Add any other context about the problem here.

[helen-m-lin]

2/20: Deleted staging folders and restarted:

• 841310_2026-02-19_21-04-00, manual__2026-02-20T00:47:10.935719+00:00
• 818152_2026-02-19_21-01-09, manual__2026-02-20T02:06:56.014414+00:00
• 828426_2026-02-20_00-32-51, manual__2026-02-20T02:12:54.127492+00:00
• 828415_2026-02-20_00-32-35, manual__2026-02-20T02:14:11.862922+00:00
• 828425_2026-02-19_21-07-56, manual__2026-02-20T02:14:21.418228+00:00
• 806527_2026-02-19_16-49-08, manual__2026-02-20T02:16:40.290690+00:00
• 822683_2026-02-19_16-49-20, manual__2026-02-20T02:16:57.914169+00:00
• 841306_2026-02-20_00-21-02, manual__2026-02-20T02:23:47.279289+00:00
• 828424_2026-02-20_00-21-58, manual__2026-02-20T02:35:59.506981+00:00
• 807093_2026-02-19_17-05-57, manual__2026-02-20T04:17:10.328624+00:00
• 807086_2026-02-19_16-48-59, manual__2026-02-20T04:18:46.246102+00:00
• multiplane-ophys_837568_2026-02-19_13-22-26, manual__2026-02-20T06:15:13.973172+00:00
• ecephys_836729_2026-02-16_13-20-53, manual__2026-02-20T03:00:52.220066+00:00
• ecephys_836729_2026-02-17_12-11-16, manual__2026-02-20T03:00:58.509699+00:00

2/23: Required delete partial data from S3:

• SmartSPIM_831965_2026-01-31_03-27-36, manual__2026-02-20T00:25:48.631286+00:00
• SmartSPIM_805154_2026-02-11_15-04-38, manual__2026-02-20T00:25:53.922875+00:00
• SmartSPIM_812476_2026-02-12_23-42-46, manual__2026-02-20T01:41:48.375489+00:00
• SmartSPIM_817967_2026-02-12_15-04-18, manual__2026-02-20T01:41:41.714544+00:00

Discovery Neuromodulation - Subproject 3 Fiber Photometry Recordings of NM Release During Behavior

• behavior_821586_2026-02-19_13-07-58, manual__2026-02-20T05:05:24.762068+00:00
• behavior_819168_2026-02-19_13-09-58, manual__2026-02-20T05:07:09.632039+00:00
• behavior_821587_2026-02-19_08-34-23, manual__2026-02-20T05:09:31.403424+00:00
• behavior_821573_2026-02-19_08-31-42, manual__2026-02-20T05:11:27.329790+00:00
• behavior_823164_2026-02-19_09-31-28, manual__2026-02-20T05:25:06.449643+00:00
• behavior_823163_2026-02-19_09-40-44, manual__2026-02-20T05:32:18.148232+00:00
• behavior_821576_2026-02-19_09-38-28, manual__2026-02-20T05:34:56.686910+00:00
• behavior_821786_2026-02-19_13-13-01, manual__2026-02-20T05:39:48.602415+00:00

Striatum

• behavior_832634_2026-02-19_17-59-46, manual__2026-02-20T05:53:48.253368+00:00
• behavior_828133_2026-02-19_18-20-59, manual__2026-02-20T05:57:10.585034+00:00
• behavior_828137_2026-02-19_18-09-10, manual__2026-02-20T06:00:02.453135+00:00
• behavior_829567_2026-02-19_18-05-19, manual__2026-02-20T06:05:00.916052+00:00
• behavior_827183_2026-02-19_18-32-51, manual__2026-02-20T06:06:17.384086+00:00
• behavior_829569_2026-02-19_18-14-04, manual__2026-02-20T06:12:55.556986+00:00
• behavior_833283_2026-02-17_19-15-28, manual__2026-02-20T06:17:41.844006+00:00

[helen-m-lin]

Bowen also asked to look into:

• behavior_824527_2026-02-03_17-09-30, manual__2026-02-19T06:13:57.301320+00:00
• behavior_828137_2026-02-14_17-41-53, manual__2026-02-19T06:32:26.173950+00:00

I'm seeing errors for behavior-videos compression:

ERROR:root:FFmpeg conversion failed for /allen/aind/scratch/dynamic_foraging_rig_transfer/behavior_828137_2026-02-14_17-41-53/behavior-videos/SideCameraRight/video.mp4
Command: ffmpeg -y -v warning -hide_banner -i /allen/aind/scratch/dynamic_foraging_rig_transfer/behavior_828137_2026-02-14_17-41-53/behavior-videos/SideCameraRight/video.mp4 -vf scale=out_color_matrix=bt709:out_range=full:sws_dither=none,format=yuv420p10le,colorspace=ispace=bt709:all=bt709:dither=none,scale=out_range=tv:sws_dither=none,format=yuv420p -c:v libx264 -preset veryslow -crf 18 -pix_fmt yuv420p -metadata author=Allen Institute for Neural Dynamics -movflags +faststart+write_colr /allen/aind/stage/svc_aind_airflow/prod/behavior_828137_2026-02-14_17-41-53_aa9feba607/behavior-videos/SideCameraRight/video.mp4
Return code: 183
Error output:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x56332b90d000] moov atom not found
[in#0 @ 0x56332b90cd40] Error opening input: Invalid data found when processing input
Error opening input file /allen/aind/scratch/dynamic_foraging_rig_transfer/behavior_828137_2026-02-14_17-41-53/behavior-videos/SideCameraRight/video.mp4.
Error opening input files: Invalid data found when processing input

Edit: Raw video files are corrupt. Bowen asked to resubmit without compression. This has been done. New uploads tracked here:

• behavior_828137_2026-02-14_17-41-53, manual__2026-02-23T20:17:18.700471+00:00
• behavior_824527_2026-02-03_17-09-30, manual__2026-02-23T20:17:17.904306+00:00

[helen-m-lin]

All failed jobs have been rerun.

edit: SmartSPIM_805154_2026-02-11_15-04-38 failed, notified Mike. All others successfully completed.