fix: paginate S3 directory listings

[ShiroKSH]

Summary

• Load all S3 object-listing pages before downloading documents.
• Preserve the existing object filter and concurrent download behavior.
• Add regression coverage for the continuation token passed to the second request.

Root cause

S3 Directory made one ListObjectsV2 request, so prefixes with more than 1,000 matching objects silently omitted later pages.

Validation

• pnpm --filter flowise-components exec jest nodes/documentloaders/S3Directory/S3Directory.test.ts --runInBand
• pnpm --filter flowise-components exec eslint nodes/documentloaders/S3Directory/S3Directory.ts nodes/documentloaders/S3Directory/S3Directory.test.ts --ext ts --max-warnings 0
• pnpm --filter flowise-components exec tsc --noEmit --pretty false
• pnpm --filter flowise-components build

[gemini-code-assist]

Code Review

This pull request introduces pagination support when listing S3 objects in S3Directory by implementing a looping helper function getS3FileKeys and adds corresponding unit tests. Feedback on these changes highlights two main issues: first, since pagination can now retrieve an unbounded number of keys, passing all of them directly to Promise.all for concurrent processing can lead to memory exhaustion, file descriptor limits, or S3 rate limiting, so a concurrency limit should be implemented; second, the prefix parameter in getS3FileKeys should be marked as optional to match the node inputs and ensure type safety.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

With the introduction of pagination in getS3FileKeys, the number of retrieved keys is no longer capped at 1,000 and can potentially be much larger (e.g., tens of thousands of files). Passing all these keys directly to Promise.all for concurrent downloading and writing to disk can lead to severe issues such as:

1. Memory Exhaustion: Buffering thousands of file contents in memory simultaneously.
2. File Descriptor Limits: Exceeding the OS limit for open files (EMFILE).
3. Rate Limiting: Getting throttled by the S3 API due to too many concurrent requests.

Consider implementing a concurrency limit (e.g., processing in batches of 10-50 files) to make the download process robust and scalable.

[gemini-code-assist]

The prefix parameter is optional in the node inputs (line 104) and can be undefined at runtime. However, the getS3FileKeys function signature types prefix as a required string. To maintain type safety and prevent potential compilation or runtime issues under strict null checks, update the parameter type to string | undefined or make it optional.

Suggested change

	const getS3FileKeys = async (s3Client: S3Client, bucketName: string, prefix: string): Promise<string[]> => {
	const getS3FileKeys = async (s3Client: S3Client, bucketName: string, prefix?: string): Promise<string[]> => {