Skip to content

Extract common S3 operation patterns into reusable functions #31

@Asi0Flammeus

Description

@Asi0Flammeus

Description

S3 endpoints contain duplicated patterns for download/upload operations that should be extracted into reusable utilities.

Current Duplication

Similar code patterns repeated across:

  • /translate/pptx_s3
  • /translate/text_s3
  • /transcribe/audio_s3
  • /translate/course_s3

Common Patterns

  1. Download files from S3 to local temp directory
  2. Process files with different tools
  3. Upload results back to S3
  4. Progress tracking and error handling
  5. Task status management

Proposed Utilities

async def process_s3_files(
    input_keys: List[str],
    processor_func: Callable,
    output_prefix: Optional[str],
    task_id: str,
    **processor_kwargs
) -> List[str]:
    """Generic S3 file processing pipeline"""

async def download_s3_batch(keys: List[str], temp_dir: Path) -> List[Path]:
    """Download multiple files from S3"""

async def upload_s3_batch(files: List[Path], prefix: str) -> List[str]:
    """Upload multiple files to S3"""

Requirements

  • Extract common download/upload patterns
  • Create generic file processing pipeline
  • Maintain progress tracking functionality
  • Preserve error handling behavior
  • Add comprehensive type hints
  • Include proper documentation

Benefits

  • Eliminates code duplication
  • Consistent error handling across endpoints
  • Easier to add new S3 operations
  • Better testability of S3 operations
  • Simplified endpoint implementations

Acceptance Criteria

  • Common S3 patterns extracted to utilities
  • All S3 endpoints use shared functions
  • No loss of functionality during refactor
  • Progress tracking preserved
  • Error handling consistent

Related

Priority

Medium - Code quality and maintainability

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions