[FEATURE] Integrate rechtspraak full text segmentation into ETL pipeline

[shashankmc]

Feature Description

Semantic search is part of Case Law Explorer front end application. This requires segments of full text for each ECLI. The current ETL pipeline only extracts full text but doesn't segment it. During loading to the database, it would be easier to segment the full text and load it avoiding any adhoc triggers.

Problem Statement

Lack of segmentation of full text by calling the lambda function as part of the Rechtspraak ETL pipeline and loading to the Postgres

Proposed Solution

• Validate if segmentation is required as part of the ETL, some self hosted users might not need it.
• Integrate segmentation of full text by calling the lambda function as part of rechtspraak etl pipeline after extraction.
• Load segments into the postgres database.

Alternative Solutions

No

Use Case

Case law explorer application as a whole. Easier to maintain with everything in a single place.

User Story

Benefits

• Streamlining ETL pipeline.
• Avoids adhoc triggers for text segmentation
• Easier maintenance

Implementation Considerations

Affected Components

• Extraction layer
• Transformation layer
• Loading layer
• Airflow DAGs
• Documentation
• CI/CD
• Other:

Technical Requirements

• Access to Lambda function API for segmentation
• Access to Postgres DB.

Potential Challenges

• API calls failing
• Cold start problem with lambda functions (API gateway timeout issues). So multiple attempts will be required if timeout issue occurs with the first attempt.
• Concurrency issues (need to be coordinated with the settings of the lambda function)

Priority

Priority: Medium

Willingness to Contribute

• I am willing to submit a PR for this feature
• I can help with testing
• I can help with documentation
• I can provide use cases and examples

Additional Context

Related Issues

[shashankmc]

#42 needs to be completed first before this is pursued.