Skip to content

Commit a102f0e

Browse files
NotYuShengNotYuSheng
authored
Merge pull request #88 from NotYuSheng/dev
Merge
2 parents 80b2c24 + 933a705 commit a102f0e

21 files changed

Lines changed: 725 additions & 10 deletions

File tree

‎.github/pull_request_template.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
### Summary
2+
3+
<!-- Provide a concise summary of the changes introduced in this PR. Link to relevant issue(s) if applicable. -->
4+
5+
Fixes #<issue_number>
6+
7+
---
8+
9+
### Changes Made
10+
11+
<!-- List out major changes in this PR. Be brief but specific. -->
12+
13+
-
14+
-
15+
-
16+
17+
---
18+
19+
### Context / Rationale
20+
21+
<!-- Why are these changes being made? Is it a bug fix, feature, refactor, etc.? Provide context to help reviewers understand your thought process. -->
22+
23+
---
24+
25+
### Related Docs or References
26+
27+
<!-- Include any design docs, spec references, screenshots, or relevant PRs -->
28+
29+
---
30+
31+
### FastAPI Application Checklist (**Delete if PR is not relevant**)
32+
33+
- [ ] API follows RESTful principles (nouns in routes, proper use of verbs)
34+
- [ ] All endpoints are async and use non-blocking I/O
35+
- [ ] `/health` endpoint is implemented and returns 200 OK
36+
- [ ] Long-running operations support both job polling (e.g., via /status/{job_id} or /progress/{job_id}) and optional webhooks (if a callback_url is provided).
37+
- [ ] If callback_url is present in the request payload, the service will POST job results to the specified URL upon completion.
38+
- [ ] If callback_url is not provided, the client can retrieve status and results via polling endpoints.
39+
- [ ] Job results are persisted or recoverable if needed
40+
- [ ] API schema (OpenAPI) is exposed and browsable at `/docs` or `/redoc`
41+
- [ ] Branch name follows conventions (e.g., `feature/*`, `bugfix/*`) — do **not** use `dev` directly
42+
43+
---
44+
45+
### General Checklist
46+
47+
- [ ] I have tested these changes locally
48+
- [ ] I have updated relevant documentation or added comments where needed
49+
- [ ] I have linked relevant issues and tagged reviewers
50+
- [ ] I have followed coding conventions and naming standards

‎.github/workflows/integration-test.yaml‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
name: Integration Test
2+
3+
on:
4+
push:
5+
branches:
6+
- dev
7+
pull_request:
8+
branches:
9+
- dev
10+
11+
jobs:
12+
cypress-run:
13+
name: Run Cypress Integration Tests
14+
runs-on: ubuntu-latest
15+
16+
steps:
17+
- name: Maximize build space
18+
uses: AdityaGarg8/remove-unwanted-software@v4.1
19+
with:
20+
remove-android: 'true'
21+
remove-haskell: 'true'
22+
remove-codeql: 'true'
23+
24+
- name: Checkout Repository
25+
uses: actions/checkout@v4
26+
27+
- name: Copy example.env to .env
28+
run: |
29+
echo "Copying example.env to .env..."
30+
find . -name "example.env" | while read f; do
31+
cp "$f" "$(dirname "$f")/.env"
32+
echo "✓ Created $(dirname "$f")/.env"
33+
done
34+
35+
- name: Build and run Compose Stack
36+
run: |
37+
docker compose \
38+
--file docker-compose.yml \
39+
up --detach --build
40+
41+
- name: Delete Docker build cache
42+
run: |
43+
docker builder prune --all --force
44+
45+
- name: Wait for all *_service containers (via Nginx)
46+
run: |
47+
echo "Detecting *_service containers..."
48+
SERVICE_NAMES=$(docker compose ps --services | grep '_service')
49+
50+
if [ -z "$SERVICE_NAMES" ]; then
51+
echo "No *_service services found. Exiting."
52+
exit 1
53+
fi
54+
55+
for NAME in $SERVICE_NAMES; do
56+
PREFIX=${NAME%_service}
57+
echo "Waiting for $NAME via http://localhost/$PREFIX/health..."
58+
59+
for i in {1..30}; do
60+
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost/$PREFIX/health")
61+
if [ "$STATUS" == "200" ]; then
62+
echo "$NAME is healthy"
63+
break
64+
fi
65+
echo "Still waiting for $NAME (HTTP $STATUS)..."
66+
sleep 3
67+
if [ "$i" -eq 30 ]; then
68+
echo "$NAME failed to become healthy. Exiting."
69+
exit 1
70+
fi
71+
done
72+
done
73+
74+
# - name: Wait for Streamlit frontend to be available
75+
# run: |
76+
# for i in {1..30}; do
77+
# STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8501)
78+
# if [ "$STATUS" == "200" ]; then
79+
# echo "Frontend is ready"
80+
# break
81+
# fi
82+
# echo "Waiting for frontend on :8501 (HTTP $STATUS)..."
83+
# sleep 3
84+
# if [ "$i" -eq 30 ]; then
85+
# echo "Frontend did not become ready. Exiting."
86+
# exit 1
87+
# fi
88+
# done
89+
90+
- name: Set up Node.js for Cypress
91+
uses: actions/setup-node@v4
92+
with:
93+
node-version: 20
94+
95+
- name: Install Cypress
96+
run: |
97+
npm init -y
98+
npm install cypress --save-dev
99+
100+
- name: Run Cypress tests (placeholder-safe)
101+
working-directory: cypress
102+
run: |
103+
npx cypress run || echo "Cypress run completed with non-zero exit (placeholder mode)"

‎.github/workflows/trivy.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
name: Trivy Scan
2+
3+
on:
4+
push:
5+
branches: [dev]
6+
workflow_dispatch:
7+
8+
jobs:
9+
trivy-scan:
10+
runs-on: ubuntu-latest
11+
env:
12+
TRIVY_EXCLUDE_SERVICES: ${{ secrets.TRIVY_EXCLUDE_SERVICES }}
13+
14+
steps:
15+
- name: Checkout repository
16+
uses: actions/checkout@v3
17+
18+
- name: Set up Docker Buildx
19+
uses: docker/setup-buildx-action@v3
20+
21+
- name: Set up QEMU
22+
uses: docker/setup-qemu-action@v3
23+
24+
- name: Install Trivy CLI
25+
run: |
26+
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
27+
28+
- name: Extract metadata
29+
id: meta
30+
run: |
31+
SHORT_SHA=$(git rev-parse --short HEAD)
32+
echo "SHORT_SHA=$SHORT_SHA" >> $GITHUB_ENV
33+
34+
git fetch --tags
35+
LATEST_TAG=$(git tag --list 'v*.*.*' | sort -V | tail -n1)
36+
FALLBACK_TAG="${LATEST_TAG:-v0.0.0}"
37+
echo "GIT_TAG=${FALLBACK_TAG}-${SHORT_SHA}" >> $GITHUB_ENV
38+
echo "REPO_OWNER_LC=${GITHUB_REPOSITORY_OWNER,,}" >> $GITHUB_ENV
39+
40+
- name: Copy example.env to .env
41+
run: |
42+
echo "Copying example.env to .env..."
43+
find . -name "example.env" | while read f; do
44+
cp "$f" "$(dirname "$f")/.env"
45+
echo "✓ Created $(dirname "$f")/.env"
46+
done
47+
48+
- name: Build all Docker Compose services
49+
run: |
50+
echo "Building services from docker-compose.yml..."
51+
docker compose build
52+
53+
- name: Run Trivy scan for all services
54+
run: |
55+
echo "Running Trivy scans..."
56+
SERVICES=$(docker compose config --services)
57+
58+
for SERVICE in $SERVICES; do
59+
if [[ " $TRIVY_EXCLUDE_SERVICES " =~ " $SERVICE " ]]; then
60+
echo "Skipping $SERVICE (excluded via secret)"
61+
continue
62+
fi
63+
64+
IMAGE_ID=$(docker images --filter=reference="*${SERVICE}*" --format "{{.ID}}" | head -n1)
65+
66+
if [[ -z "$IMAGE_ID" ]]; then
67+
echo "Skipping $SERVICE (no image built)"
68+
continue
69+
fi
70+
71+
echo "Scanning $SERVICE ($IMAGE_ID)..."
72+
trivy image --exit-code 1 --severity CRITICAL,HIGH --no-progress "$IMAGE_ID"
73+
done

‎chat_service/main.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
99
)
1010

11-
app = FastAPI()
11+
app = FastAPI(root_path="/chat")
1212

1313
app.include_router(health.router)
1414
app.include_router(chat.router)

‎cypress/cypress.config.js‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
const { defineConfig } = require('cypress');
2+
3+
module.exports = defineConfig({
4+
e2e: {
5+
baseUrl: 'http://localhost:8501',
6+
specPattern: 'cypress/e2e/**/*.cy.js',
7+
supportFile: false,
8+
},
9+
});

‎cypress/e2e/placeholder.cy.js‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
describe('Placeholder Test', () => {
2+
it('skips if app is not ready', () => {
3+
cy.request({
4+
url: '/',
5+
failOnStatusCode: false,
6+
}).then((res) => {
7+
if (res.status !== 200) {
8+
cy.log('App not ready, skipping test');
9+
return;
10+
}
11+
12+
// Add real checks later
13+
cy.contains('Streamlit');
14+
});
15+
});
16+
});

‎docker-compose.yml‎

Lines changed: 32 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,6 @@ services:
44
context: .
55
dockerfile: ./pdf_processor_service/Dockerfile
66
container_name: pdf_processor_service
7-
ports:
8-
- "8000:8000"
97
env_file:
108
- ./pdf_processor_service/.env
119

@@ -14,26 +12,53 @@ services:
1412
context: .
1513
dockerfile: ./chat_service/Dockerfile
1614
container_name: chat_service
17-
ports:
18-
- "8001:8000"
1915
env_file:
2016
- ./chat_service/.env
2117

22-
pdf_extraction:
18+
pdf_extraction_service:
2319
build:
2420
context: .
2521
dockerfile: ./pdf_extraction_service/Dockerfile
2622
container_name: pdf_extraction_service
27-
ports:
28-
- "8002:8000"
2923
depends_on:
3024
- minio
3125

26+
embedder_service:
27+
build:
28+
context: .
29+
dockerfile: ./embedder_service/Dockerfile
30+
container_name: embedder_service
31+
env_file:
32+
- ./embedder_service/.env
33+
depends_on:
34+
- chromadb
35+
36+
nginx:
37+
container_name: nginx
38+
build:
39+
context: .
40+
dockerfile: ./nginx/Dockerfile
41+
ports:
42+
- "80:8080"
43+
env_file:
44+
- ./nginx/.env
45+
depends_on:
46+
- pdf_extraction_service
47+
- pdf_processor_service
48+
- chat_service
49+
- embedder_service
50+
3251
redis:
3352
container_name: redis
3453
image: redis:7.4.4-alpine
3554
expose:
3655
- "6379"
56+
57+
chromadb:
58+
container_name: chromadb
59+
image: chromadb/chroma:1.0.13
60+
expose:
61+
- "5100"
3762

3863
minio:
3964
image: minio/minio

‎embedder_service/Dockerfile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
FROM python:3.13-slim
2+
3+
WORKDIR /app
4+
5+
ENV PYTHONPATH=/app/embedder_service:/app/shared_utils
6+
7+
RUN apt-get update && apt-get install -y build-essential && rm -rf /var/lib/apt/lists/*
8+
9+
# Copy requirements.txt from embedder_service folder in root context
10+
COPY embedder_service/requirements.txt .
11+
12+
RUN pip install --no-cache-dir -r requirements.txt
13+
14+
# Copy utils folder from root context into /app/utils
15+
COPY shared_utils ./shared_utils
16+
17+
# Copy embedder_service folder from root context into /app/embedder_service
18+
COPY embedder_service ./embedder_service
19+
20+
EXPOSE 8000
21+
22+
CMD ["sh", "-c", "PYTHONPATH=/app/embedder_service uvicorn embedder_service.main:app --host 0.0.0.0 --port 8000"]

‎embedder_service/example.env‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
EMBEDDING_MODEL=all-MiniLM-L6-v2

‎embedder_service/main.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
from fastapi import FastAPI
2+
from routers import health, embed
3+
import logging
4+
5+
6+
# Set up logger
7+
logging.basicConfig(
8+
level=logging.INFO,
9+
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
10+
)
11+
12+
app = FastAPI(root_path="/embedder")
13+
14+
app.include_router(health.router)
15+
app.include_router(embed.router)

0 commit comments

Comments
 (0)