Skip to content

Commit 9c62a33

Browse files
refactor: Vertical domain decomposition with FastAPI migration (#67)
* refactor: Phase 0 — FastAPI foundation with uvicorn, AppContext, EventBus, security Pre-Phase 0: Fix queue polling loops in helpers.py — replace time.sleep(5) polling with blocking queue.get(timeout=N) to eliminate 0-5s latency per queued item. Phase 0: Create the FastAPI app skeleton and core infrastructure: - AppContext dataclass replacing 130+ module-level globals - EventBus with thread-safe publish_sync() via call_soon_threadsafe - JWT auth (PyJWT, HS256 pinned, generation counter for revocation) - API key auth (factory pattern for 3 scopes) - OPDS HTTP Basic auth with bcrypt migration support - Global CSRF middleware (not per-route) - Security headers middleware (CSP, HSTS, X-Frame-Options, etc.) - Setup gate middleware for first-run setup - DomainError exception hierarchy with app-level handlers - Common utilities extracted from helpers.py (strings, dates, numbers, filesystem) - FastAPI lifespan with CherryPy WSGI bridge via a2wsgi - Uvicorn startup in Comicarr.py replacing CherryPy HTTP server - Health check at /api/health - Test context factory + 31 unit tests (all passing) Dependencies added: fastapi, uvicorn, sse-starlette, PyJWT, a2wsgi, python-multipart, httpx (dev) CherryPy CSRF and setup gate tools disabled under WSGI bridge to avoid double-checking with FastAPI middleware. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Phase 1 — System domain (auth, SSE, config, admin) Migrate auth, SSE, and system admin endpoints to FastAPI: Backend: - system/router.py: JWT login/logout, session check, setup, SSE stream, config CRUD, shutdown/restart, version, logs, jobs endpoints - system/service.py: Login verification with rate limiting and bcrypt migration, config management, initial setup, scheduler info - Login is sync def (bcrypt blocks event loop on ARM NAS hardware) - SSE via sse-starlette with per-subscriber async queues - JWT token revocation via generation counter Frontend: - api.ts: Switch AUTH_BASE to /api/auth, add apiRequest() for RESTful endpoints, convert login/setup to JSON body (from form-encoded) - useServerEvents.ts: Switch SSE endpoint to /api/events/stream, update session check URL 14 new tests (301 total, all passing) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Phase 2 — Metadata domain (ComicVine, Metron, MangaDex) Migrate metadata endpoints to FastAPI: - metadata/router.py: Search (comics + manga), comic/issue info, artwork serving, series image lazy-load, metatag (single/bulk/group) - metadata/service.py: Provider routing (CV/Metron/MangaDex), in_library flag enrichment, pagination passthrough, metatag delegation - metadata/queries.py: Cover image cache queries, comics needing images 15 new tests (316 total, all passing) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Phase 3 — Story Arcs domain (arcs, reading list, upcoming) Migrate story arc CRUD, reading list, and upcoming endpoints from CherryPy cmd= dispatch to FastAPI RESTful routes. Follows the vertical domain pattern established in Phases 0-2. Backend: storyarcs/queries.py (SQLAlchemy Core), service.py (module-level functions), router.py (11 endpoints). Frontend: useStoryArcs.ts and useQueue.ts switched from apiCall() to apiRequest() for all migrated endpoints. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Phase 4 — Series domain (comics, issues, imports) Migrate series CRUD, issue management, wanted list, and import endpoints from CherryPy cmd= dispatch to FastAPI RESTful routes. Includes reusable paginated_query() helper for consistent pagination. Backend: series/queries.py (SQLAlchemy Core with column projections), service.py (12 functions), router.py (16 endpoints). Frontend: useSeries.ts (8 hooks), useImport.ts (5 hooks), and useQueue.ts (wanted + bulk queue/unqueue) switched to apiRequest(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Phase 5 — Search domain (providers, RSS, manga) Migrate comic/manga search, add-to-library, force search, and RSS monitoring endpoints from CherryPy cmd= dispatch to FastAPI routes. Wraps existing search.py and rsscheck.py engines. Backend: search/queries.py, service.py (7 functions), router.py (7 endpoints under /api/search). Frontend: useSearch.ts (4 hooks) and useQueue.ts (forceSearch + removed unused apiCall import) switched to apiRequest(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Save ideation doc * feat: Phase 6 — Downloads domain (history, post-processing, DDL queue) Migrate download history, post-processing, and DDL queue management from CherryPy to FastAPI. Preserves PP_QUEUE integration and ComicRN/APC compatibility for external script callbacks. Backend: downloads/queries.py, service.py, router.py (7 endpoints under /api/downloads). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: Add workflow to trigger docs site deploy on push to main Sends a POST to the Cloudflare Pages deploy hook so the comicarr-docs static site rebuilds whenever this repo's main branch is updated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: Phases 2-8 — Complete FastAPI migration, remove CherryPy Finish the vertical domain decomposition plan: - Extract all 101 functions from helpers.py into domain modules (common/, core/security, series/, search/, downloads/, metadata/, storyarcs/, system/) with helpers.py retained as re-export shim - Create OPDS router with 12 Atom XML feed endpoints and HTTP Basic auth - Add DDL download endpoints (POST /ddl, GET /file/{id}) - Merge /rest API endpoints into series domain router - Delete webserve.py (12k lines), webstart.py, auth.py, opds.py, api.py - Remove CherryPy, cheroot, portend, Mako, a2wsgi from dependencies - Mount frontend SPA via CachedStaticFiles with content-hash caching - Migrate all frontend hooks from apiCall() to apiRequest() - Remove legacy API key query parameter auth (now JWT cookie-based) - Update MSW test handlers for RESTful URL patterns - Move LoginRateLimiter to core/security.py - Fix all broken imports from deleted modules All tests pass: 316 backend, 26 frontend. Linting clean. TypeScript clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: Resolve CI lint and test failures - Fix import sorting in helpers.py, maintenance_webstart.py, webviewer.py - Set comicarr.LOG_LEVEL in test_system_domain.py to prevent TypeError when logger.info checks LOG_LEVEL > 0 in CI Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: Apply ruff format to all new/modified files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: Add BYOK AI features ideation document Ideation for 8 BYOK AI features: collection intelligence dashboard, natural language library search, story arc reading order generator, weekly pull list curation, filename parsing fallback, search query expansion, ComicInfo.xml metadata enrichment, and metadata conflict reconciliation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: Add docs/ideation/ to .gitignore Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: Remove AI-generated comment slop from FastAPI migration files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: Address all code review findings (P1-P3) Security (P1): - Add config write allowlist to prevent privilege escalation via PUT /api/config - Change CSP from report-only to enforcing mode - Remove /api from CSRF exempt prefixes (was exempting all endpoints) - Add Depends(require_session) to SSE endpoint - Fix path traversal: fail closed when no dirs configured, use commonpath - Add path validation to OPDS file serving endpoints - Fix datetime.utcnow() deprecation → datetime.now(timezone.utc) - Fix remove_apikey broken loop logic Performance (P2): - Store ThreadPoolExecutor reference and shut down in lifespan teardown - Replace OPDS N+1 queries with batch loads (recent, storyarc, readlist) - Add 30-second TTL cache to havetotals() - Fix OPDS XML escaping (escape in _entry_xml/_feed_xml, quoteattr for attrs) - Deduplicate search logic (metadata delegates to search service) - Extract shared paginated_query to core/database.py Cleanup (P3): - Remove dead code: multikeysort, checked, radio, urlretrieve (~99 LOC) - Fix all 56 bare except: clauses → except Exception: - Remove unused ctx parameter from 21 service functions + callers - Extract _build_arc_summary to deduplicate storyarcs service - Fix DDL_QUEUED type mismatch (list → set, append → add) - Remove debug print(flipflop) from cleanhtml - Simplify clean_url to url.strip() All tests pass: 306 backend, 26 frontend. Lint clean. TypeScript clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent b7ed820 commit 9c62a33

80 files changed

Lines changed: 14140 additions & 24649 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/deploy-docs.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
name: Trigger docs site deploy
2+
3+
on:
4+
push:
5+
branches: [main]
6+
7+
jobs:
8+
deploy-docs:
9+
runs-on: ubuntu-latest
10+
steps:
11+
- name: Trigger Cloudflare Pages deploy
12+
run: curl -s -X POST "${{ secrets.CLOUDFLARE_DOCS_DEPLOY_HOOK }}"

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ AGENTS.md
5858
docs/plans/
5959
docs/solutions/
6060
docs/brainstorms/
61+
docs/ideation/
6162

6263
# Pencil design files
6364
*.pen

Comicarr.py

Lines changed: 33 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -134,10 +134,8 @@ def check_it(self):
134134
filechecker,
135135
logger,
136136
maintenance,
137-
maintenance_webstart,
138137
dependency_check,
139138
versioncheck,
140-
webstart,
141139
)
142140

143141
import argparse
@@ -476,8 +474,12 @@ def main():
476474
loggermode = '[MAINTENANCE-MODE]'
477475
versioncheck.versionload()
478476

479-
# Try to start the server.
480-
maintenance_webstart.initialize(maint_config)
477+
# Try to start the maintenance server.
478+
try:
479+
from comicarr import maintenance_webstart
480+
maintenance_webstart.initialize(maint_config)
481+
except ImportError:
482+
logger.warn('%s maintenance_webstart not available (CherryPy removed)' % loggermode)
481483

482484
restart_method = True #True will restart, False will shutdown.
483485

@@ -541,8 +543,12 @@ def main():
541543

542544
#restart automatically after maintenance has completed...
543545

544-
maintenance_webstart.shutdown()
545-
logger.info('%s Maintenance webserver has been shut down.'% (loggermode))
546+
try:
547+
from comicarr import maintenance_webstart
548+
maintenance_webstart.shutdown()
549+
logger.info('%s Maintenance webserver has been shut down.'% (loggermode))
550+
except (ImportError, NameError):
551+
pass
546552
comicarr.shutdown(restart=restart_method, maintenance=True)
547553

548554
# Force the http port if neccessary
@@ -552,37 +558,6 @@ def main():
552558
else:
553559
http_port = int(comicarr.CONFIG.HTTP_PORT)
554560

555-
# Check if pyOpenSSL is installed. It is required for certificate generation
556-
# and for cherrypy.
557-
if comicarr.CONFIG.ENABLE_HTTPS:
558-
try:
559-
import OpenSSL
560-
except ImportError:
561-
logger.warn("The pyOpenSSL module is missing. Install this " \
562-
"module to enable HTTPS. HTTPS will be disabled.")
563-
comicarr.CONFIG.ENABLE_HTTPS = False
564-
565-
# Try to start the server. Will exit here is address is already in use.
566-
web_config = {
567-
'http_port': http_port,
568-
'http_host': comicarr.CONFIG.HTTP_HOST,
569-
'http_root': comicarr.CONFIG.HTTP_ROOT,
570-
'enable_https': comicarr.CONFIG.ENABLE_HTTPS,
571-
'https_cert': comicarr.CONFIG.HTTPS_CERT,
572-
'https_key': comicarr.CONFIG.HTTPS_KEY,
573-
'https_chain': comicarr.CONFIG.HTTPS_CHAIN,
574-
'http_username': comicarr.CONFIG.HTTP_USERNAME,
575-
'http_password': comicarr.CONFIG.HTTP_PASSWORD,
576-
'authentication': comicarr.CONFIG.AUTHENTICATION,
577-
'login_timeout': comicarr.CONFIG.LOGIN_TIMEOUT,
578-
'cherrypy_logging': comicarr.CONFIG.CHERRYPY_LOGGING,
579-
'opds_enable': comicarr.CONFIG.OPDS_ENABLE,
580-
'opds_authentication': comicarr.CONFIG.OPDS_AUTHENTICATION,
581-
'opds_username': comicarr.CONFIG.OPDS_USERNAME,
582-
'opds_password': comicarr.CONFIG.OPDS_PASSWORD,
583-
'opds_pagesize': comicarr.CONFIG.OPDS_PAGESIZE,
584-
}
585-
586561
# Generate setup token if first-run setup is needed
587562
if not comicarr.CONFIG.HTTP_USERNAME or not comicarr.CONFIG.HTTP_PASSWORD:
588563
import secrets
@@ -591,9 +566,6 @@ def main():
591566
logger.info('[SETUP] Setup token: %s' % comicarr.SETUP_TOKEN)
592567
logger.info('[SETUP] Provide this token when setting up credentials via the web interface.')
593568

594-
# Try to start the server.
595-
webstart.initialize(web_config)
596-
597569
#check for version here after web server initialized so it doesn't try to repeatidly hit github
598570
#for version info if it's already running
599571
versioncheck.versionload()
@@ -610,26 +582,28 @@ def main():
610582

611583
signal.signal(signal.SIGTERM, handler_sigterm)
612584

613-
while True:
614-
if not comicarr.SIGNAL:
615-
try:
616-
time.sleep(1)
617-
except KeyboardInterrupt:
618-
comicarr.GLOBAL_MESSAGES = {'status': 'success', 'event': 'shutdown', 'message': 'Now shutting down system.'}
619-
time.sleep(1)
620-
comicarr.SIGNAL = 'shutdown'
621-
else:
622-
logger.info('Received signal: ' + comicarr.SIGNAL)
623-
if comicarr.SIGNAL == 'shutdown':
624-
comicarr.GLOBAL_MESSAGES = {'status': 'success', 'event': 'shutdown', 'message': 'Now shutting down system.'}
625-
time.sleep(2)
626-
comicarr.shutdown()
627-
elif comicarr.SIGNAL == 'restart':
628-
comicarr.shutdown(restart=True)
629-
else:
630-
comicarr.shutdown(restart=True, update=True)
585+
import uvicorn
631586

632-
comicarr.SIGNAL = None
587+
ssl_kwargs = {}
588+
if comicarr.CONFIG.ENABLE_HTTPS:
589+
ssl_kwargs["ssl_certfile"] = comicarr.CONFIG.HTTPS_CERT
590+
ssl_kwargs["ssl_keyfile"] = comicarr.CONFIG.HTTPS_KEY
591+
592+
uvicorn.run(
593+
"comicarr.app.main:app",
594+
host=comicarr.CONFIG.HTTP_HOST,
595+
port=http_port,
596+
log_level="info",
597+
workers=1,
598+
**ssl_kwargs,
599+
)
600+
601+
if comicarr.SIGNAL == 'restart':
602+
comicarr.shutdown(restart=True)
603+
elif comicarr.SIGNAL == 'update':
604+
comicarr.shutdown(restart=True, update=True)
605+
else:
606+
comicarr.shutdown()
633607

634608
return
635609

comicarr/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -235,7 +235,7 @@ def __exit__(self, exc_type, exc_val, exc_tb):
235235
ISSUE_WATCH_LIST = queue.Queue()
236236
MASS_REFRESH = None
237237
REFRESH_QUEUE = queue.Queue()
238-
DDL_QUEUED = []
238+
DDL_QUEUED = set()
239239
DDL_STUCK_NOTIFIED = set()
240240
DDL_HEALTH_SCHEDULER = None
241241
PACK_ISSUEIDS_DONT_QUEUE = {}

0 commit comments

Comments
 (0)