Skip to content

Commit fc7b789

Browse files
committed
feat: version post content, add Wikipedia support, and ship Firefox pipeline
Introduce a versioned content model across the API/database: - Add PostVersion, ContentBlob, ImageOccurrenceSet/ImageOccurrence, and PostVersionViewCredit. - Move Investigation linkage from (postId, contentHash) to postVersionId. - Extend Platform metadata with WikipediaMeta and add WIKIPEDIA platform support. Add and align migrations for the new model: - Add 0011 post image occurrences, 0012 wikipedia platform/meta, 0014 wikipedia title index drop, and 0015 post versions backfill. - Backfill uses deterministic occurrence payload hashing and version hashing (sha256(contentHash + '\n' + occurrencesHash)). - Add collision guards and restore DB invariants/lineage enforcement via constraints + triggers. - Enforce PostVersion provenance consistency (SERVER_VERIFIED vs CLIENT_FALLBACK fields). Refactor extension-facing API contracts: - Introduce post.registerObservedVersion and keep post.recordViewAndGetStatus as a postVersionId-based view-credit/status call. - Update investigateNow to accept postVersionId. - Extend batchStatus lookup with versionHash-aware inputs. Expand canonicalization and investigation internals: - Add canonical-resolution service and dependency-injected request-identity/openai-key-validation-core helpers. - Extend canonical content fetching with Wikipedia revision-based fetch/normalization. - Move investigation orchestration to postVersion-backed content/image context. - Add image-occurrence-aware multimodal prompt assembly with resolved/missing/omitted image states and validation context notes. Upgrade extension architecture and platform coverage: - Add Wikipedia URL parsing + adapter extraction (content/image occurrences/metadata). - Split background logic into reusable modules (api-client-core, browser-compat, investigation/page-content state helpers). - Switch background view flow to registerObservedVersion -> recordViewAndGetStatus and use postVersionId for investigateNow. - Keep popup/options/content plumbing aligned with new protocol and schema changes. Ship cross-browser build/release flow: - Build Chrome + Firefox artifacts (dist + dist/firefox) with manifest transformation and Firefox background/content bundling. - Strengthen artifact assertions for both browser targets. - Update release workflow to run full checks, package/sign/publish Chrome and Firefox zips, and report bundle sizes. - Update PR checks to include formatting and coverage gates. Improve tooling, tests, and documentation: - Add Prettier config/ignore, format scripts, and c8 coverage configs for shared/api/extension. - Tighten ESLint strict-boolean-expressions and clean type-safe conditionals across code. - Expand unit/integration/e2e coverage across shared/api/extension for new protocol/platform behavior. - Update README, SPEC, and AGENTS docs for post-version flow, Wikipedia support, and Chrome+Firefox usage.
1 parent 51576d6 commit fc7b789

167 files changed

Lines changed: 10597 additions & 3976 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/pr-checks.yml

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ jobs:
124124
run: pnpm db:generate
125125

126126
- name: Lint and dead-code checks
127-
run: pnpm lint && pnpm knip
127+
run: pnpm lint && pnpm knip && pnpm format:check
128128

129129
test:
130130
needs: [should-run]
@@ -174,6 +174,9 @@ jobs:
174174
- name: Run tests
175175
run: pnpm test
176176

177+
- name: Run coverage checks
178+
run: pnpm test:coverage
179+
177180
build:
178181
needs: [should-run]
179182
if: >-
@@ -210,6 +213,38 @@ jobs:
210213
- name: Build workspace
211214
run: pnpm build
212215

216+
- name: Summarize extension bundle sizes
217+
shell: bash
218+
run: |
219+
set -euo pipefail
220+
221+
chrome_dir="extension/dist"
222+
firefox_dir="extension/dist/firefox"
223+
224+
chrome_bytes="$(du -sb "$chrome_dir" | awk '{print $1}')"
225+
firefox_bytes="$(du -sb "$firefox_dir" | awk '{print $1}')"
226+
227+
{
228+
echo "### Extension Bundle Size Report"
229+
echo
230+
echo "| Artifact | Bytes | MiB |"
231+
echo "|---|---:|---:|"
232+
echo "| Chrome dist dir | ${chrome_bytes} | $(awk "BEGIN {printf \"%.2f\", ${chrome_bytes}/1048576}") |"
233+
echo "| Firefox dist dir | ${firefox_bytes} | $(awk "BEGIN {printf \"%.2f\", ${firefox_bytes}/1048576}") |"
234+
echo
235+
echo "#### Largest JS files (top 10)"
236+
echo
237+
echo "| File | Bytes | KiB |"
238+
echo "|---|---:|---:|"
239+
} >> "$GITHUB_STEP_SUMMARY"
240+
241+
find "$chrome_dir" -type f -name "*.js" -printf '%s\t%p\n' \
242+
| sort -nr \
243+
| head -n 10 \
244+
| while IFS=$'\t' read -r size path; do
245+
echo "| ${path} | ${size} | $(awk "BEGIN {printf \"%.2f\", ${size}/1024}") |" >> "$GITHUB_STEP_SUMMARY"
246+
done
247+
213248
no-relevant-changes:
214249
needs: [should-run]
215250
if: >-

.github/workflows/release-extension.yml

Lines changed: 108 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,27 @@ permissions:
1010
contents: write
1111

1212
jobs:
13-
release:
13+
build-and-test:
1414
runs-on: ubuntu-latest
15+
outputs:
16+
version: ${{ steps.version.outputs.version }}
1517
defaults:
1618
run:
1719
working-directory: src/typescript
1820
env:
1921
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/postgres
22+
services:
23+
postgres:
24+
image: postgres:17
25+
env:
26+
POSTGRES_PASSWORD: postgres
27+
ports:
28+
- 5432:5432
29+
options: >-
30+
--health-cmd "pg_isready -U postgres"
31+
--health-interval 10s
32+
--health-timeout 5s
33+
--health-retries 5
2034
steps:
2135
- name: Checkout
2236
uses: actions/checkout@v4
@@ -36,17 +50,23 @@ jobs:
3650
- name: Install dependencies
3751
run: pnpm install --frozen-lockfile
3852

53+
- name: Install Playwright Chromium
54+
run: pnpm --filter @openerrata/extension exec playwright install --with-deps chromium
55+
3956
- name: Generate Prisma client
4057
run: pnpm db:generate
4158

4259
- name: Typecheck
4360
run: pnpm typecheck
4461

4562
- name: Lint and dead-code checks
46-
run: pnpm lint && pnpm knip
63+
run: pnpm lint && pnpm knip && pnpm format:check
64+
65+
- name: Run full test suite
66+
run: pnpm test
4767

48-
- name: Run extension tests
49-
run: pnpm run test:unit:shared && pnpm run test:unit:extension
68+
- name: Run coverage checks
69+
run: pnpm test:coverage
5070

5171
- name: Build extension
5272
run: pnpm --filter @openerrata/extension build
@@ -61,28 +81,85 @@ jobs:
6181
echo "version=$(date +%Y%m%d)-${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
6282
fi
6383
64-
- name: Package extension zip
84+
- name: Package Chrome extension zip
6585
run: |
6686
cd extension/dist
67-
zip -r "$GITHUB_WORKSPACE/openerrata-extension-${{ steps.version.outputs.version }}.zip" .
87+
zip -r "$GITHUB_WORKSPACE/openerrata-extension-chrome-${{ steps.version.outputs.version }}.zip" .
88+
89+
- name: Package Firefox extension zip
90+
run: |
91+
cd extension/dist/firefox
92+
zip -r "$GITHUB_WORKSPACE/openerrata-extension-firefox-${{ steps.version.outputs.version }}.zip" .
93+
94+
- name: Enforce bundle size hard limit
95+
shell: bash
96+
run: |
97+
set -euo pipefail
98+
99+
chrome_zip="$GITHUB_WORKSPACE/openerrata-extension-chrome-${{ steps.version.outputs.version }}.zip"
100+
firefox_zip="$GITHUB_WORKSPACE/openerrata-extension-firefox-${{ steps.version.outputs.version }}.zip"
101+
limit_bytes=5242880
102+
103+
chrome_bytes="$(stat -c%s "$chrome_zip")"
104+
firefox_bytes="$(stat -c%s "$firefox_zip")"
105+
106+
if [ "$chrome_bytes" -gt "$limit_bytes" ]; then
107+
echo "Chrome zip exceeds 5 MiB: ${chrome_bytes} bytes"
108+
exit 1
109+
fi
110+
111+
if [ "$firefox_bytes" -gt "$limit_bytes" ]; then
112+
echo "Firefox zip exceeds 5 MiB: ${firefox_bytes} bytes"
113+
exit 1
114+
fi
115+
116+
- name: Verify Firefox zip integrity
117+
run: unzip -t "$GITHUB_WORKSPACE/openerrata-extension-firefox-${{ steps.version.outputs.version }}.zip"
118+
119+
- name: Lint Firefox extension package
120+
run: npx -y web-ext@8.9.0 lint --source-dir extension/dist/firefox
121+
122+
- name: Upload packaged zips
123+
uses: actions/upload-artifact@v4
124+
with:
125+
name: openerrata-extension-zips-${{ steps.version.outputs.version }}
126+
path: |
127+
${{ github.workspace }}/openerrata-extension-chrome-${{ steps.version.outputs.version }}.zip
128+
${{ github.workspace }}/openerrata-extension-firefox-${{ steps.version.outputs.version }}.zip
129+
130+
release:
131+
needs: [build-and-test]
132+
if: github.ref_type == 'tag'
133+
runs-on: ubuntu-latest
134+
steps:
135+
- name: Download packaged zips
136+
uses: actions/download-artifact@v4
137+
with:
138+
name: openerrata-extension-zips-${{ needs.build-and-test.outputs.version }}
139+
path: .
68140

69141
- name: Create GitHub Release
70-
if: github.ref_type == 'tag'
71142
uses: softprops/action-gh-release@v2
72143
with:
73-
files: openerrata-extension-${{ steps.version.outputs.version }}.zip
144+
files: |
145+
openerrata-extension-chrome-${{ needs.build-and-test.outputs.version }}.zip
146+
openerrata-extension-firefox-${{ needs.build-and-test.outputs.version }}.zip
74147
generate_release_notes: true
75148

76-
- name: Upload artifact (non-tag builds)
77-
if: github.ref_type != 'tag'
78-
uses: actions/upload-artifact@v4
149+
publish:
150+
needs: [build-and-test, release]
151+
if: github.ref_type == 'tag'
152+
runs-on: ubuntu-latest
153+
environment: extension-store-publish
154+
steps:
155+
- name: Download packaged zips
156+
uses: actions/download-artifact@v4
79157
with:
80-
name: openerrata-extension-${{ steps.version.outputs.version }}
81-
path: openerrata-extension-${{ steps.version.outputs.version }}.zip
158+
name: openerrata-extension-zips-${{ needs.build-and-test.outputs.version }}
159+
path: .
82160

83161
- name: Publish to Chrome Web Store
84162
if: >-
85-
github.ref_type == 'tag' &&
86163
env.CHROME_EXTENSION_ID != '' &&
87164
env.CHROME_CLIENT_ID != '' &&
88165
env.CHROME_CLIENT_SECRET != '' &&
@@ -94,9 +171,25 @@ jobs:
94171
CHROME_REFRESH_TOKEN: ${{ secrets.CHROME_REFRESH_TOKEN }}
95172
run: |
96173
npx -y chrome-webstore-upload-cli@3 upload \
97-
--source "$GITHUB_WORKSPACE/openerrata-extension-${{ steps.version.outputs.version }}.zip" \
174+
--source "openerrata-extension-chrome-${{ needs.build-and-test.outputs.version }}.zip" \
98175
--extension-id "$CHROME_EXTENSION_ID" \
99176
--client-id "$CHROME_CLIENT_ID" \
100177
--client-secret "$CHROME_CLIENT_SECRET" \
101178
--refresh-token "$CHROME_REFRESH_TOKEN" \
102179
--auto-publish
180+
181+
- name: Publish to Firefox Add-ons
182+
if: >-
183+
env.FIREFOX_JWT_ISSUER != '' &&
184+
env.FIREFOX_JWT_SECRET != ''
185+
env:
186+
FIREFOX_JWT_ISSUER: ${{ secrets.FIREFOX_JWT_ISSUER }}
187+
FIREFOX_JWT_SECRET: ${{ secrets.FIREFOX_JWT_SECRET }}
188+
run: |
189+
mkdir -p firefox-upload
190+
unzip -q "openerrata-extension-firefox-${{ needs.build-and-test.outputs.version }}.zip" -d firefox-upload
191+
npx -y web-ext@8.9.0 sign \
192+
--source-dir firefox-upload \
193+
--api-key "$FIREFOX_JWT_ISSUER" \
194+
--api-secret "$FIREFOX_JWT_SECRET" \
195+
--channel listed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ src/typescript/api/src/lib/generated/prisma/
99

1010
# Build artifacts
1111
*.tsbuildinfo
12+
**/coverage/
1213
src/typescript/extension/test-results/
1314
src/typescript/extension/playwright-report/
1415

README.md

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -32,14 +32,18 @@ inspectable by users.
3232

3333
**Chrome Web Store** — Coming soon.
3434

35+
**Firefox Add-ons (AMO)** — Coming soon.
36+
3537
**Manual install** from [GitHub Releases](https://github.com/ZeroPathAI/openerrata/releases):
3638

37-
1. Download the latest `openerrata-extension-*.zip` from the [Releases page](https://github.com/ZeroPathAI/openerrata/releases)
39+
1. Download the latest browser-specific zip from the [Releases page](https://github.com/ZeroPathAI/openerrata/releases):
40+
- Chrome: `openerrata-extension-chrome-*.zip`
41+
- Firefox: `openerrata-extension-firefox-*.zip`
3842
2. Unzip the file
39-
3. Open `chrome://extensions` in Chrome
40-
4. Enable **Developer mode** (toggle in the top-right)
41-
5. Click **Load unpacked** and select the unzipped folder
42-
6. (Optional) Set your [OpenAI API Key](https://developers.openai.com/api/docs/quickstart) in the settings
43+
3. Install by browser:
44+
- Chrome: open `chrome://extensions`, enable **Developer mode**, then click **Load unpacked** and select the unzipped folder.
45+
- Firefox: open `about:debugging#/runtime/this-firefox`, click **Load Temporary Add-on...**, then choose the extension `manifest.json` in the unzipped folder.
46+
4. (Optional) Set your [OpenAI API Key](https://developers.openai.com/api/docs/quickstart) in the settings
4347

4448
Errata are first pulled from the public instance before being regenerated. If
4549
you do not have an OpenAI API key, you can receive the errata generated by
@@ -91,7 +95,7 @@ src/
9195
└── typescript/
9296
├── shared/ # @openerrata/shared — types, Zod schemas, normalization
9397
├── api/ # @openerrata/api — SvelteKit + tRPC backend, Prisma, job queue
94-
├── extension/ # @openerrata/extension — Chrome MV3 browser extension
98+
├── extension/ # @openerrata/extension — WebExtension (Chrome + Firefox)
9599
└── pulumi/ # @openerrata/pulumi — deploys the Helm chart for hosted env
96100
```
97101

@@ -156,11 +160,19 @@ pnpm db:generate # Regenerate Prisma client
156160

157161
### Loading the Extension
158162

159-
After `pnpm dev:ext`, load the built extension as an unpacked extension in Chrome:
163+
After `pnpm dev:ext`, load the built extension as an unpacked extension:
164+
165+
1. Chrome:
166+
- Navigate to `chrome://extensions`
167+
- Enable "Developer mode"
168+
- Click "Load unpacked" and select `src/typescript/extension/dist`
169+
2. Firefox:
170+
- Navigate to `about:debugging#/runtime/this-firefox`
171+
- Click "Load Temporary Add-on..."
172+
- Select `src/typescript/extension/dist/firefox/manifest.json`
160173

161-
1. Navigate to `chrome://extensions`
162-
2. Enable "Developer mode"
163-
3. Click "Load unpacked" and select the extension build output directory
174+
For signed Firefox releases, set `FIREFOX_GECKO_ID=<your-addon-id>` before
175+
building to control the generated `browser_specific_settings.gecko.id`.
164176

165177
## Deployment
166178

0 commit comments

Comments
 (0)