feat: import and export elements#4984
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 26381906 | Triggered | Generic Password | b035920 | .agents/skills/agent-browser/templates/authenticated-session.sh | View secret |
| 1509424 | Triggered | Generic Password | 220f6a6 | .devcontainer/docker-compose.yml | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Confidence Score: 3/5The core import/export logic is well-structured, but several defects flagged in the prior review round appear unaddressed — most critically the stale SAS URL cache in DownloadModal that permanently breaks repeated downloads after a network failure — and a few new quality concerns were identified in this pass. The ownership checks on export operations are now properly implemented via exportPermissionFilter, the single-mutation import flow correctly places onClose() outside any loop, and the ZIP parser uses maxOutputLength to bound per-entry decompression. However, DownloadModal.tsx still caches the download package before createDownload succeeds and does not clear the cache on failure, leaving expired SAS URLs stuck in the ref until a full page reload. In addition, the entry→ref mapping in importElementPackageBuffer relies on PostgreSQL insertion-order IDs without an explicit guard, and skippedElements is an always-zero constant that pollutes metrics. apps/frontend-manage/src/components/elements/manipulation/DownloadModal.tsx — stale SAS URL cache on failure; packages/graphql/src/services/elementImportExport.ts — entry-to-ref mapping by sort position and misleading skippedElements metric.
|
| Filename | Overview |
|---|---|
| packages/graphql/src/services/elementImportExport.ts | Core import/export service — 2357 lines; implements rate-limiting, HMAC-signed import tokens, ownership-scoped package parsing, duplicate detection via fingerprints, and transactional element creation. Export uses exportPermissionFilter correctly; import validates blob ownership via assertUserPackageBlob and verifies SHA-256 hash before applying. |
| packages/graphql/src/lib/zip.ts | Custom ZIP parser and writer. Validates paths (no traversal), checks entry count, accumulates uncompressed size from headers before decompressing, and uses inflateRawSync with maxOutputLength: uncompressedSize to cap per-entry decompression. CRC32 is verified post-decompress. |
| packages/graphql/src/services/packageStorage.ts | Handles Azure Blob / local dev storage for import/export ZIP packages. Correctly scopes all blob paths under imports/{userId}/ or exports/{userId}/ and enforces this via assertUserPackageBlob. SAS tokens are short-lived (15 min). Local mode has no auth on the Express routes (intentional for dev/test only). |
| apps/frontend-manage/src/components/elements/manipulation/DownloadModal.tsx | Download modal with SAS-URL caching via a parent-held ref. The cached entry is written before createDownload is awaited but not removed when it fails, meaning expired SAS URLs stay cached and cause repeated failures (flagged previously, still present). |
| apps/frontend-manage/src/components/elements/manipulation/UploadModal.tsx | Previously flagged issues (wrong MIME type, contents[0] guard) appear resolved — the dropzone now accepts application/zip and application/x-zip-compressed, and the validation flow is ZIP-based rather than JSON-based. |
| apps/frontend-manage/src/components/elements/details/ImportedElementsOverviewTable.tsx | Previously flagged onClose() inside a for-loop is resolved — the import now uses a single importElementPackage mutation rather than a per-element loop. onClose() is called once after the mutation succeeds. |
| packages/graphql/src/services/importExportFingerprints.ts | Computes stable SHA-256 fingerprints for elements and answer collections using canonical JSON serialisation, enabling advisory duplicate detection across import/export. Lazy per-user backfill processes up to 500 items per validate call. |
| packages/graphql/src/services/mediaStorage.ts | Handles staging and finalisation of imported media blobs in Azure, with deduplication via originalId. Cleanup of orphaned staged blobs is implemented. Stream reading is capped at MAX_IMPORT_EXPORT_MEDIA_BYTES. |
| apps/backend-docker/src/app.ts | Adds unauthenticated /api/import-export-packages/:blobName PUT/GET routes activated only in local dev/test mode. Path traversal is guarded in getLocalPackagePath. Production uses signed Azure SAS URLs instead. |
| packages/prisma/src/prisma/schema/migrations/20260707120000_import_export_fingerprints/migration.sql | Adds nullable importFingerprint TEXT columns to Element and AnswerCollection, with composite owner+fingerprint indexes for efficient duplicate lookup. No breaking schema changes. |
Sequence Diagram
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant FE as Frontend
participant GQL as GraphQL API
participant Pkg as PackageStorage (Azure/Local)
participant IE as elementImportExport
participant DB as Database
Note over FE,DB: Export Flow
FE->>GQL: getElementExportPackagePreview(elementIds)
GQL->>DB: findMany(elements + answerCollections, exportPermissionFilter)
DB-->>GQL: elements + collections
GQL-->>FE: preview (with warnings)
FE->>GQL: getElementExportPackageLink(elementIds)
GQL->>IE: createElementExportPackage
IE->>DB: fetch elements + media
IE->>IE: createZip(manifest + elements + media)
IE->>Pkg: uploadPrivateImportExportBlob
GQL-->>FE: downloadLink + filename + expiresAt
Note over FE,DB: Import Flow
FE->>GQL: prepareElementImportPackageUpload(filename)
GQL->>Pkg: createUploadSasUrl imports userId uuid.zip
GQL-->>FE: uploadSasURL + blobName
FE->>Pkg: PUT file (direct to Azure/local)
FE->>GQL: validateElementImportPackage(blobName)
GQL->>Pkg: downloadElementImportPackage (assertUserPackageBlob)
GQL->>IE: parseElementImportPackage (zip, schema, dep validation)
GQL->>DB: backfillMissingImportFingerprintsForOwner
GQL->>DB: findMany fingerprints for duplicate detection
GQL-->>FE: importToken (HMAC) + preview elements + warnings
FE->>GQL: importElementPackage(importToken, selectedRefs)
GQL->>IE: verifyImportToken (HMAC + userId + expiry)
GQL->>Pkg: downloadElementImportPackage
GQL->>IE: "hashBuffer == token.sha256"
GQL->>IE: stagePackageMediaFiles
GQL->>DB: transaction create answerCollections + elements
GQL-->>FE: importedElements count
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant FE as Frontend
participant GQL as GraphQL API
participant Pkg as PackageStorage (Azure/Local)
participant IE as elementImportExport
participant DB as Database
Note over FE,DB: Export Flow
FE->>GQL: getElementExportPackagePreview(elementIds)
GQL->>DB: findMany(elements + answerCollections, exportPermissionFilter)
DB-->>GQL: elements + collections
GQL-->>FE: preview (with warnings)
FE->>GQL: getElementExportPackageLink(elementIds)
GQL->>IE: createElementExportPackage
IE->>DB: fetch elements + media
IE->>IE: createZip(manifest + elements + media)
IE->>Pkg: uploadPrivateImportExportBlob
GQL-->>FE: downloadLink + filename + expiresAt
Note over FE,DB: Import Flow
FE->>GQL: prepareElementImportPackageUpload(filename)
GQL->>Pkg: createUploadSasUrl imports userId uuid.zip
GQL-->>FE: uploadSasURL + blobName
FE->>Pkg: PUT file (direct to Azure/local)
FE->>GQL: validateElementImportPackage(blobName)
GQL->>Pkg: downloadElementImportPackage (assertUserPackageBlob)
GQL->>IE: parseElementImportPackage (zip, schema, dep validation)
GQL->>DB: backfillMissingImportFingerprintsForOwner
GQL->>DB: findMany fingerprints for duplicate detection
GQL-->>FE: importToken (HMAC) + preview elements + warnings
FE->>GQL: importElementPackage(importToken, selectedRefs)
GQL->>IE: verifyImportToken (HMAC + userId + expiry)
GQL->>Pkg: downloadElementImportPackage
GQL->>IE: "hashBuffer == token.sha256"
GQL->>IE: stagePackageMediaFiles
GQL->>DB: transaction create answerCollections + elements
GQL-->>FE: importedElements count
Reviews (7): Last reviewed commit: "Merge branch 'v3' of github.com:uzh-bf/k..." | Re-trigger Greptile
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Too many files changed for review. ( Bypass the limit by tagging |
| ) : error ? ( | ||
| <UserNotification | ||
| type="error" | ||
| message={error || t('manage.elements.packagePreviewError')} |
ClickUp Links