Skip to content

Live e2e: documents connector object-store support on Azure Blob and GCS #178

Description

@abbccdda

Context

Follow-up from PR #171 review comment: #171 (comment)

PR #171 added S3 / object-store support to the documents connector (source path and image_store can be s3:// prefixes) and validated it with a live e2e run against real S3 (us-east-1). The object_store crate backing this also supports Azure Blob Storage and GCS, so the same code path should work against those providers — but it hasn't been exercised live.

Ask

Run the same live e2e coverage PR #171 did on S3, but against:

  • Azure Blob Storage — we have Azure credits.
  • GCP Cloud Storage — only if we already have GCP credits; otherwise skip/defer.

Scope

Mirror the S3 e2e checklist from the PR against each provider:

  • Nested corpus scan with glob + prefix scoping (sibling-prefix decoy and non-matching extensions excluded).
  • image_store page-render / crop uploads land as real objects with the correct Content-Type, and the write-probe object is cleaned up.
  • Self-ingestion guard holds when the image_store prefix sits inside the source path.
  • Negative startup cases fail with actionable errors: cross-container/cross-bucket image_store, nonexistent container/bucket, credentials smuggled into options, missing region/credentials in env, and a write preflight failure on a read-only target.

Notes / open questions

  • Confirm the env-only credential contract and region/endpoint config translate cleanly to Azure (AZURE_STORAGE_ACCOUNT / AZURE_STORAGE_ACCESS_KEY etc.) and GCS (GOOGLE_SERVICE_ACCOUNT / ADC), and update docs/documents.md if the per-provider setup differs.
  • The same-bucket constraint may need a per-provider analog (same container / same bucket).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions