M7 — Delivery, scale & more FeedTypes (final milestone) - #117
Merged
Conversation
- OutputWriter orchestrates per-context output: buffers one part at a time and
rotates to {contextKey}-{n}.{ext} when maxItems/maxBytes (format default merged
with Feed.formatConfig['split']) is exceeded, reusing the same writer/config
(CSV re-emits its header per part). Single-part output stays byte-identical to
before (un-suffixed path).
- SplitManifest strategy (registry): none (self-describing parts) / supplemental
(google_rss -> a <manifest> naming the parts at the canonical path).
- Optional gzip (Feed.formatConfig['gzip']) via a bounded zlib.deflate filter,
.gz extension, per part.
- GenerationResult carries all written paths (parts + manifest) for delivery.
+7 test files (594 total); all gates (incl. PHP 8.1) green.
Five thinner feed types (SS8.2-8.6) with their DataSources + resolvers, proving the engine is resource-agnostic: - order [CHANNEL] (completed orders), customer [], taxon [LOCALE], product_review [CHANNEL,LOCALE] (accepted), promotion [CHANNEL] (non-archived). - Resolver names namespaced where they collide in the global resolver registry (promotion_*, review_created_at). - Acceptance: an Order CSV export and a Taxon feed generate via the real FeedGenerator + Doctrine data sources against the database. +116 tests (710 total); 7 feed types registered; all gates (incl. PHP 8.1) green.
- DeliveryTarget (child of Feed): transport, transportConfig, pathTemplate
(placeholders), match {channel?,locale?,currency?}. Pluggable
DeliveryTransport registry: local (always) + ftp/sftp/s3 gated on class_exists
(Flysystem adapters are the integrator's optional install; listed under
composer suggest, not required).
- DeliveryMatcher (wildcard per dimension, empty=all) + PathTemplateRenderer.
- DeliveryService pushes the whole file set (parts + manifest) to every matching
target, best-effort + isolated (per-target/per-file try-catch, never throws),
recording {target,path,status,error?} on FeedContextResult.deliveries.
- Hooked into finalize: MoveGeneratedFeedSubscriber delivers a context only after
it is promoted to canonical AND published; a gate-blocked context is never
delivered. FeedContextResult carries channel/locale/currency codes + paths.
- DeliveryTargetType admin form + a deliveryTargets collection on the feed form.
+24 tests (734 total); all gates (incl. PHP 8.1) green.
setono:feed:process already processes every enabled feed by default; add an explicit --all flag (the documented cron form) and a Scheduling section in the README with a crontab example. Fix the README's stale command name (setono:sylius-feed:process -> setono:feed:process).
For a large single-source plain feed, GenerateFeedContextHandler partitions the
context into id-ranges (ChunkPartitioner) and fans out GenerateFeedChunk messages
instead of generating inline. Each chunk renders a body-only partial
({contextKey}.chunk-{i}); a per-context FeedChunk barrier table tracks completion
with an atomic idempotent counter, so exactly one chunk triggers
FinalizeFeedContext, which concatenates the ordered partials (preamble + bodies +
epilogue) via OutputWriter::finalizeFromPartials.
- Byte-identical to the inline single-chunk path (asserted at unit + functional
level over the same catalog).
- Chunk-level resumability: a failed chunk is retried by Messenger without failing
the feed; the finalize claim (atomic DELETE of the barrier rows) fires exactly
once; a stray re-dispatched finalize is a no-op.
- Shared FeedContextFinalizer (record -> gate -> counter -> complete) used by both
the inline and fan-out paths so they finalize identically.
- Fan-out engages only for single-source, non-split, non-gzip feeds (threshold
formatConfig['chunk']['size'], default 10k); split/gzip stay on the inline path.
- CsvWriter BOM moved to writePreamble so body-only partials omit it.
+4 test files (747 total); all gates (incl. PHP 8.1 + byte-identical + resume) green.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #117 +/- ##
============================================
- Coverage 95.25% 92.17% -3.09%
- Complexity 1231 1727 +496
============================================
Files 161 228 +67
Lines 3057 4382 +1325
============================================
+ Hits 2912 4039 +1127
- Misses 145 343 +198 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
On --prefer-lowest, league/flysystem-local resolves to 3.15 (its LocalFilesystemAdapter implements League\Flysystem\ChecksumProvider, added in flysystem 3.14) while league/flysystem could still resolve to 3.0, whose interface set lacks ChecksumProvider -> 'Interface ChecksumProvider not found' broke kernel boot + PHPStan on every lowest-deps CI leg once the delivery transports (Chunk 3) began instantiating a Filesystem. Raising league/flysystem + -local to ^3.15 keeps the pair matched; flysystem-bundle stays ^2.4 || ^3.0 (Sylius pins ^2.4). Highest (flysystem 3.35) unchanged; 747 tests green.
…sv typing) Two lowest-deps issues the CI matrix surfaced: 1. symfony/translation's PhpAstExtractor (booted by PHPStan's Symfony container loader) is version-coupled to nikic/php-parser: <=6.4.2 call ParserFactory:: create() (php-parser 4), >=6.4.3 call createForHostVersion() (php-parser 5). --prefer-lowest resolved a mismatched pair -> 'Call to undefined method'. Pin require-dev to the coherent forward combo: nikic/php-parser ^5.0 + symfony/translation ^6.4.3. 2. The fan-out byte-identity tests did array_map(fn(array $row)...) over league/csv records, which are typed mixed on the 9.6 floor (param too narrow) yet array on newer (guard redundant). Extract a static rowId(mixed): mixed helper -- a hard type boundary clean on both. Highest unchanged (php-parser 5.7 + translation 6.4.42); 747 tests + PHPStan max (incl. PHP 8.1) green on both highest and the lowest resolve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M7 — Delivery, scale & more FeedTypes (§6.3, §8.2–8.6, §12) — the final milestone
Five chunks; 747 tests; the delivery-targets form + all 7 feed types are browser-verified.
gzip + splitting + SplitManifest
OutputWriterrotates a context into{contextKey}-{n}.{ext}parts when amaxItems/maxByteslimit (format default merged with
Feed.formatConfig['split']) is hit, reusing the samewriter/config (CSV re-emits its header per part). Single-part output stays byte-identical.
none(self-describing parts) /supplemental(google_rss → a<manifest>naming the parts at the canonical path).Feed.formatConfig['gzip']) via a boundedzlib.deflatefilter, per part.Multi-chunk fan-out + resumability
GenerateFeedChunkmessagesrender body-only partials; a
FeedChunkbarrier table with an atomic idempotent counter letsexactly one chunk trigger
FinalizeFeedContext, which concatenates the ordered partials.simulated mid-run chunk failure — the finalize claim (atomic barrier-row delete) fires once, a
stray finalize is a no-op. A shared
FeedContextFinalizer(record → gate → counter → complete) isused by both paths. Fan-out engages only for single-source, non-split, non-gzip feeds (threshold
formatConfig['chunk']['size'], default 10k); split/gzip stay inline.Per-context multi-target delivery
DeliveryTarget(transport / transportConfig / pathTemplate / match); pluggable transports —localalways,ftp/sftp/s3gated onclass_exists(Flysystem adapters are the integrator'soptional install, listed under composer
suggest).DeliveryServicepushes the whole file set (parts + manifest) to every matching target,best-effort + isolated (never throws), recording
{target,path,status,error?}onFeedContextResult.deliveries. Hooked into finalize: a context is delivered only once promoted tocanonical and published — a gate-blocked context is never delivered.
More FeedTypes (resource-agnosticism)
order,customer,taxon,product_review,promotionfeed types + data sources + resolvers.An Order CSV export and a Taxon feed generate via the real engine (functional tests). 7 feed
types now show in the admin source picker.
Cron
setono:feed:process --all(explicit form) + a README Scheduling section with a crontab example;fixed the README's stale command name.
CI note
Verified against the updated PHPStan/Rector/ECS and PHP-8.1-pinned PHPStan + league/csv 9.6, so
the gate (Unit / Integration / Static Code Analysis / Coding Standards) should be green. BC,
Dependency Analysis and Mutation remain the standing rewrite reds.
This is the last milestone — it completes the M0–M7 resource-agnostic rewrite.