Skip to content

Commit 0ae53b0

Browse files
committed
Docs: bring CLAUDE.md + README.md up to date with the 1.x engine
CLAUDE.md: rewrite the Architecture section for the new async flow (ProcessFeed -> GenerateFeedContext -> inline or fan-out via GenerateFeedChunk/FinalizeFeedContext -> FeedContextFinalizer -> publish gate -> per-context gated promotion + delivery), the FeedGraph states (ready/processing/completed/failed), the tagged extension-point registries, field-mapping-and-writer output model, preview/audit diagnostics, delivery/split/gzip, and LookupTable enrichment. Document the dependency-analysis require-only gotcha (split Sylius packages + the flysystem constraint split) and that only English translations ship so far. README.md: describe the resource-agnostic engine + its capabilities, fix the routing import to the real files (routes.yaml + routes/admin.yaml), and align the usage/cron section with setono:feed:process.
1 parent 36a54d8 commit 0ae53b0

2 files changed

Lines changed: 113 additions & 54 deletions

File tree

CLAUDE.md

Lines changed: 95 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,7 @@ These are enforced by `.github/workflows/build.yaml` and will fail the build if
161161
- **PHP 8.1 is the floor**: the package supports PHP `>=8.1`, and CI runs against 8.1/8.2/8.3 with Symfony `~6.4`. Coding-standards run on **8.1** and Rector targets `LevelSetList::UP_TO_PHP_81`, so do **not** use syntax/features newer than 8.1.
162162
- **`lowest` and `highest` dependencies** are both tested — avoid relying on behavior only present in newer versions of a `^`-constrained dependency.
163163
- **`composer normalize --dry-run`** must pass — keep `composer.json` normalized (run `composer normalize`).
164-
- **Dependency analysis** (`shipmonk/composer-dependency-analyser`, config in `composer-dependency-analyser.php`) checks that every used package is a direct dependency.
164+
- **Dependency analysis** (`shipmonk/composer-dependency-analyser`, config in `composer-dependency-analyser.php`) must pass: every symbol used in `src/` maps to a declared `require`, and every `require` is used. **Gotcha:** the job runs `composer config --unset require-dev` and resolves **`require`-only**, so it never sees the `sylius/sylius` monorepo (a require-dev dependency) — it pulls the *split* component packages (`sylius/core`, `sylius/order`, …) instead. Two consequences: (a) `require` must directly declare every Sylius component the code uses (the split packages don't `replace` each other); (b) the split `sylius/core` caps `league/flysystem` at `^2.4`, so `require` keeps `league/flysystem: ^2.4 || ^3.0` and the 3.x floor + `league/flysystem-local` live in **require-dev** (the test app and the full install still resolve flysystem 3.15 + flysystem-local 3.15, matched so `ChecksumProvider` is present). Reproduce the exact job locally with: `cp composer.json /tmp/bak && composer config --unset require-dev && composer require --dev --no-install shipmonk/composer-dependency-analyser && composer update --prefer-lowest --ignore-platform-req=php+ && vendor/bin/composer-dependency-analyser` (then `cp /tmp/bak composer.json && composer update`).
165165

166166
### Test Application
167167
The plugin includes a test Symfony application in `tests/Application/` for development and testing:
@@ -194,74 +194,119 @@ Examples:
194194

195195
## Architecture
196196

197-
### Feed Processing Flow
198-
199-
The pipeline is a fan-out of async messages, and the *lifecycle* is driven by Symfony Workflow transition events — not by the handlers calling each other directly.
200-
201-
1. `ProcessFeedsCommand` (`setono:sylius-feed:process`) calls `FeedProcessor::process()`, which dispatches one `ProcessFeed` per enabled feed.
202-
2. `ProcessFeedHandler` validates the feed type's template, applies the `process` transition, and dispatches one `GenerateFeed` per channel/locale combination.
203-
3. `GenerateFeedHandler` asks the feed type's `DataProvider` for batches (`getBatches()`) and dispatches one `GenerateBatch` per batch.
204-
4. `GenerateBatchHandler` resolves the batch's items, runs each through the item context, validates every context, renders the Twig `item` block, writes a **partial file** per channel/locale, then dispatches `BatchGeneratedEvent`.
205-
5. `FinishGenerationHandler` concatenates the partials into the final feed (wrapping them with the feed start/end rendered from `@SetonoSyliusFeedPlugin/Feed/feed.txt.twig`, split on the `<!-- ITEM_BOUNDARY -->` marker), deletes the partials, and applies the `processed` transition.
206-
207-
**Completion detection is counter-based, not "last handler wins".** The total batch count is set on the feed when the `process` transition fires (`StartProcessingSubscriber``Feed::setBatches()`). On each `BatchGeneratedEvent`, `IncrementFinishedBatchesSubscriber` (priority 100) increments the counter, then `SendFinishGenerationCommandSubscriber` dispatches `FinishGeneration` only once `FeedRepository::batchesGenerated()` is true. This is what makes the flow safe under out-of-order async batch processing.
197+
The plugin is a **resource-agnostic transformation engine**: *iterate any Sylius resource → map each
198+
entity to named output fields → transform/filter → validate → stream to a format → gate → publish/
199+
deliver*. "Google Shopping product feed" is just the richest `FeedType` plugged into that engine. The
200+
authoritative design lives in `.notes/sylius-feed-plugin-spec.md` (gitignored).
208201

209-
**Workflow-transition subscribers** (`workflow.setono_sylius_feed.feed.transition.*`) handle side effects so handlers stay focused:
210-
- `process``StartProcessingSubscriber` resets and sets the batch count.
211-
- `processed``MoveGeneratedFeedSubscriber` moves the feed from the temporary filesystem to its final (public) location.
212-
- `errored``DeleteGeneratedFilesSubscriber` cleans up generated files.
213-
214-
**Validation/violation behavior:** each context is validated with the feed type's validation groups. A violation with severity `error` causes that item to be **skipped** (not written to the feed); other severities are recorded as `Violation`s on the feed but the item is still written. Any thrown error transitions the feed to `error`.
215-
216-
### Key Components
202+
### Feed Processing Flow
217203

218-
- **FeedType** (`FeedTypeInterface`): Defines a feed format. Contains data provider, templates, feed context, and item context. Register with tag `setono_sylius_feed.feed_type`.
219-
- **DataProvider** (`DataProviderInterface`): Provides items to be included in the feed (e.g., products)
220-
- **FeedContext/ItemContext**: Transform raw data into context for Twig templates
221-
- **Workflow** (`FeedGraph`): States: unprocessed, processing, ready, error. Transitions: process, processed, errored
204+
Generation is an async Messenger fan-out; the *lifecycle* is driven by Symfony Workflow transitions,
205+
not by handlers calling each other.
206+
207+
1. `ProcessFeedCommand` (`setono:feed:process [--feed=CODE] [--all]`) dispatches one `ProcessFeed` per
208+
enabled feed.
209+
2. `ProcessFeedHandler` applies the `process` transition and, via `ContextFactory`, fans out one
210+
`GenerateFeedContext` per **context** — the cartesian product of the feed's scope dimensions
211+
(channel × locale × currency) — recording the expected context count on the feed.
212+
3. `GenerateFeedContextHandler` generates one context. Small feeds run **inline**
213+
(`FeedGenerator::generate()`); a large single-source, non-split, non-gzip feed **fans out**:
214+
`ChunkPartitioner` splits the source into id-ranges, one `GenerateFeedChunk` per range renders a
215+
**body-only partial**, a per-context `FeedChunk` barrier table tracks completion with an atomic
216+
idempotent counter, and the last chunk dispatches `FinalizeFeedContext`, which concatenates the
217+
ordered partials into the context file (**byte-identical** to the inline path, and resumable after
218+
a mid-run chunk failure).
219+
4. Either path finishes through the shared `FeedContextFinalizer`: record a `FeedContextResult`
220+
(item/excluded counts, bytes, per-item exclusion reasons), run the **publish gate**, increment the
221+
completed-context counter, and on the last context apply the `complete` transition.
222+
5. On `complete`, `MoveGeneratedFeedSubscriber` does the **per-context gated promotion** staging→
223+
canonical (only `published` contexts move; `blocked` ones keep their prior live file), then runs
224+
**delivery** per published context to any matching `DeliveryTarget`s.
225+
226+
**Per-item pipeline** (`FeedGenerator` and `PreviewService` share it): per source entity — bind the
227+
on-demand source resolver (`SourceResolverBinder`) → `pre`-filters (`FilterEvaluator`) → apply field
228+
mappings (`FieldMappingEvaluator`, resolved by `MappingResolver`: `FeedField` rows first, else the
229+
`MappingPreset`) → `post`-filters → `FeedItemBuiltEvent` → validation (`FeedItemValidator`: typed
230+
Symfony constraints when the format declares validation groups, else the required-field fallback) →
231+
write. Excluded items are counted and reported with a reason; they never abort the run.
232+
233+
**Publish gate (spec §6.6):** generating and *publishing* are separate acts. Per context, before the
234+
canonical swap and delivery, the candidate `FeedContextResult` is compared against the last-good
235+
baseline via the feed's `publishConfig` guardrails (`min_items`, `max_drop_pct`, `non_empty`,
236+
`min_bytes`, `max_exclusion_pct`, `max_growth_pct`). A tripped `block` guardrail sets
237+
`publishState = blocked`, keeps the live file, and dispatches `FeedPublishBlockedEvent`; a `warn`
238+
publishes and still dispatches. A "publish anyway" admin action promotes a retained blocked candidate.
239+
240+
### Workflow (`FeedGraph`, graph `setono_sylius_feed_feed`)
241+
242+
States `ready → processing → completed | failed`. Transitions: `process` (ready→processing),
243+
`complete` (processing→completed), `fail` (processing→failed), `reset` (completed|failed→ready).
244+
Marking store: `Feed::state`.
245+
246+
### Extension points (tagged registries — no autowiring)
247+
248+
Each is an interface collected into an FQCN registry via a tag (the extension calls
249+
`registerForAutoconfiguration` only as a DX aid for apps that add their own):
250+
251+
- `FeedTypeInterface``setono_sylius_feed.feed_type` (product_variant, product, order, customer, taxon, product_review, promotion)
252+
- `ValueResolverInterface``setono_sylius_feed.value_resolver`
253+
- `TransformationInterface``setono_sylius_feed.transformation`
254+
- `OperatorInterface``setono_sylius_feed.operator` (shared by filters, `FeedField.condition`, and the `conditional` transform)
255+
- `MappingPresetInterface``setono_sylius_feed.mapping_preset` (Google Shopping, Meta, Bing, Pinterest, TikTok, Partner-ads)
256+
- `FormatInterface``setono_sylius_feed.format` (google_rss, csv, generic_xml, partner_ads)
257+
- `FeedWriterInterface``setono_sylius_feed.writer` (XmlWriter, CsvWriter)
258+
- `SplitManifestInterface``setono_sylius_feed.split_manifest` (none, supplemental)
259+
- `LookupSourceInterface``setono_sylius_feed.lookup_source` (csv, url) — powers `LookupTable` enrichment referenced as `lookup:{code}:{column}`
260+
- `DeliveryTransportInterface``setono_sylius_feed.delivery_transport` (local always available; ftp/sftp/s3 gated on `class_exists`, their Flysystem adapters are optional `suggest` installs)
261+
- `GuardrailInterface``setono_sylius_feed.guardrail`
262+
263+
### Output: field mappings + writers (not Twig templates)
264+
265+
A feed's output is defined by its sources' `FeedField` rows (output field, source picker,
266+
transformation sub-collection, condition), evaluated into a `FeedItem` bag and streamed by the
267+
format's `FeedWriterInterface`. Sandboxed Twig and expression scripting exist as *transformations*
268+
(`FeedTemplateSecurityPolicy`, `ScriptingVariables`), not as a per-feed render template.
222269

223270
### Message Commands
224271

225-
All commands implement `CommandInterface` and can be routed to async transport:
226-
- `ProcessFeed` - Start processing a feed
227-
- `GenerateFeed` - Generate feed for a specific channel/locale
228-
- `GenerateBatch` - Process a batch of items
229-
- `FinishGeneration` - Finalize feed after all batches complete
272+
All implement `CommandInterface` and can be routed to an async transport: `ProcessFeed`,
273+
`GenerateFeedContext`, `GenerateFeedChunk`, `FinalizeFeedContext`.
230274

231-
### Feed Templates
275+
### Diagnostics
232276

233-
Templates in `src/Resources/views/Feed/` must define an `item` block. Example structure:
234-
```twig
235-
{% block item %}
236-
{# Render single feed item #}
237-
{% endblock %}
238-
```
277+
- **Preview** (`PreviewService`; admin `/feeds/{id}/preview` + `--preview[=N]`): runs the full pipeline
278+
over the first N sampled items **without writing** — a funnel (source → after-pre → after-mapping/
279+
validation → after-post → included), included/excluded samples annotated with reasons, and a
280+
single-item tester.
281+
- **Audit** (`FeedAuditService`; `--audit`): per-field fill rates, soft warnings (title > 150,
282+
HTML-in-description, price = 0) and value distributions — advisory only, excludes nothing.
239283

240-
### Extension Points
284+
### Delivery, splitting, gzip
241285

242-
- Implement `FeedTypeInterface` for custom feed formats
243-
- Use event listeners on `QueryBuilderEvent` to filter data
244-
- Filter listeners in `EventListener/Filter/` (channel, enabled, in-stock filters)
245-
- Subscribe to `GenerateBatchItemEvent` and `GenerateBatchViolationEvent` for item processing hooks
286+
Canonical Flysystem storage is always written and served at the public route. On top, a feed's
287+
`DeliveryTarget`s push the whole file set (parts + manifest) to the contexts they match
288+
(`{channel?,locale?,currency?}` matcher + a `pathTemplate`), best-effort and isolated. A context past
289+
its format's split limit is split into numbered parts plus a `SplitManifest` (`supplemental` for
290+
google_rss); `formatConfig.gzip` gzips each part.
246291

247-
### Model Interfaces
292+
### Enrichment (`LookupTable`)
248293

249-
Product models can implement optional interfaces for feed data:
250-
- `BrandAwareInterface`, `GtinAwareInterface`, `MpnAwareInterface`
251-
- `ColorAwareInterface`, `SizeAwareInterface`, `ConditionAwareInterface`
252-
- Localized variants: `LocalizedBrandAwareInterface`, etc.
294+
A separate Sylius resource (admin `/admin/lookup-tables`) importing rows from a `csv`/`url`
295+
`LookupSource` with keep-last-good refresh; feed mappings reference it by code as
296+
`lookup:{code}:{column}` (e.g. GTIN backfill).
253297

254298
### Translations
255299

256-
The plugin provides multilingual support through translation files in `src/Resources/translations/`:
300+
Translation files live in `src/Resources/translations/`. The rewrite ships **English only** so far
301+
(`messages.en.yaml`, `flashes.en.yaml`, `validators.en.yaml`) — other locales are welcome additions.
257302

258-
- **Translation Files**: Available in 10 languages (en, da, de, es, fr, it, nl, no, pl, sv)
259303
- **Translation Domains**:
260304
- `messages.*` - UI labels and general translations
261305
- `flashes.*` - Flash message translations (success/error messages)
262306
- `validators.*` - Validation error messages
263307

264308
Key translation keys:
265-
- `setono_sylius_feed.ui.*` - UI labels (feeds, violations, states)
266-
- `setono_sylius_feed.form.*` - Form field labels
309+
- `setono_sylius_feed.ui.*` - UI labels (feeds, preview/funnel/audit, states)
310+
- `setono_sylius_feed.form.*` - Form field labels (feed, feed_source, feed_field, feed_filter, delivery_target, lookup_table)
267311
- `setono_sylius_feed.feed_type.*` - Feed type names
312+
- `setono_sylius_feed.mapping_preset.*` / `setono_sylius_feed.value_resolver.*` - preset + resolver labels

README.md

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,18 @@
44
[![Software License][ico-license]](LICENSE)
55
[![Build Status][ico-github-actions]][link-github-actions]
66

7-
A plugin for creating all kinds of feeds to any given service. Do you want to create product feeds for
8-
your Google Merchant center? Then this is the right plugin for you.
7+
A **resource-agnostic feed-generation engine** for Sylius. It maps any Sylius resource — product
8+
variants, products, orders, customers, taxons, product reviews, promotions — to named output fields,
9+
transforms and filters them, validates against a channel spec, and streams the result to a format
10+
(Google RSS, CSV, generic XML, Partner-ads). Google Shopping is the richest built-in target, but the
11+
same engine ships presets for Meta, Bing, Pinterest, TikTok and Partner-ads, plus fully custom feeds.
12+
13+
Highlights: a target-first admin create flow with a field-mapping editor; a shared transformation +
14+
filter vocabulary; `LookupTable` enrichment (e.g. GTIN backfill from a CSV/URL); channel × locale ×
15+
currency fan-out; a dry-run **preview** (funnel + included/excluded samples) and a **feed audit**;
16+
a **publish gate** that refuses to replace a good feed with a broken one; gzip, file splitting with a
17+
supplemental manifest, multi-chunk fan-out for large catalogs; and per-context **delivery** to
18+
FTP/SFTP/S3 targets — all on top of a canonical, always-served public feed URL.
919

1020
## Installation
1121

@@ -48,10 +58,14 @@ return [
4858
```yaml
4959
# config/routes/setono_sylius_feed.yaml
5060
setono_sylius_feed:
51-
resource: "@SetonoSyliusFeedPlugin/Resources/config/routing.yaml"
61+
resource: "@SetonoSyliusFeedPlugin/Resources/config/routes.yaml"
62+
63+
setono_sylius_feed_admin:
64+
resource: "@SetonoSyliusFeedPlugin/Resources/config/routes/admin.yaml"
5265
```
5366
54-
If you don't use localized URLs, use this routing file instead: `@SetonoSyliusFeedPlugin/Resources/config/routing_non_localized.yaml`
67+
`routes.yaml` exposes the public feed at `/feed/{code}/{filename}`; `routes/admin.yaml` adds the admin
68+
screens (feed CRUD, generate, preview, results, publish) under `/admin`.
5569

5670
### Step 4: Configure plugin
5771

0 commit comments

Comments
 (0)