M5 — Channel breadth + CSV + multi-source + LookupTable enrichment - #115
Merged
Conversation
- CsvWriter (league/csv, 'csv' writer family): streams a header row (union of the feed's output fields) then one row per item, cells keyed by header column so heterogeneous multi-source rows line up; missing field -> empty cell, list -> joined. Optional UTF-8 BOM written directly (league/csv only BOMs its own output methods, not an external stream). Cross-version league/csv 9.0-9.28 API. - CsvWriterConfig (delimiter/enclosure/header/bom) + withHeader(). - csv FormatInterface preset (XmlWriter unchanged). - FeedGenerator computes the union header from all sources' mappings and injects it for CsvWriterConfig before opening the writer. +6 tests (442 total); PHPStan max + ECS + Rector + container lint green.
- generic_xml + partner_ads XmlWriter format presets (no new writer class): partner_ads roots produkter/produkt. - Mapping presets targeting product_variant: Meta (csv, space-separated availability via value_map), Bing + Pinterest (google_rss, = Google), TikTok (csv, sku_id/product_page_url), Partner-ads (partner_ads, Danish element names nypris/glpris/VareURL/BilledURL/koen). Registered + translated; they appear in the admin target picker. - Acceptance: a valid Meta CSV and a valid Partner-ads XML are produced from the same catalog purely by choosing the preset -- no code changes. +27 tests (469 total); PHPStan max + ECS + container/yaml lint green.
- ProductAwareTrait: shared resolvers (id, title, description, link, main_image,
additional_images, is_configurable) now support both ProductVariant and Product
entities, deriving the product from either (variant -> getProduct()); variant
behaviour is unchanged. id resolves to the variant/product code respectively.
- New product-level resolvers: variant_count (both entities), from_price
(cheapest enabled variant price), product_availability (in_stock if any variant
is). Variant-only resolvers (item_group_id, channel_price, availability, ...)
untouched.
- ProductDataSource (iterate enabled, channel-assigned Product) + ProductFeedType
('product', scope channel x locale x currency); wired + translated.
- Multi-source proven end to end: one CSV feed with a product source and a
product_variant source emits both product-level and variant-level rows under a
union header.
+24 tests (493 total); PHPStan max + ECS + Rector + container lint green.
- LookupTable resource entity/interface + Doctrine mapping (setono_sylius_feed__lookup_table; the 'rows' property maps to a lookup_rows json column since rows is a MySQL reserved word) + repository; registered as a Sylius resource in Configuration. - DatabaseLookup implements LookupInterface against the LookupTable repository (each table loaded once and cached per instance); LookupInterface now aliases to DatabaseLookup in production. InMemoryLookup remains for tests/programmatic use. - ScriptingTest now persists a LookupTable and proves expression + sandboxed twig lookup() resolves through the database. 493 tests green; PHPStan max + ECS + Rector + container lint green. (sources/refresh, lookup: source-ref, and the admin UI follow.)
- CsvLookupSource ('csv'): parses an uploaded CSV from the feed filesystem by
header. UrlLookupSource ('url'): downloads CSV/TSV from a URL (covers a
published Google Sheet export). LookupRow normalizes records to column=>scalar.
- LookupTableRefresher: imports rows via the source registered for the table's
sourceType, keyed by keyColumn; stamps refreshedAt on success. On any failure
it keeps the last-good rows + refreshedAt and logs a warning -- a broken source
never blanks the serving data.
- Wired: both sources tagged; refresher + interface alias.
+7 tests (500 total); PHPStan max + ECS + Rector + container lint green.
- LookupReferenceResolver: parses a lookup:{code}:{column} reference, reads the
table's joinField value off the item (via the source resolvers), and returns
the requested column. Wired into FieldMappingEvaluator's source resolver, so a
mapping like concat(['value','lookup:badges:badge']) enriches each row per item
(the exact 10.1 case) and GTIN backfill (g:gtin <- lookup:...) works.
+1 test (501 total); PHPStan max + ECS + Rector + container lint green.
- Resource CRUD grid (code/sourceType/refreshedAt) + routes; LookupTableType form (code/name/sourceType/location/keyColumn/joinField/refreshPolicy) that packs the single location into sourceConfig as path (csv) or url (url) and unpacks it on edit; RefreshLookupTableAction (re-import + flash); a 'Lookup tables' admin menu item; translations (incl. the flashes-domain refresh flash + page titles). - Configuration uses LookupTableType as the resource form. Browser-verified: grid loads, create persists, Refresh fetches a CSV URL and imports the keyed rows (refreshedAt stamped), keep-last-good on failure. +2 tests (503 total); PHPStan max + ECS + Rector + container/yaml lint green.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #115 +/- ##
============================================
- Coverage 98.48% 95.32% -3.17%
- Complexity 822 998 +176
============================================
Files 108 134 +26
Lines 1918 2417 +499
============================================
+ Hits 1889 2304 +415
- Misses 29 113 +84 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- league/csv 9.0-9.5 ignore setDelimiter() after createFromStream() (insertOne read the delimiter from the document, unsynced) -> the custom-delimiter CSV test failed on Deps:lowest. Raise the constraint to ^9.6 (verified: 9.6 honours the delimiter and supports PHP 8.1). - league/csv <9.28 types getRecords() loosely (mixed), tripping PHPStan on lowest; LookupRow::normalize now accepts mixed + guards is_iterable/is_scalar, and the generator tests read records through a version-robust array_filter(is_array(...)) helper. Verified clean on both 9.6 and 9.28. 503 tests; PHPStan max + ECS + Rector + validate/normalize green.
On PHP 8.1/8.2 PHPStan types iterator_to_array's param as Traversable, but league/csv getRecords() and the lookup sources' fetch() are declared iterable, so Static Code Analysis failed on those legs. Spread ([...iterable]) accepts any Traversable and avoids the stub mismatch. Verified clean under PHPStan phpVersion 8.1 and league/csv 9.6 + 9.28.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M5 — Channel breadth + CSV + multi-source + external enrichment
Broadens the engine to more destinations and adds the LookupTable enrichment resource. Four
chunks (7 commits); 503 tests; the admin LookupTable flow is browser-verified.
CSV writer + format presets
CsvWriter(league/csv, thecsvwriter family): a header row that is the union of thefeed's output fields (computed by the generator, injected into the config) then one row per
item keyed by header — so heterogeneous multi-source rows line up; missing → empty cell, list →
joined, optional UTF-8 BOM.
csvformat preset.generic_xml+partner_adsXmlWriter format presets — new destinations are formatpresets, no new writer class.
Channel mapping presets
Meta(csv, space-separated availability viavalue_map),Bing+Pinterest(google_rss),TikTok(csv,sku_id/product_page_url),Partner-ads(partner_ads, Danish element names). Theyshow up in the admin target picker. Acceptance: a valid Meta CSV and a valid Partner-ads XML are
produced from the same catalog purely by choosing the preset — no code changes.
productFeedType + multi-sourceid,title,description,link,main_image,additional_images,is_configurable) now support bothProductVariantandProduct(ProductAwareTrait);variant behaviour is unchanged. New product-level resolvers:
variant_count,from_price,product_availability.ProductDataSource+ProductFeedType. Multi-source proven end to end: one CSV feed with aproductsource and aproduct_variantsource emits both product-level and variant-level rowsunder a union header.
External enrichment — the LookupTable resource (§10.1)
LookupTableSylius resource (entity/mapping/repository/admin grid + CRUD + Refreshaction + menu). The
rowsproperty maps to alookup_rowsjson column (rowsis a MySQLreserved word).
LookupSourceInterfacewith built-incsv(uploaded file) andurl(downloads CSV/TSV — covers a published Google Sheet
…/export?format=csv).LookupTableRefresherimports rows keyed bykeyColumn, stampingrefreshedAt; on anyfailure it keeps the last-good rows and logs a warning — a broken source never blanks the
serving data.
DatabaseLookup(the productionLookupInterface) resolveslookup(table, key, column)inexpressions/twig against the tables (loaded once + cached per run).
lookup:{code}:{column}source references resolve in the generator via the table'sjoinField— soconcat(["value", "lookup:badges:badge"])(the exact §10.1 case) and GTINbackfill (
g:gtin ← lookup:…) work end to end.rows (
refreshedAtstamped) → the flash shows.CI note
Verified locally against the updated PHPStan/Rector/ECS (CI runs
composer update), so thesubstantive gate (Unit / Integration / Static Code Analysis / Coding Standards) should be green.
The Backwards Compatibility Check carries intentional breaks (the
FieldMappingEvaluatorconstructor gained a lookup dependency;
Configurationgained the resource) — left visible per therewrite policy — and Dependency Analysis + Mutation remain the standing rewrite reds.