-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathabout.html
More file actions
429 lines (318 loc) · 33.3 KB
/
Copy pathabout.html
File metadata and controls
429 lines (318 loc) · 33.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>About — UK fuel price tracker</title>
<link rel="icon" type="image/svg+xml" href="/static/g-roundel.svg">
<link rel="shortcut icon" href="/static/g-roundel.svg">
<link rel="stylesheet" href="/static/docs.css">
<style>
main { max-width: 800px; margin: 0 auto; padding: 2rem; }
</style>
</head>
<body>
<header>
<h1>
<a href="/#dashboard" class="brand-link">
<img src="/static/g-roundel.svg" alt="" class="brand-icon" width="24" height="24">
<span>UK fuel price tracker</span>
</a>
</h1>
<nav>
<a href="/#dashboard">Dashboard</a>
<a href="/docs/api">API</a>
<a href="/docs/about">About</a>
</nav>
</header>
<main>
<p>A historical UK fuel price tracker built on the GOV.UK Fuel Finder API. Prices are checked every 30 minutes and updates are stored to enable analysis the live API can't provide.</p>
<ul style="margin: 1rem 0 1.5rem 1.5rem; line-height: 1.8;">
<li><strong>Live dashboard</strong> — current averaged prices by region, brand and forecourt category</li>
<li><strong>Price history</strong> — trends over time for individual stations or filtered groups</li>
<li><strong>Interactive map</strong> — colour-coded price distribution across the UK</li>
<li><strong>Station search</strong> — filter by brand, location, postcode area, rural/urban classification and more</li>
<li><strong>Anomaly & outlier detection</strong> — suspicious or statistically extreme prices flagged and excluded from averages</li>
<li><strong>Price correction tools</strong> — editors can review and correct misreported values <em style="color:var(--muted,#666)">(editor/admin only)</em></li>
<li><strong>Data correction tools</strong> — brand normalisation, postcode overrides and coordinate fixes for source data issues <em style="color:var(--muted,#666)">(editor/admin only)</em></li>
<li><strong>REST API</strong> — all data available programmatically; see the <a href="/docs/api">API reference</a></li>
</ul>
<h2>How it works</h2>
<p>This tool gathers data from the government's <a href="https://www.find-fuel-prices.service.gov.uk/">consumer fuel price finder</a> — and stores it in a PostgreSQL database to build a <strong>historical price record</strong> that the API itself doesn't provide.</p>
<p>The GOV.UK API only serves <strong>live snapshots</strong>: current prices at the time of the request. There is no way to retrieve yesterday's prices or see how prices have changed over time. By scraping regularly and storing every price change, we build a time-series dataset that enables trend analysis, regional comparisons, and anomaly detection.</p>
<h2 id="data-source">Data source</h2>
<p>The Fuel Finder API is a government service that aggregates fuel prices reported by petrol station operators. It covers approximately <strong>7,500 stations</strong> across England, Wales, Scotland and Northern Ireland, reporting prices for up to six fuel types.</p>
<table>
<thead><tr><th>Code</th><th>Fuel type</th><th>Category</th></tr></thead>
<tbody>
<tr><td><code>E10</code></td><td>Unleaded (standard, up to 10% ethanol)</td><td>Petrol</td></tr>
<tr><td><code>E5</code></td><td>Super Unleaded (up to 5% ethanol)</td><td>Petrol</td></tr>
<tr><td><code>B7_STANDARD</code></td><td>Diesel (standard, up to 7% biodiesel)</td><td>Diesel</td></tr>
<tr><td><code>B7_PREMIUM</code></td><td>Premium Diesel</td><td>Diesel</td></tr>
<tr><td><code>B10</code></td><td>Diesel (up to 10% biodiesel)</td><td>Diesel</td></tr>
<tr><td><code>HVO</code></td><td>Hydrotreated Vegetable Oil (renewable)</td><td>Diesel</td></tr>
</tbody>
</table>
<h2 id="the-scraper">The scraper</h2>
<p>We use an API key granted by gov.uk to check <a href="https://www.developer.fuel-finder.service.gov.uk/public-api">developer.fuel-finder.service.gov.uk/public-api</a> for updates to prices at UK fuel stations.</p>
<div class="diagram">
<pre>
Fuel Finder API (GOV.UK)
│
│ OAuth2 client credentials → bearer token (1h TTL)
│ GET /api/v1/pfs/fuel-prices?batch-number=N (500 stations/batch, ~15 batches)
│ GET /api/v1/pfs?batch-number=N (station details)
▼
┌───────────────────────────────┐
│ Scraper │ Python — api_client.py + scrape.py
│ │
│ 1. Authenticate │ OAuth2 client credentials
│ 2. Fetch batches │ Paginate through all stations + prices
│ 3. Upsert stations │ Insert/update station records
│ 4. Insert prices │ Append-only, deduplicated
│ 5. Detect anomalies │ Flag suspicious prices (not filter)
│ 6. Refresh view │ Rebuild current_prices snapshot
│ 7. Enrich postcodes │ Lookup new postcodes via postcodes.io
│ 8. Backup to S3 │ Raw JSON (optional)
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ PostgreSQL │
│ │
│ stations │ ~7,500 fuel stations with lat/lng
│ fuel_prices │ append-only price change events
│ brand_aliases │ raw → canonical brand mapping
│ brand_categories │ brand → forecourt type
│ station_brand_overrides │ per-station brand corrections
│ station_postcode_overrides │ per-station postcode corrections
│ fuel_type_labels │ fuel code → human name
│ postcode_regions │ postcode → ONS region
│ postcode_lookups │ postcodes.io enrichment cache
│ scrape_runs │ scrape execution history
│ current_prices │ materialised view (live snapshot)
└───────────────────────────────┘
</pre>
</div>
<h3>API usage guidelines</h3>
<p>The scraper is designed to honour the <a href="https://www.developer.fuel-finder.service.gov.uk/dev-guideline">GOV.UK Fuel Finder developer guidelines</a>. The following safeguards are built in:</p>
<ul>
<li><strong>Rate limiting</strong> — A client-side sliding window enforces the 30 requests-per-minute limit. If a scrape requires more than 30 API calls, the scraper waits until capacity is available rather than exceeding the quota.</li>
<li><strong>Retry with backoff</strong> — If the API returns a <code>429 Too Many Requests</code> or a transient server error (500/502/503/504), the scraper retries with exponential backoff, respecting the <code>Retry-After</code> header when present.</li>
<li><strong>Compression</strong> — All requests include <code>Accept-Encoding: gzip, deflate</code> to reduce bandwidth, per the performance guidelines.</li>
<li><strong>No data redistribution</strong> — Raw API responses backed up to S3 are stored in a private bucket with all public access blocked. This data is retained solely for internal debugging, reprocessing, and data-integrity checks, and is never redistributed.</li>
<li><strong>Incremental fetching</strong> — Rather than re-fetching all prices every run, the scraper uses the <code>effective-start-timestamp</code> parameter to request only prices that have changed, minimising unnecessary API load.</li>
<li><strong>Sequential requests</strong> — Only one API request is in-flight at a time, respecting the one-concurrent-request-per-client limit.</li>
</ul>
<h3>Scrape modes</h3>
<div class="card">
<h4>Full scrape</h4>
<p>Fetches all stations and all prices from scratch (~15 batches of 500 stations each). Used for the initial load and daily refreshes. Updates station metadata (addresses, amenities, opening times) and inserts any new prices.</p>
</div>
<div class="card">
<h4>Incremental scrape</h4>
<p>Uses the <code>effective-start-timestamp</code> API parameter to fetch only prices that have changed since the last successful scrape. Much faster — typically returns a few hundred changes instead of ~24,000 prices. Ideal for frequent polling (every 30 minutes).</p>
</div>
<div class="card">
<h4>Auto mode</h4>
<p>Checks the database for the most recent successful scrape. If one exists, runs incremental; otherwise runs a full scrape. This is the default — set it up on a schedule and forget about it.</p>
</div>
<h2 id="how-prices-are-stored">How prices are stored</h2>
<p>The <code>fuel_prices</code> table is <strong>append-only</strong>, but with deduplication. When a scrape runs:</p>
<ol>
<li>For each (station, fuel type) pair, the scraper looks up the most recently stored price</li>
<li>If the new price is <strong>the same</strong> as the stored price, it's skipped</li>
<li>If the price has <strong>changed</strong>, a new row is inserted with the current timestamp</li>
<li>The new price is checked against anomaly rules and flagged if suspicious</li>
</ol>
<p>This means every row in <code>fuel_prices</code> represents a <strong>genuine price change event</strong>, keeping storage lean and making time-series analysis straightforward.</p>
<h3>Anomaly detection</h3>
<p>On insert, each price is checked against three rules. Suspicious prices are <strong>flagged as 'anomalous', but not ignored</strong>. On-site visualisations for average prices and trends over time exclude anomalous values to to avoid being skewed by obvious errors in the source data. However, the original reported values are preserved and are included in station price history tables and exports.</p>
<table>
<thead><tr><th>Flag</th><th>Anomaly rule</th></tr></thead>
<tbody>
<tr><td><code>price_below_floor</code></td><td>Price below 80p/litre</td></tr>
<tr><td><code>price_above_ceiling</code></td><td>Price above 300p/litre</td></tr>
<tr><td><code>likely_decimal_error</code></td><td>Price looks like pounds instead of pence (e.g. 1.45 instead of 145.0)</td></tr>
<tr><td><code>large_price_jump</code></td><td>Price changed by more than 30% from previous</td></tr>
</tbody>
</table>
<p>In practice the <code>large_price_jump</code> test can mean legitimate price reports are flagged when the previous price report was incorrect. If the price passes the first three tests, our tool should not automatically pass judgment on whether the old or the new value is correct. Such problems go away naturally from the dashboard when a station reports its prices accurately twice in a row.</p>
<h3 id="outlier-methodology">Statistical outlier exclusion</h3>
<p>Anomaly flags catch the most obvious errors, but the GOV.UK source data is manually entered by fuel station operators and inevitably contains subtler mistakes — for example, stations often misattribute B7 Standard Diesel and B7 Premium Diesel prices, or a stale or test value slips through. Even a few of these can skew averages noticeably.</p>
<h4>Dashboard & current snapshot (Tukey IQR)</h4>
<p>All averages shown on the dashboard cards, and the current-price breakdowns by region, brand, category, etc., exclude statistical outliers using the <strong>Tukey IQR (interquartile range) fence</strong> method — the same technique commonly used in box-plot charts:</p>
<ol>
<li>For each fuel type, compute <strong>Q1</strong> (25th percentile) and <strong>Q3</strong> (75th percentile) of all current non-anomalous prices</li>
<li>Calculate <strong>IQR = Q3 − Q1</strong> — the spread of the middle 50% of prices</li>
<li>Set fences at <strong>Q1 − 1.5 × IQR</strong> (lower) and <strong>Q3 + 1.5 × IQR</strong> (upper)</li>
<li>Any price outside these fences is marked as an outlier</li>
</ol>
<p>The 1.5 × IQR multiplier is the standard Tukey threshold (introduced in John Tukey's <em>Exploratory Data Analysis</em>, 1977). It identifies values that are far enough from the central distribution to be suspect, without being so aggressive that it trims legitimate price variation.</p>
<p>The fences are recomputed each time the materialised view is refreshed (after every scrape), so they adapt automatically as prices change over time. Since IQR is applied to a single snapshot (the latest price per station), trending prices are not an issue.</p>
<h4>Historical trend charts (Hampel filter)</h4>
<p>For trend charts that show price history over weeks or months, a static IQR fence would be inappropriate — if prices trend upward steadily, earlier (legitimately lower) prices would be wrongly excluded as outliers. Instead, trend data uses a <strong>Hampel filter</strong>, a technique widely used in financial time series and signal processing:</p>
<ol>
<li>For each time bucket (day or hour), compute the <strong>median</strong> and <strong>MAD</strong> (median absolute deviation) of prices in a sliding window around that bucket</li>
<li>If the bucket's average price deviates from the window median by more than <strong>3 × MAD</strong>, it is replaced with the window median</li>
</ol>
<p>This approach correctly handles trending data because each data point is only compared against its temporal neighbours, not the entire range. The window covers ±3 days for daily data and approximately ±1 day for hourly data.</p>
<div class="card">
<strong>Transparency:</strong> Outlier prices are never deleted or modified. IQR outliers are flagged
in the materialised view (<code>price_is_outlier = true</code>) and can be inspected on the
<a href="/#anomalies/outliers">Anomalies → Statistical outliers</a> page, which shows the current IQR bounds for each
fuel type and every excluded price with its exclusion reason. Hampel-smoothed trend averages
are computed on the fly and do not alter stored data.
</div>
<h2 id="brand-normalisation">Brand normalisation</h2>
<p>The API provides brand names inconsistently — <code>ESSO</code>, <code>Esso</code>, <code>esso</code> all appear for the same company. Three layers of normalisation clean this up:</p>
<ol>
<li><strong>Brand aliases</strong> — bulk mapping of raw API strings to canonical names (e.g. <code>"TESCO"</code> → <code>"Tesco"</code>). Managed via the <a href="/#data/aliases">Data Cleanup tab</a>.</li>
<li><strong>Station overrides</strong> — per-station corrections for edge cases where the API brand is wrong or the alias isn't granular enough.</li>
<li><strong>Resolution order</strong> — <code>station_override > brand_alias > raw_brand_name</code>. Overrides always win.</li>
</ol>
<p>Raw brand values are <strong>never modified</strong> in the database. Normalisation is applied in the materialised view, so any change can be reversed.</p>
<h2 id="forecourt-categories">Forecourt categories</h2>
<p>Stations are classified into categories based on their <strong>canonical brand name</strong> via a <a href="/#data/categories">lookup table</a>, with the exception of those flagged as <code>is_motorway_service_station</code> in the raw data. This flag always takes priority over our lookup.</p>
<p>We do not honour the <code>is_supermarket_service_station</code> flag from the raw data because it has proved to be too unreliable. For example BP, Texaco, and Maxol forecourts are frequently flagged as supermarkets.</p>
<table>
<thead><tr><th>Category</th><th>Examples</th><th>Description</th></tr></thead>
<tbody>
<tr><td><span class="category-tag supermarket">Supermarket</span></td><td>Tesco, Asda, Sainsburys, Morrisons, Waitrose</td><td>Supermarket-operated forecourts</td></tr>
<tr><td><span class="category-tag major-oil">Major Oil</span></td><td>Shell, BP, Esso, Texaco, Jet, Gulf</td><td>Major oil company brands</td></tr>
<tr><td><span class="category-tag motorway">Motorway Operator</span></td><td>Welcome Break, EG On The Move, Applegreen</td><td>Motorway service area operators</td></tr>
<tr><td><span class="category-tag fuel-group">Fuel Group</span></td><td>Motor Fuel Group, Rontec, Harvest Energy</td><td>Fuel wholesalers / groups</td></tr>
<tr><td><span class="category-tag convenience">Convenience</span></td><td>Spar, Circle K, Maxol</td><td>Convenience store forecourts</td></tr>
<tr><td><span class="category-tag independent">Independent</span></td><td>Brands explicitly categorised as independent</td><td>Independent operators with a known brand identity</td></tr>
<tr><td><span class="category-tag uncategorised">Uncategorised</span></td><td>(any unmapped brand)</td><td>Brand not yet reviewed or assigned a category</td></tr>
<tr><td><span class="category-tag motorway">Motorway</span></td><td>(any station with motorway flag in the source data)</td><td>Motorway flag always takes priority</td></tr>
</tbody>
</table>
<div class="note info">Categories are managed on the <a href="/#data/categories">Data Cleanup tab</a>. After changes, hit <strong>Refresh View</strong> to rebuild the snapshot.</div>
<p style="font-size:0.85rem;">See also <a href="#category-vs-flags">glossary: forecourt categories and the “Supermarket / Motorway” filters</a>.</p>
<h2 id="regional-mapping">Regional mapping</h2>
<p>Each station's postcode is mapped to an ONS-style region using the first 1–2 letters of the postcode (the <em>postcode area</em>). For example, <code>SW1A</code> → <code>SW</code> → <strong>London</strong>, <code>M1</code> → <code>M</code> → <strong>North West</strong>.</p>
<p>This enables regional price comparisons (e.g. "London is 3p/litre more expensive than the North East") without requiring geocoding.</p>
<h2 id="postcodes-enrichment">Postcodes.io enrichment</h2>
<p>Each unique station postcode is looked up via <a href="https://postcodes.io">postcodes.io</a>, a free and open API for UK postcode data. The results are cached in a <code>postcode_lookups</code> table, providing:</p>
<ul>
<li><strong>Authoritative coordinates</strong> — fixes ~85 stations where the Fuel Finder API reports incorrect lat/lng (e.g. sign errors placing stations in the North Sea)</li>
<li><strong>Administrative geography</strong> — local authority district, county, ward, parish</li>
<li><strong>Parliamentary constituency</strong> — for political analysis</li>
<li><strong>Rural/urban classification</strong> — <a href="#rural-urban-classification">ONS RUC 2021</a> (England/Wales) and <a href="#rural-urban-classification">Scottish Government 6-fold</a> (Scotland) categories</li>
<li><strong>Statistical areas</strong> — <a href="https://www.ons.gov.uk/methodology/geography/ukgeographies/statisticalgeographies">LSOA, MSOA</a>, built-up area</li>
</ul>
<p>Postcodes that postcodes.io doesn't recognise are recorded (so they aren't retried) and flagged in the <a href="/#data/postcodes">Data tab</a> as potential data quality issues. Two correction tools are available for editors — see <a href="#data-correction-tools">Data correction tools</a> below.</p>
<h3 id="rural-urban-classification">Rural/urban classification</h3>
<p>The UK does not have a single rural/urban classification — England/Wales and Scotland use separate systems with different categories. Postcodes.io returns the raw classification for the relevant country, so our database stores two distinct sets of labels:</p>
<table>
<thead><tr><th>System</th><th>Source</th><th>Applies to</th></tr></thead>
<tbody>
<tr><td><a href="https://www.ons.gov.uk/methodology/geography/geographicalproducts/ruralurbanclassifications/2021ruralurbanclassification">ONS RUC 2021</a></td><td>Office for National Statistics</td><td>England & Wales</td></tr>
<tr><td><a href="https://www.gov.scot/publications/scottish-government-urban-rural-classification-2020/">Scottish Government Urban Rural Classification</a></td><td>Scottish Government</td><td>Scotland</td></tr>
</tbody>
</table>
<p>The <strong>search and trends filters</strong> expose the full original labels from each system, so you can filter on exactly the category you want. The <strong>dashboard chart</strong> groups each system's categories separately — it does not attempt to equate them, since the two systems use different population thresholds, spatial units, and definitions. Colour coding indicates approximate analogies (red = urban, green = rural, blue = small towns) without asserting equivalence. The mapping is:</p>
<table>
<thead><tr><th>Dashboard label</th><th>Colour</th><th>Source categories</th></tr></thead>
<tbody>
<tr><td><strong>Urban (Eng & Wales)</strong></td><td style="color:#d4351c;">■ Red</td><td>ONS RUC: Urban: Nearer to a major town or city<br>ONS RUC: Urban: Further from a major town or city</td></tr>
<tr><td><strong>Urban (Scot)</strong></td><td style="color:#e8756a;">■ Light red</td><td>SG: Large Urban Areas<br>SG: Other Urban Areas</td></tr>
<tr><td><strong>Small towns (Scot)</strong></td><td style="color:#1d70b8;">■ Blue</td><td>SG: Accessible Small Towns<br>SG: Remote Small Towns</td></tr>
<tr><td><strong>Rural (Eng & Wales)</strong></td><td style="color:#00703c;">■ Green</td><td>ONS RUC: Smaller rural: Nearer to / Further from a major town or city<br>ONS RUC: Larger rural: Nearer to / Further from a major town or city</td></tr>
<tr><td><strong>Rural (Scot)</strong></td><td style="color:#4da67a;">■ Light green</td><td>SG: Accessible Rural<br>SG: Remote Rural</td></tr>
</tbody>
</table>
<div class="note info">
<strong>Why not merge the two systems?</strong> The ONS and Scottish Government classifications use different methodologies. A settlement of 5,000 people could be "Larger rural" in England but "Accessible Small Towns" in Scotland. Keeping the labels separate avoids implying a cross-border comparison that the data doesn't support. Hover over any bar on the dashboard chart to see exactly which source categories it includes.
</div>
<p>Within each system, the proximity dimension is collapsed on the dashboard: the England/Wales "nearer to" and "further from a major town or city" variants are merged, as are Scotland's "Accessible" and "Remote" variants. The full granularity remains available in the search and trends filters.</p>
<h3>Licensing</h3>
<p>Postcodes.io source code is available under the <a href="https://opensource.org/licenses/MIT">MIT Licence</a>. The underlying postcode data is used under the following terms:</p>
<ul>
<li>Great Britain postcode data is used under the <a href="https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/">Open Government Licence</a></li>
<li>Northern Ireland postcode data (BT prefix) is used under the <a href="https://www.ons.gov.uk/methodology/geography/licences">ONSPD licence</a></li>
</ul>
<p style="font-size: 0.85rem; color: var(--muted);">
Contains Ordnance Survey data © Crown copyright and database right 2026.<br>
Contains Royal Mail data © Royal Mail copyright and database right 2026.<br>
Contains National Statistics data © Crown copyright and database right 2026.<br>
Contains NRS data © Crown copyright and database right 2026.
</p>
<h2 id="materialised-view">The materialised view</h2>
<p><code>current_prices</code> is a PostgreSQL materialised view that provides the <strong>latest price per station per fuel type</strong>. It joins together:</p>
<ul>
<li>The most recent price from <code>fuel_prices</code></li>
<li>Station details from <code>stations</code></li>
<li>Canonical brand name (via aliases and overrides)</li>
<li>Forecourt type (via <code>brand_categories</code>)</li>
<li>Region (via <code>postcode_regions</code>)</li>
<li>Human-friendly fuel names (via <code>fuel_type_labels</code>)</li>
<li>Authoritative coordinates, admin district, constituency, rural/urban (via <code>postcode_lookups</code>)</li>
<li>Corrected postcodes (via <code>station_postcode_overrides</code>)</li>
</ul>
<p>The view is refreshed after each scrape run and when you press <strong>Refresh View</strong> on the <a href="/#data/report">Data Cleanup tab</a>. All dashboard, map, search and API queries read from this view for fast, consistent results.</p>
<h2>Running on a schedule</h2>
<p>For production use, the scraper is designed to run on <strong>AWS Lambda</strong> with EventBridge Scheduler:</p>
<ul>
<li><strong>Every 30 minutes</strong>: incremental scrape (fetch changed prices only)</li>
<li><strong>Daily at 03:00 UTC</strong>: full scrape (refresh all station metadata + prices)</li>
</ul>
<p>Raw JSON responses from each scrape are optionally backed up to S3 for audit and replay purposes.</p>
<h2 id="data-correction-tools">Data correction tools</h2>
<p>The GOV.UK Fuel Finder API data is manually entered by station operators and contains various quality issues beyond price errors. The <a href="/#data">Data tab</a> provides tools for editors and admins to correct these without modifying original source data.</p>
<h3>Brand normalisation</h3>
<p>Inconsistent brand strings are cleaned up via <strong>brand aliases</strong> (bulk mapping) and <strong>station overrides</strong> (per-station corrections). See <a href="#brand-normalisation">Brand normalisation</a> above for details. Both are managed on the <a href="/#data/aliases">Data tab</a>.</p>
<h3>Postcode overrides</h3>
<p>Some stations have mistyped, expired or otherwise incorrect postcodes. Since the postcode drives geographic enrichment (region, constituency, district, rural/urban classification), a bad postcode means the station is missing from geographic breakdowns or placed in the wrong area.</p>
<p><strong>Postcode overrides</strong> let editors replace a station's postcode for enrichment purposes. The original postcode is preserved in the <code>stations</code> table. When an override is saved:</p>
<ol>
<li>The corrected postcode is looked up via <a href="https://postcodes.io">postcodes.io</a> to populate full enrichment data</li>
<li>The override is stored in <code>station_postcode_overrides</code></li>
<li>After a view refresh, the <code>current_prices</code> view uses the corrected postcode for all geographic joins</li>
</ol>
<p>Overrides are managed on the <a href="/#data/postcode-overrides">Data tab → Postcode Overrides</a> section, or directly from the <a href="/#data/postcodes">Postcode Issues</a> table via the "Fix postcode" button.</p>
<h3>Coordinate fixes</h3>
<p>When a postcode is not recognised by postcodes.io (and therefore has no authoritative coordinates), the station falls back to the API-reported latitude and longitude — which can be incorrect. Some stations report coordinates outside the UK entirely (e.g. sign errors placing them in the North Sea).</p>
<p>The <a href="/#data/postcodes">Postcode Issues</a> table highlights these stations and provides a "Fix coords" button to manually set latitude and longitude for the postcode. This updates the <code>postcode_lookups</code> table so the corrected coordinates are used in the materialised view.</p>
<p>In many cases, a <strong>postcode override</strong> is a better fix than a coordinate correction — if the postcode itself is wrong, overriding it also fixes the region, constituency and other geographic fields, not just the coordinates.</p>
<h3>Price corrections</h3>
<p>Misreported prices can be corrected via the price editor on the <a href="/#anomalies">Anomalies tab</a>. See <a href="#how-prices-are-stored">How prices are stored</a> for details on anomaly detection. Corrections are stored separately in <code>price_corrections</code> — original price records are never modified.</p>
<h2 id="glossary">Glossary</h2>
<p>Terms used throughout the dashboard that may not be immediately obvious.</p>
<h3>Stations and brands</h3>
<dl>
<dt><strong>Station</strong></dt>
<dd>A physical forecourt at a specific address. Each station has a unique <strong>node ID</strong> assigned by the Fuel Finder API (see below). A station's <em>trading name</em> is what it displays to the public — usually the brand name, but sometimes a locally branded variant (e.g. "Welcome Break Corley").</dd>
<dt><strong>Brand</strong></dt>
<dd>The operator or brand a station trades under — Shell, Tesco, BP, and so on. The API reports a raw brand string per station; normalisation maps these to canonical names (see <em>Brand normalisation</em> above). Multiple stations can share a brand; one brand maps to exactly one forecourt category.</dd>
<dt><strong>Station node ID</strong></dt>
<dd>The unique identifier for a specific station in the Fuel Finder API — a short alphanumeric string (e.g. <code>A1B2C3</code>). It is stable across scrapes and is used as the primary key throughout the database. You need a node ID to add a station-level brand override, or to look up a single station's price history via the API.</dd>
</dl>
<h3 id="category-vs-flags">Forecourt categories and the “Supermarket / Motorway” filters</h3>
<p>There are <strong>two separate ways</strong> to filter by supermarket or motorway, and they mean different things:</p>
<dl>
<dt><strong>Category filter</strong> (the "Supermarket" and "Motorway Operator" / "Motorway" options in the Category multi-select)</dt>
<dd>These use the <em>derived category</em> — the forecourt type we assign based on the station's canonical brand name (see the category table above). "Supermarket" means Tesco, Asda, Sainsbury's, Morrisons, Waitrose; "Motorway" means any station whose brand is in the Motorway Operator category <em>or</em> has the motorway flag set.</dd>
<dt><strong>"Supermarket only" and "Motorway only" checkboxes</strong></dt>
<dd>These use the raw flags reported directly by the Fuel Finder API — <code>is_supermarket_service_station</code> and <code>is_motorway_service_station</code>. The API flags are set by the station operators themselves and are <strong>unreliable</strong>: some major oil company sites are flagged as supermarkets, and some genuine motorway sites are not flagged at all. Use the Category filter for the more accurate, curated classification; use these checkboxes when you specifically want to query what the source data reports.</dd>
</dl>
<h3>Brand normalisation report</h3>
<p>The normalisation report (<a href="/#data/report">Data tab → Report</a>) shows how each raw brand string in the database has been resolved to a canonical name. The <strong>Method</strong> column and the filter dropdown use three terms:</p>
<dl>
<dt><strong>Aliased</strong></dt>
<dd>The raw brand string from the API matched a <em>brand alias</em> rule, so it was mapped to a canonical name (e.g. <code>TESCO PFS</code> → <code>Tesco</code>).</dd>
<dt><strong>Overridden</strong></dt>
<dd>This specific station has a <em>station override</em> that sets its canonical brand directly, regardless of the alias table. Overrides are used for one-off corrections — a station whose API brand is wrong, or where the network-wide alias isn't granular enough. Overrides always take priority over aliases.</dd>
<dt><strong>Unmapped</strong></dt>
<dd>The raw brand string has no alias and no override, <em>and</em> the brand has no entry in the category table. Consequently, it falls back to showing the raw string as its brand name and is classified as "Uncategorised". Unmapped brands are worth reviewing — they may be new entrants to the API, typo variants, or obscure regional operators that need an alias adding.</dd>
</dl>
<h3>Country filter: "Other / Unknown"</h3>
<p>The country field for each station comes from two sources: the Fuel Finder API itself (a free-text country string) and the postcodes.io enrichment lookup. Both are normalised to England, Scotland, Wales, or Northern Ireland.</p>
<p>"Other / Unknown" covers stations where the country cannot be determined — typically because the postcode wasn't recognised by postcodes.io and the API's own country string was blank, misspelled, or didn't match a known value. There are usually only a handful of these at any time; they often correspond to stations with postcode data quality issues.</p>
<h2 id="contact">Contact</h2>
<p>To report problems, request API keys or ask questions, contact Luke Hoyland at <a href="mailto:luke@hoy.la">luke@hoy.la</a>.</p>
<p>Source code is available on <a href="https://github.com/hoyla/fuel-finder" target="_blank" rel="noopener">GitHub</a>.</p>
</main>
</body>
</html>