Skip to content

Commit 65a4867

Browse files
loks0nclaude
loks0n
and
claude
authored
feat(replication): add utopia-php/replication package (#15)
* feat(replication): add utopia-php/replication package A Swoole-native MySQL binlog replication reader for streaming row changes (CDC). A consumer tails a server locally and reacts to writes — cache invalidation, projections, event fan-out — instead of relying on application-level messaging. - Replication: connect, COM_BINLOG_DUMP_GTID, blocking getChanges() generator; GTID checkpoint advanced on commit; schema filter; optional TLS. - Connection: coroutine socket framing, caching_sha2_password auth (fast-auth, RSA full-auth, mysql_native_password fallback), STARTTLS. - EventParser: TABLE_MAP + WRITE/UPDATE/DELETE_ROWS; column names from FULL row metadata; JSON values skipped. - GtidSet, BinaryReader, Change, Exception. Tests: unit (pure decoders) + opt-in e2e (basic CRUD, >16MiB rows, caching_sha2 full-auth, TLS) against a real MySQL 8 via docker compose. Pint + PHPStan level 9 clean. * fix(replication): address PR review (TLS verify, signed ints, ext-swoole, docs) - TLS now verifies the server certificate by default (ssl_verify_peer=true) with optional CA file and an explicit opt-out for self-signed/test setups, instead of silently disabling verification. - Decode signed integers correctly: parse the TABLE_MAP SIGNEDNESS metadata and apply two's-complement to signed TINY/SHORT/INT24/LONG columns (previously all integers were returned unsigned). Validated against real MySQL. - Declare ext-swoole in require (it's a hard runtime dependency, not a suggestion). - Fix the Change::$gtid docblock — the checkpoint is the set committed *before* this event's transaction, not after. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(replication): polymorphic Adapter interface; MySQL under Adapter\MySQL Introduce `Utopia\Replication\Adapter` — a CDC source interface (start / getChanges / stop) — so the package can grow Postgres logical-replication or MongoDB change-stream backends alongside MySQL. The MySQL binlog implementation and its protocol internals move under `Utopia\Replication\Adapter\MySQL`: - Utopia\Replication\Adapter (new interface) - Utopia\Replication\Adapter\MySQL (was Replication; implements Adapter) - Utopia\Replication\Adapter\MySQL\{Connection,EventParser,GtidSet,BinaryReader,Constants} `Change` and `Exception` stay top-level (shared contract). Schema filtering moved from the fluent setSchema() into the MySQL constructor. Naming follows utopia-php/database (Adapter\MySQL, MariaDB, Postgres, …). Pint + PHPStan level 9 clean; unit + e2e suites updated and green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0 parents  commit 65a4867

23 files changed

Lines changed: 2217 additions & 0 deletions

‎.github/workflows/mirror.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
name: Mirror
2+
3+
on:
4+
pull_request_target:
5+
types: [opened]
6+
7+
permissions:
8+
pull-requests: write
9+
10+
jobs:
11+
redirect:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Redirect pull request to the monorepo
15+
env:
16+
GH_TOKEN: ${{ github.token }}
17+
PR: ${{ github.event.pull_request.html_url }}
18+
run: |
19+
gh pr comment "$PR" --body "Thanks for contributing! This repository is a read-only mirror — development for this library happens in [\`packages/replication\`](https://github.com/utopia-php/monorepo/tree/main/packages/replication) in the [utopia-php monorepo](https://github.com/utopia-php/monorepo). Please open this pull request there instead."
20+
gh pr close "$PR"

‎.github/workflows/tests.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: "Tests"
2+
3+
concurrency:
4+
group: ${{ github.workflow }}-${{ github.ref }}
5+
cancel-in-progress: true
6+
7+
on: [pull_request]
8+
9+
jobs:
10+
test:
11+
name: Tests
12+
runs-on: ubuntu-latest
13+
steps:
14+
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # ratchet:actions/checkout@v6
15+
16+
- uses: shivammathur/setup-php@f3e473d116dcccaddc5834248c87452386958240 # ratchet:shivammathur/setup-php@v2
17+
with:
18+
php-version: '8.4'
19+
extensions: swoole, pdo_mysql, openssl
20+
tools: phpunit
21+
coverage: none
22+
23+
- name: Install dependencies
24+
run: composer install --prefer-dist --no-progress
25+
26+
- name: Unit tests
27+
run: composer test
28+
29+
- name: Start services
30+
run: docker compose up -d --wait
31+
32+
- name: E2E tests
33+
run: composer test:e2e

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
/vendor/
2+
/composer.lock
3+
/.phpunit.cache/
4+
/.phpunit.result.cache
5+
/.DS_Store
6+
.idea/
7+
.vscode/

‎CHANGELOG.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Changelog
2+
3+
## 0.1.0
4+
5+
### Added
6+
7+
- Initial release: a Swoole-native MySQL binlog replication reader.
8+
- `Replication` — connect, `COM_BINLOG_DUMP_GTID`, blocking `getChanges()`
9+
generator yielding `Change` objects; GTID checkpoint advanced on commit;
10+
schema filtering; optional TLS.
11+
- `Connection` — coroutine socket framing with `caching_sha2_password`
12+
authentication (fast-auth, RSA full-auth, `mysql_native_password` fallback)
13+
and STARTTLS.
14+
- `EventParser` — TABLE_MAP + WRITE/UPDATE/DELETE_ROWS decoding; column names
15+
from FULL row metadata; JSON values skipped.
16+
- `GtidSet`, `BinaryReader`, `Change`, `Exception`.

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Appwrite Team
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Utopia Replication
2+
3+
> [!IMPORTANT]
4+
> This repository is a read-only mirror of the [utopia-php monorepo](https://github.com/utopia-php/monorepo). Development happens in [`packages/replication`](https://github.com/utopia-php/monorepo/tree/main/packages/replication) — please open issues and pull requests there.
5+
6+
[![Build Status](https://github.com/utopia-php/replication/actions/workflows/tests.yml/badge.svg)](https://github.com/utopia-php/replication/actions/workflows/tests.yml)
7+
![Total Downloads](https://img.shields.io/packagist/dt/utopia-php/replication.svg)
8+
[![Discord](https://img.shields.io/discord/564160730845151244?label=discord)](https://appwrite.io/discord)
9+
10+
Utopia Replication is a Swoole-native MySQL binlog reader. It streams row-level changes (change data capture) from a MySQL server over the replication protocol, so a consumer can react to writes — invalidate caches, build projections, fan out events — by tailing a server locally instead of relying on application-level messaging. This library is maintained by the [Appwrite team](https://appwrite.io).
11+
12+
Although this library is part of the [Utopia Framework](https://github.com/utopia-php/framework) project it is dependency free and can be used as standalone with any other PHP project or framework.
13+
14+
## Getting Started
15+
16+
Install using composer:
17+
```bash
18+
composer require utopia-php/replication
19+
```
20+
21+
## System Requirements
22+
23+
Utopia Replication requires PHP 8.3 or later, and the [Swoole](https://github.com/swoole/swoole-src) extension (>=6.0) for coroutine socket I/O. The [OpenSSL](https://www.php.net/manual/en/book.openssl.php) extension is required for `caching_sha2_password` full authentication and TLS.
24+
25+
### Source server
26+
27+
The MySQL source must be configured for GTID-based row replication:
28+
29+
```ini
30+
binlog_format = ROW
31+
gtid_mode = ON
32+
enforce_gtid_consistency = ON
33+
binlog_row_metadata = FULL # so column names arrive in the stream
34+
```
35+
36+
The connecting user needs the `REPLICATION SLAVE` and `REPLICATION CLIENT` privileges.
37+
38+
## Usage
39+
40+
```php
41+
use Utopia\Replication\Adapter;
42+
use Utopia\Replication\Adapter\MySQL;
43+
44+
// Adapter is the polymorphic interface; MySQL is the binlog implementation.
45+
$replication = new MySQL(
46+
host: '127.0.0.1',
47+
port: 3306,
48+
username: 'replicator',
49+
password: 'secret',
50+
serverId: 1001, // unique among replicas
51+
schema: 'appwrite', // only emit changes for this database
52+
ssl: false, // when true, TLS verifies the server cert by default
53+
);
54+
55+
$replication->start($checkpoint ?? null); // resume from a GTID set, or null for "now"
56+
57+
foreach ($replication->getChanges() as $change) {
58+
// $change->action — Change::INSERT | UPDATE | DELETE
59+
// $change->table — physical table name
60+
// $change->rows — list of column => value maps (after-image for updates)
61+
// $change->gtid — executed-GTID-set checkpoint (advance on commit)
62+
63+
foreach ($change->rows as $row) {
64+
// react to the change ...
65+
}
66+
67+
$checkpoint = $change->gtid; // persist to resume after a restart
68+
}
69+
```
70+
71+
The reader runs inside a Swoole coroutine; `getChanges()` blocks (yielding the
72+
coroutine) while waiting for events. The GTID checkpoint advances on transaction
73+
commit, so a crash mid-transaction re-streams it — treat changes as idempotent.
74+
75+
## Scope
76+
77+
Deliberately minimal: MySQL 8, ROW-format binlog, and the event types needed for
78+
CDC (TABLE_MAP, WRITE/UPDATE/DELETE_ROWS, GTID, ROTATE, XID). JSON column values
79+
are skipped (bytes advanced, not decoded). Column names come from FULL row
80+
metadata, so no `INFORMATION_SCHEMA` lookups are required.
81+
82+
## Tests
83+
84+
```bash
85+
composer test # unit (pure decoders — no server needed)
86+
docker compose up -d --wait
87+
composer test:e2e # against a real MySQL 8 (basic CRUD, >16MiB rows,
88+
# caching_sha2 full-auth, TLS)
89+
```
90+
91+
## Copyright and license
92+
93+
The MIT License (MIT) [http://www.opensource.org/licenses/mit-license.php](http://www.opensource.org/licenses/mit-license.php)

‎composer.json‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
{
2+
"name": "utopia-php/replication",
3+
"description": "A Swoole-native MySQL binlog replication reader for streaming row changes (CDC)",
4+
"type": "library",
5+
"license": "MIT",
6+
"authors": [
7+
{
8+
"name": "Appwrite Team",
9+
"email": "team@appwrite.io"
10+
}
11+
],
12+
"require": {
13+
"php": ">=8.3",
14+
"ext-swoole": "*"
15+
},
16+
"require-dev": {
17+
"swoole/ide-helper": "*"
18+
},
19+
"suggest": {
20+
"ext-openssl": "Required for caching_sha2_password full authentication and TLS"
21+
},
22+
"autoload": {
23+
"psr-4": {
24+
"Utopia\\Replication\\": "src/"
25+
}
26+
},
27+
"autoload-dev": {
28+
"psr-4": {
29+
"Utopia\\Replication\\Tests\\": "tests/"
30+
}
31+
},
32+
"scripts": {
33+
"test": "phpunit --testsuite unit",
34+
"test:e2e": "phpunit --testsuite e2e"
35+
},
36+
"config": {
37+
"platform": {
38+
"php": "8.3"
39+
},
40+
"optimize-autoloader": true,
41+
"sort-packages": true
42+
},
43+
"minimum-stability": "stable",
44+
"prefer-stable": true
45+
}

‎docker-compose.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
services:
2+
mysql:
3+
image: mysql:8.0
4+
command: >
5+
--server-id=1
6+
--log-bin=mysql-bin
7+
--binlog-format=ROW
8+
--gtid-mode=ON
9+
--enforce-gtid-consistency=ON
10+
--binlog-row-metadata=FULL
11+
environment:
12+
MYSQL_ROOT_PASSWORD: password
13+
MYSQL_DATABASE: replication_test
14+
ports:
15+
- "13306:3306"
16+
healthcheck:
17+
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-uroot", "-ppassword"]
18+
interval: 2s
19+
timeout: 2s
20+
retries: 30

‎phpstan.neon‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
parameters:
2+
level: 9
3+
paths:
4+
- src
5+
- tests
6+
excludePaths:
7+
- vendor
8+
treatPhpDocTypesAsCertain: false

‎phpunit.xml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
3+
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/11.5/phpunit.xsd"
4+
bootstrap="vendor/autoload.php"
5+
colors="true"
6+
cacheDirectory=".phpunit.cache"
7+
failOnWarning="true"
8+
failOnNotice="true">
9+
<testsuites>
10+
<testsuite name="unit">
11+
<directory>tests/Unit</directory>
12+
</testsuite>
13+
<testsuite name="e2e">
14+
<directory>tests/E2E</directory>
15+
</testsuite>
16+
</testsuites>
17+
<source>
18+
<include>
19+
<directory>src</directory>
20+
</include>
21+
</source>
22+
<php>
23+
<env name="REPLICATION_TEST_HOST" value="127.0.0.1"/>
24+
<env name="REPLICATION_TEST_PORT" value="13306"/>
25+
<env name="REPLICATION_TEST_USER" value="root"/>
26+
<env name="REPLICATION_TEST_PASS" value="password"/>
27+
</php>
28+
</phpunit>

0 commit comments

Comments
 (0)