Skip to content

Update docs on new table formats #582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kkin-here wants to merge 1 commit into
base: lib_data
Choose a base branch
from
Open

Update docs on new table formats #582

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/latest/JBON2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -554,7 +554,7 @@ When a replication is done, it depends on the implementation. Technically, when

In the reference implementation using PostgresQL or SqLite the storage engine will not accept tuples as input. It will only accept the `feature` as [JSON] map, and the `attachment` as `byte[]` _(with some metadata, so in a wrapper)_. Then it will directly encode the tuple by itself. Finally, it will truncate the tuple before the `attachment`, and store all the truncated values in dedicated database columns. It will split the `storage_book` as well into own dedicated columns, and fill a helper table for the `tags`. The design is intentionally made, so that a database can index values, while a pure cache just takes the tuple as is, and stores it. Every storage can basically restore the `object` and `attachment` from a tuple, and then re-encode the tuple so that it is optimized for the storage. Even while this costs some CPU time, it allows to have a very efficient storage, and makes all replicas very efficient, and optimized. It is actually a form of logical replication.

When reading back a tuple from the storage, the PostgresQL storage-engine re-constructs the full tuple binary from the database table. It will restore the basic tuple form the table, appending the `attachment` again, it as well knows the final byte-size. Then it will restore the `global_book_tn`, `tuple_number`, `next_version`, and `prev_version` from database columns. The `storage_book` is restored from custom columns, while the tags are restored from a helper table. The resulting tuple contains all data again, without that it has to be re-encoded. All of this is just copy values into a fixed size buffer, where the size is already well known.
When reading back a tuple from the storage, the PostgresQL storage-engine re-constructs the full tuple binary from the database table. It will restore the basic tuple form the table, appending the `attachment` again, it as well knows the final byte-size. Then it will restore the `global_book_tn`, `tuple_number`, and `next_version` from database columns. The `storage_book` is restored from custom columns, while the tags are restored from a helper table. The resulting tuple contains all data again, without that it has to be re-encoded. All of this is just copy values into a fixed size buffer, where the size is already well known.

This concept makes reading of data very efficient and quite simple, while writing is slightly more expensive. The design of [books] is very supportive for this design.

Expand Down
59 changes: 29 additions & 30 deletions docs/latest/LIB_DATA.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -601,13 +601,13 @@ SELECT * FROM table WHERE version <= $version AND next_version > $version AND {o

This general query will only return one [tuple] with the latest state of the [feature] that belongs to this [version]. Beware, the returned [tuple] can be in a lower version, this query just ensured that the [tuple] that belongs logically to the queried [version] of the [database] is returned. Let's review this, assume we have the following data:

| db-row | id | version | next_version | prev_version | action |
|--------|-------|----------------------|------------------|--------------|----------------|
| 1 | `foo` | 590 (`10010011_10b`) | 9007199254740991 | 77 | DELETE (`10b`) |
| 2 | `foo` | 77 (`10011_01b`) | 590 | 33 | UPDATE (`01b`) |
| 3 | `foo` | 33 (`1000_01b`) | 77 | 13 | UPDATE (`01b`) |
| 4 | `foo` | 13 (`11_01b`) | 33 | 4 | UPDATE (`01b`) |
| 6 | `foo` | 4 (`10_00b`) | 13 | 0 | CREATE (`00b`) |
| db-row | id | version | next_version | action |
|--------|-------|----------------------|------------------|----------------|
| 1 | `foo` | 590 (`10010011_10b`) | 9007199254740991 | DELETE (`10b`) |
| 2 | `foo` | 77 (`10011_01b`) | 590 | UPDATE (`01b`) |
| 3 | `foo` | 33 (`1000_01b`) | 77 | UPDATE (`01b`) |
| 4 | `foo` | 13 (`11_01b`) | 33 | UPDATE (`01b`) |
| 6 | `foo` | 4 (`10_00b`) | 13 | CREATE (`00b`) |

**Note**: The lower two bit of the version of a [tuple] encodes the action!

Expand Down Expand Up @@ -636,13 +636,13 @@ We can see, that the version condition will select row `#1`, but the added secon
### Query multiple versions
Assuming the same data state as above:

| db-row | id | version | next_version | prev_version | action |
|--------|-------|----------------------|------------------|--------------|----------------|
| 1 | `foo` | 590 (`10010011_10b`) | 9007199254740991 | 77 | DELETE (`10b`) |
| 2 | `foo` | 77 (`10011_01b`) | 590 | 33 | UPDATE (`01b`) |
| 3 | `foo` | 33 (`1000_01b`) | 77 | 13 | UPDATE (`01b`) |
| 4 | `foo` | 13 (`11_01b`) | 33 | 4 | UPDATE (`01b`) |
| 6 | `foo` | 4 (`10_00b`) | 13 | 0 | CREATE (`00b`) |
| db-row | id | version | next_version | action |
|--------|-------|----------------------|------------------|----------------|
| 1 | `foo` | 590 (`10010011_10b`) | 9007199254740991 | DELETE (`10b`) |
| 2 | `foo` | 77 (`10011_01b`) | 590 | UPDATE (`01b`) |
| 3 | `foo` | 33 (`1000_01b`) | 77 | UPDATE (`01b`) |
| 4 | `foo` | 13 (`11_01b`) | 33 | UPDATE (`01b`) |
| 6 | `foo` | 4 (`10_00b`) | 13 | CREATE (`00b`) |

We can search for multiple versions of a feature, an only limit the lower or upper end. So, search for all version till version `500`:

Expand All @@ -652,33 +652,33 @@ SELECT * FROM table WHERE version <= 503 AND id = 'foo';

Result is:

| db-row | id | version | next_version | prev_version | action |
|--------|-------|----------------------|------------------|--------------|----------------|
| 2 | `foo` | 77 (`10011_01b`) | 590 | 33 | UPDATE (`01b`) |
| 3 | `foo` | 33 (`1000_01b`) | 77 | 13 | UPDATE (`01b`) |
| 4 | `foo` | 13 (`11_01b`) | 33 | 4 | UPDATE (`01b`) |
| 6 | `foo` | 4 (`10_00b`) | 13 | 0 | CREATE (`00b`) |
| db-row | id | version | next_version | action |
|--------|-------|----------------------|------------------|----------------|
| 2 | `foo` | 77 (`10011_01b`) | 590 | UPDATE (`01b`) |
| 3 | `foo` | 33 (`1000_01b`) | 77 | UPDATE (`01b`) |
| 4 | `foo` | 13 (`11_01b`) | 33 | UPDATE (`01b`) |
| 6 | `foo` | 4 (`10_00b`) | 13 | CREATE (`00b`) |

To query for all versions beyond version `500`:

```sql
SELECT * FROM table WHERE version > 503 AND id = 'foo';
```

| db-row | id | version | next_version | prev_version | action |
|--------|-------|----------------------|------------------|--------------|----------------|
| 1 | `foo` | 590 (`10010011_10b`) | 9007199254740991 | 77 | DELETE (`10b`) |
| db-row | id | version | next_version | action |
|--------|-------|----------------------|------------------|----------------|
| 1 | `foo` | 590 (`10010011_10b`) | 9007199254740991 | DELETE (`10b`) |

Query for all versions in the range of version `15` and `503`:

```sql
SELECT * FROM table WHERE version <= 503 AND version > 15 AND id = 'foo
```

| db-row | id | version | next_version | prev_version | action |
|--------|-------|----------------------|------------------|--------------|----------------|
| 2 | `foo` | 77 (`10011_01b`) | 590 | 33 | UPDATE (`01b`) |
| 3 | `foo` | 33 (`1000_01b`) | 77 | 13 | UPDATE (`01b`) |
| db-row | id | version | next_version | action |
|--------|-------|----------------------|------------------|----------------|
| 2 | `foo` | 77 (`10011_01b`) | 590 | UPDATE (`01b`) |
| 3 | `foo` | 33 (`1000_01b`) | 77 | UPDATE (`01b`) |

We additionally can filter on the `action`.

Expand Down Expand Up @@ -719,7 +719,7 @@ Next to the **historic partitioning** of the _HISTORY_ section, there is a gener

The _HEAD_ section and each historic partition are optionally distribution partitioned, if enabled. This is an optional feature that by default is disabled, but can be enabled to store a huge number of [features] and [tuple] in a [collection]. When enabled, we distribute [features] across distribution partitions. The number of distribution partitions can be configured when creating a collection, and defaults to `1` _(so no distribution partitioning)_. To assign [features], and all their [tuple], to the same distribution partition, the lower 16-bit of the record-number are used as **distribution key**. This means, all [tuple] of a [feature] are stored in the same distribution partition. So, when loading a [tuple] of a [feature] in a specific [version], only a single partition has to be accessed. When searching for data, all partitions can be queried in parallel, improving search performance. This layout therefore speeds up searching for data, while making access to known data faster. Loading the _HEAD_ state _([tuple])_ technically means to query a single partition and is therefore rather very fast. Loading multiple tuple can be done in parallel from all partitions they are in. The distribution is simply done by dividing the unsigned lower 16-bit value of the record-number by the amount of distribution partitions _(`n`)_, using the division rest as partition index. For example, assume the distribution key of a [feature] is `1234`, so the unsigned lower 16-bit of the record-number is decimal `1234`, and we have `8` distribution partitions configured in the [collection], then dividing `1234` by `8` gives us `154` with a rest of `2`. Therefore, the partition-number to search in is `2`. This guarantees that we always have a partition index between `0` and `n`-1.

The **historic partitioning** is done by the `next_version` of each [tuple]. All [tuple] with `next_version` being `9,007,199,254,740,991`, are located in the _HEAD_ section, and are only distribution partitioned. When a new [tuple] of a [feature] is created, the current [tuple] in the _HEAD_ section becomes historic data. It now needs to be moved to history, and `next_version` must be set to the version of the new [tuple]. The [tuple] should be relocated into the _HISTORY_, which is where **historic partitioning** happens. It will stay in _HISTORY_ immutable until being purged. The purging is normally done by deletion of complete historic partitions, which is the reason for this design. Beware that formally the immutability of a [Tuple] slightly broken here, because the `next_version` is modified while moving the [Tuple] into history. However, this is the only exception, and a not significant one for the caches. Actually `next_version` is no reliable field, applications should ignore it. The value can be calculated using the back-references from `prev_version`, starting at _HEAD_.
The **historic partitioning** is done by the `next_version` of each [tuple]. All [tuple] in the _HEAD_ section have no `next_version` (it is intrinsically [HEAD], `9,007,199,254,740,991`, and is therefore not stored in HEAD rows). When a new [tuple] of a [feature] is created, the current [tuple] in the _HEAD_ section becomes historic data. It is moved into _HISTORY_, and at insertion time `next_version` is set to the version of the new [tuple]. From that point on the [tuple] is immutable in _HISTORY_ until being purged. The purging is normally done by deletion of complete historic partitions, which is the reason for this design. The walk-back from a known _HEAD_ to its predecessors is performed via `next_version`, by searching _HISTORY_ for the row whose `next_version` matches the known later version.

Now, when deciding in which historic partition a [Tuple] should be located a **partition-key** is needed. To generate the **partition-key** the value from `next_version` is used. For this, the `next_version` is bitwise-ANDed with `0x001F_FFFF_FFFF_FFFF` _(effectively clearing the top 12-bit)_. Then value is shifted right by a configured `shift` amount. The `shift` is configured when creating a [collection] and must stay constant for the whole lifetime of a [collection]. The `shift` defaults to `41`, which means we store one historic partition per year. Reducing the `shift` to `37` would result in one historic partition per month, and reducing it to `32` would result in one historic partition per day.

Expand Down Expand Up @@ -906,8 +906,7 @@ The minimal **mandatory** members that are defined by this specification for all

- `number`: `int64` - The record number. If negative, the `id` of the record is stored in the `meta` section of the collection. If positive, the `id` of the record is the number as decimal string.
- `version`: `uint52` - The version of the record.
- `prev_version`: `uint52` - The previous version of the record, defaulting to [NULL].
- `next_version`: `uint52` - The next version of the record, defaulting to [HEAD].
- `next_version`: `uint52` - The next version of the record, defaulting to [HEAD]. Only stored in _HISTORY_ — in _HEAD_ this is intrinsically [HEAD].
- `global_book_id`: `uint32?` - The _(optional)_ identifier into the constants of this collection that store the identifier of the global book to use for this record.
- `uuid`: `string` - The _(optional)_ identifier into the constants of this collection that store the identifier of the global book to use for this record.

Expand Down
19 changes: 9 additions & 10 deletions docs/latest/LIB_DATA_POSTGRES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,7 @@ All tables used in the Naksha PostgresQL implementation have the same general la
|--------------|-------|------------------|---------------|----------------|------------------------------------------------------------------------------------------------|
| fn | int8 | `FnCol` | `fn` | NOT NULL | The feature number. |
| version | int8 | `VersionCol` | `version` | NOT NULL | The feature version. |
| next_version | int8 | `NextVersionCol` | `nextVersion` | NOT NULL | The next-version does not exist in _HEAD_, because it would always be `4,503,599,627,370,495`. |
| prev_version | int8 | `PrevVersionCol` | `prevVersion` | | The previous version, `null` if this is the first tuple. |
| next_version | int8 | `NextVersionCol` | `nextVersion` | NOT NULL | _HISTORY only_. The next-version does not exist in _HEAD_, because it would always be `4,503,599,627,370,495`. |
| id | text | `IdCol` | `id` | | The _(optional)_ unique identifier of the feature. |
| data | bytea | `DataCol` | `data` | | The [JBON] encoded tuple. |

Expand Down Expand Up @@ -349,20 +348,20 @@ Writing will implement these steps:
- Sort all features by partition-number, feature-number
- **This is very important to avoid deadlocks!**
- Technically we now need to do, in order:
- Select the current _HEAD_ state of all features to be modified.
- Select the current _HEAD_ state of all features to be modified, capturing each row's existing `version` as `existing_version` so the client can do atomic-write checks.
- Copy the _HEAD_ state into _HISTORY_, setting `next_version` to the current version.
- Insert all tuple that do not exist yet in _HEAD_ table with `version` set to the current version, `prev_version` set to `null`.
- Insert all tuple that do not exist yet in _HEAD_ table with `version` set to the current version.
- This includes `DELETE` entries for deleted and purged objects.
- This should return `action=INSERT`, `fn`, `version`, and `prev_version` as `null`.
- Update all _HEAD_ rows, copy `version` into `prev_version` and set `version` to current version.
- This should return `action=INSERT`, `fn`, `version`, and `existing_version` as `null`.
- Update all _HEAD_ rows, setting `version` to current version.
- This includes `DELETE` entries for deleted and purged objects.
- This should return `action=UPDATE`, `fn`, `version`, and `prev_version`.
- This should return `action=UPDATE`, `fn`, `version`, and `existing_version`.
- Finally, for those that should be purged.
- Copy the _HEAD_ state into _HISTORY_.
- Delete the _HEAD_ state.
- Return `action=PURGE`, `fn`, `version`, and `prev_version`.
- Eventually, the whole CTE selects the `action`, `fn`, `version`, and `prev_version` from all sub-queries _(UNION ALL)_.
- The client now compares if all features are modified, and their `version` and `prev_version` is in the expected state.
- Return `action=PURGE`, `fn`, `version`, and `existing_version`.
- Eventually, the whole CTE selects the `action`, `fn`, `version`, and `existing_version` from all sub-queries _(UNION ALL)_.
- The client now compares if all features are modified, and their `version` and `existing_version` is in the expected state.
- If it is, the client can commit.
- Otherwise, it needs to roll back and return a conflict for those objects with wrong version.

Expand Down