Is the API documented somewhere?

[chmac]

Firstly, thanks for all your work on crdt, it's associated packages, and tudo. It's a rich tapestry of code to learn from as somebody exploring Flutter and Dart.

Is the API for crdt or any of the sub packages (hive_crdt, sqlite_crdt) documented somewhere?

I'm new to the whole Dart ecosystem so maybe I'm missing something obvious. I'm familiar with CRDTs but I'm struggling to see how all the pieces you've built fit together. I've checked out the examples that I can find, and spent some time looking into the tudo code. But I'm still not quite able to see the bigger picture.

[cachapa]

Thanks for the comments.
You reminded me that discussions were not yet enabled for this repository, so I just did that and transferred your comment here.

I tried to document all of the crdt* packages with inline docs. Those are shown when using the API in any modern IDE, but also get auto-generated by pub.dev, e.g.: https://pub.dev/documentation/crdt or https://pub.dev/documentation/sqlite_crdt

As for usage I'm afraid there's only the package examples and tudo, but if you have specific questions I'm happy to answer them as well.

[chmac]

Aha, that's great, I didn't realise that docs were published on pub.dev. Thanks for that link, that's super helpful. I guess I can also find docs for most other Dart packages on there! 🧡

I just watched the CRDTs for Mortals talk. I think you're using a different approach in sql_crdt if I am not mistaken.

I'll try to outline how I understand your code works, and perhaps you can correct me if I've misunderstood some parts.

• Data is stored in SQLite tables
• Additional metadata gets added to every table (hlc, modified, is_deleted)
• A list of CRDT compliant update operations are produced by getChangeset()
• Those updates can be merged into another database with merge()

Did I get the gist of that more or less correct?

I'm coming at this from the talk Turning the database inside out, which I think is where I'm getting confused. I think your implementation doesn't actually store a list of events. It merely stores the latest version of the state, alongside an HLC value.

One concrete question which I haven't been able to figure out yet:

• CREATE TABLE users (id INTEGER NOT NULL, username TEXT, first_name TEXT, last_name TEXT)
• On node a do INSERT INTO users (id, username) VALUES ("jd")
• On node a do UPDATE users SET first_name = 'John'
• On node b do UPDATE users SET last_name = 'Doe'
• Merge changes from a into b

What is the final output? Does the row contain both first_name = 'John' and last_name = 'Doe'?

I don't have Dart / Flutter installed and running (yet), so I can't run the examples to see what actually gets stored in the database. I would assume that if there is only a single hlc value for the whole row, then the algorithm is latest row version wins, which means in my example above, the last_name value would be lost. But I'm not 100% sure if I got this right or not.

[cachapa]

Did I get the gist of that more or less correct?

Perfectly.

I think your implementation doesn't actually store a list of events. It merely stores the latest version of the state, alongside an HLC value.

Correct, my crdt packages implement a last-write-wins strategy since they are intended to be used in clients where storage space and performance are at a premium, e.g. mobile.

Depending on the use case your implementation could store the list of events itself and use the crdt as a source of truth for the current state.

What is the final output? Does the row contain both first_name = 'John' and last_name = 'Doe'?

The result would be a bunch of errors since your insert is missing the id :-)

But seriously, if I understand your intent correctly, the result would be that either first_name or last_name would be lost.

At some point I experimented with an implementation which worked on a field level, meaning each field had its own HLC and thus your example would merge correctly.

However there are a number of fundamental problems with that strategy:

• Atomic merges cannot be guaranteed, i.e. you might inadvertently sync only part of an object, which means you can't guarantee the merged value is valid without making a lot of client-side verifications.
• All columns except for ids in your table need to be nullable otherwise you wouldn't be able to merge field by field.
• Since you need to store hlc and modified timestamps for each field, the database grows very quickly.
• The performance was pretty bad. Granted my implementation may not have been ideal but you're still dealing with at least an order or magnitude more data than in an object model.

If your implementation foresees that first_name and last_name may be independently updated by different nodes, then you can reflect that in your database design, e.g. by storing them in separate tables, or in a single table with id & field as primaries.

[chmac]

Okay, gotcha. Now it all starts to make sense to me. I was struggling to put all the pieces together, but row level set operations make sense. Store the latest version with a HLC, newest write wins. Cool.

Thanks for taking the time to explain. I've got some thinking to do.

Again, I really like the look of your packages and flutter in general. Thanks for publishing it.

[cachapa]

Let me know if you decide to use it for your projects, I'm always curious about how others are applying it.