feat: cover dictionary-encoded bind parameter columns#256
Conversation
Add a test for binds a dictionary-encoded string column (what pandas often produce). Dictionary encoding is an encoding of the same logical values, not a different logical type (Arrow columnar format, Dictionary-encoded Layout), so a driver that binds plain string columns should accept the dictionary-encoded equivalent, decoding it if the database has no native counterpart: > https://arrow.apache.org/docs/format/Columnar.html#dictionary-encoded-layout Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
5f79835 to
e980017
Compare
👍 Made the transition to .txtcase in abe8932, adding the necessary dictionary support to the test machinery. I didn't see a clean fit for deriving from an existing .txtcase, but let me know if you think otherwise. |
|
Sorry, I should've explained more...This is what I mean: validation/adbc_drivers_validation/tests/ingest.py Lines 628 to 642 in 94f8332 It's "parametrized" but the parameterization only ever uses a single test case. The test itself then transforms the data before doing something with it. That way it effectively inherits whatever customization was done for that test case without requiring a separate test case. |
|
Thanks! I transitioned to parameterization in 2ac53ff - does that match what you meant? I found that support for dictionary encoded types other than strings is lacking in at least the sqlite/postgres driver (looking into a PR for that), so for now I'm just checking |
Add a test for binds a dictionary-encoded column (what pandas often produce).
Dictionary encoding is an encoding of the same logical values, not a different logical type (Arrow columnar format, Dictionary-encoded Layout), so a driver that binds plain string columns should accept the dictionary-encoded equivalent, decoding it if the database has no native counterpart:
Verified against existing drivers: sqlite, postgres, mysql and flightsql (for the example server it needed a fix: apache/arrow-go#936).
Context: Missed this when implementing a new driver, so having this test would have helped.