Skip to content

[DRAFT] Add decoder for acquiring data in NDJSON format#129

Draft
amotl wants to merge 1 commit into
bulk-json-compactfrom
bulk-ndjson
Draft

[DRAFT] Add decoder for acquiring data in NDJSON format#129
amotl wants to merge 1 commit into
bulk-json-compactfrom
bulk-ndjson

Conversation

@amotl

@amotl amotl commented Mar 4, 2023

Copy link
Copy Markdown
Member

Hi there,

this adds a decoder for NDJSON payloads. NDJSON is the Newline Delimited JSON format, and it is suitable for encoding/transmitting multiple JSON records in bulk, or for streaming them.

The standard JSON variant implemented with GH-40 is not really suitable for submitting data from embedded devices, because wrapping up multiple JSON records into an array will probably blow up your memory quickly.

NDJSON has been called LDJSON before being reasonably specified, and has also been known as JSON Lines, see also JSON streaming. It looks like this:

{"temperature":21.42,"humidity":41.55}
{"temperature":42.84,"humidity":83.1}

With kind regards,
Andreas.

/cc @Tonkenfo, @wetterfrosch, @msweef, @elektra42

@amotl amotl force-pushed the bulk-json-compact branch 3 times, most recently from 3011c2a to bef0e8b Compare February 11, 2026 04:16
@coderabbitai

coderabbitai Bot commented Feb 11, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch bulk-ndjson

Comment @coderabbitai help to get the list of available commands and usage tips.

@amotl amotl force-pushed the bulk-ndjson branch 2 times, most recently from cba1ce2 to b2ab31f Compare February 12, 2026 05:45
@codecov

codecov Bot commented Feb 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.79%. Comparing base (bef0e8b) to head (d181b1c).

Additional details and impacted files
@@                  Coverage Diff                  @@
##           bulk-json-compact     #129      +/-   ##
=====================================================
+ Coverage              78.70%   78.79%   +0.09%     
=====================================================
  Files                     57       58       +1     
  Lines                   3062     3075      +13     
=====================================================
+ Hits                    2410     2423      +13     
  Misses                   652      652              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment on lines +31 to +41
@staticmethod
def decode(payload):

# Decode from NDJSON, using pandas.
import io
import pandas as pd
df = pd.read_json(io.StringIO(payload), lines=True)

# Transform to records again.
data = df.to_dict(orient="records")
return data

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The decoder could use orjson and orjsonl from the start, skipping pandas completely?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant