Add support for Nvidia Jetson V4L2 M2M video encoder by chenosaurus · Pull Request #1161 · livekit/rust-sdks

chenosaurus · 2026-06-11T06:22:54Z

Add support for Nvidia Jetson V4L2 M2M video encoder
Support H264, H265, AV1 encoding
Update the local_video example with --source flag that can be uvc or argus

…to dc/exp/video_timing

Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-06-11T06:23:05Z

Changeset

The following package versions will be affected by this PR:

Package	Bump
`libwebrtc`	`patch`
`webrtc-sys`	`patch`

…dks into dc/exp/jetson_encode_0527

ladvoc · 2026-06-11T15:51:52Z

 nokhwa = { git = "https://github.com/l1npengtul/nokhwa", rev = "4923ecab7cf26f9dba83867a15a9d8662d021296", default-features = false, features = ["input-msmf"] }
+
+[build-dependencies]
+cc = "1"


Use the workspace dependency here (see top-level Cargo.toml). The parallel feature is enabled for better performance

ladvoc · 2026-06-11T15:54:33Z

    fallback
 }

+fn find_video_outbound_stats(


note: Outside the scope of this PR, but we might want to add some helpers in the core SDK to handle this.

yea good idea

ladvoc · 2026-06-11T15:55:52Z

+
+/// `CLOCK_MONOTONIC` value used to translate Argus sensor timestamps into wall time.
+#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
+fn monotonic_time_ns_now() -> Option<u64> {


suggestion (non-blocking): Consider keeping Argus specific helpers in their own module to keep the main module lean.

moved them to the argus mod

ladvoc · 2026-06-11T16:06:34Z

+use std::io;
+
+/// Opaque handle to an Argus capture session.
+pub struct ArgusCaptureSession {


note: While this is probably fine for an example, C/C++ bindings typically live in their own -sys crates (e.g., webrtc-sys, argus-sys) and then provide a higher level Rust wrapper that hides any FFI types. Also outside the scope of this PR, but I do see value in eventually shipping a dedicated crate for this and updating the example to use it.

good call out, I've created a new project to create new livekit-capture crate and migrate all capture related code there. Will refactor examples once we have that.

ladvoc · 2026-06-11T16:08:27Z

+    /// Captures a Jetson DMA-buffer backed video frame.
+    ///
+    /// `pixel_format` is `0` for NV12 and `1` for YUV420M.
+    pub fn capture_dmabuf_frame(


suggestion: This and the following method should only be compiled for Linux.

alan-george-lk · 2026-06-11T16:15:28Z

+unsafe impl Send for ArgusCaptureSession {}
+
+extern "C" {
+    fn lk_argus_create_session(


This example is cool--if I'm tracking the pieces right, couldn't this code be used by consumers in library form (vs example runnable form) to do Argus frame capture? Or is the expectation that user pipelines will do their own Argus interfacing, this is just an example of how it connects with LiveKit?

Or am I not following right?

yep, we plan on creating a dedicated capture crate that contains all the different platform capture paths. For now this is just inline in the example app so it can run on a jetson. I would expect customers to do their own integration, and they will probably use C++ SDK if they were using libargus anyways.

alan-george-lk · 2026-06-11T16:32:26Z

+      int scaled_height) override;
+
+  // DMA buffer accessors
+  int dmabuf_fd() const { return dmabuf_fd_; }


Nit: in the C++ SDK we aligned all methods to camelBack() case, since this isn't in the SDK not a biggie just for your awareness. If the webrtc stuff does snake_case for base accessors and PascalCase for operations (which it appears to from the override lines above), nothing to change such that it's consistent.

alan-george-lk · 2026-06-11T16:39:06Z

+                              const ::webrtc::VideoFrame& input_frame,
+                              bool is_keyframe);
+
+  const webrtc::Environment& env_;


Minor: I am generally avoidant of inheritance (esp double inheritance, given this already extends VideoEncoder, but there seems to be a lot of shared state between these _impl classes.

Consider a JetsonVideoEncoder class that extends VideoEncoder which these then extend, or for composition instead, a helper object that defines all the common variables/methods and then is instantiated in each of these. One layer of indirection, but less copy/pasta

alan-george-lk · 2026-06-11T16:41:41Z

+ public:
+  struct LayerConfig {
+    int simulcast_idx = 0;
+    int width = -1;


Nit: std::optional is more modern/clear than default negative values, but this certainly works given the context of the data, and is more lightweight/C-like

alan-george-lk · 2026-06-11T17:05:38Z

+      (!output_is_nv12_ && buffer->n_planes > 2) &&
+      GetPitchAndHeightFromNvBufSurfaceFd(buffer->planes[2].fd, 2, &v_pitch, &v_h, &v_np);
+
+  auto stride_from_plane = [&](int plane_index,


I wonder if this lambda (and similar sections of this method) could be written in a way that don't depend on HW state, such that they could be unit tested

refactored into a new file, added unit tests

alan-george-lk · 2026-06-11T17:12:17Z

+                let jetson_mmapi_include = PathBuf::from("/usr/src/jetson_multimedia_api/include");
+                if jetson_mmapi_include.exists() {
+                    let jetson_classes_dir =
+                        PathBuf::from("/usr/src/jetson_multimedia_api/samples/common/classes");


Should we check if this path is valid as well?

hmm, I wonder if we should just copy the files over and include them in our repo like we do w/ the nvenc related files

@cloudwebrtc any opinions? I see for nvenc we added NvEncoder.x NvDecoder.x into the repo.

…dks into dc/exp/jetson_encode_0527

chenosaurus and others added 21 commits May 23, 2026 11:42

improving timing rendering

ae8760d

make latency display faster

a94cd40

improve formatting of overlay

ff77f24

add --codec flag

e19bcbd

display codec & encoder/decoder in overlay

7108520

codec display

5f654e9

ensure we only capture timing when observers are attached

7b6a84f

gate timing capture on observers

75f8f41

Merge branch 'dc/exp/video_timing' of github.com:livekit/rust-sdks in…

9cda94e

…to dc/exp/video_timing

clean up timing overlay consistency between publisher/subscriber

e251478

Reexport timing events, use an async stream for timing event broadcasts

8ac46e1

porting old argus capture & jetson encoder

742e121

Co-authored-by: Cursor <cursoragent@cursor.com>

fix

1f76c40

Co-authored-by: Cursor <cursoragent@cursor.com>

only recreate room if playout delay is set

4c961c0

Co-authored-by: Cursor <cursoragent@cursor.com>

fix nvdec use on jetson

784c52a

Co-authored-by: Cursor <cursoragent@cursor.com>

add --format to publisher

f022fbf

Co-authored-by: Cursor <cursoragent@cursor.com>

fix video rendering on linux

3f943dd

Co-authored-by: Cursor <cursoragent@cursor.com>

document jetson video publishing

40cbc78

Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'main' into dc/exp/jetson_encode_0527

0a43464

add av1 encoder support

625b8d5

Merge branch 'main' into dc/exp/jetson_encode_0527

29520ec

chenosaurus requested review from cloudwebrtc and ladvoc as code owners June 11, 2026 06:22

github-actions Bot and others added 6 commits June 11, 2026 06:24

generated protobuf

d01c821

add license

d7aaeb7

Merge branch 'dc/exp/jetson_encode_0527' of github.com:livekit/rust-s…

db85737

…dks into dc/exp/jetson_encode_0527

Merge branch 'dc/exp/jetson_encode_av1' into dc/exp/jetson_encode_0527

5511fdb

update changeset msg

522a332

remove dupe changeset

d56a74c

chenosaurus and others added 2 commits June 10, 2026 23:37

revert protocl

f141d72

generated protobuf

cc73a86

ladvoc approved these changes Jun 11, 2026

View reviewed changes

alan-george-lk approved these changes Jun 11, 2026

View reviewed changes

chenosaurus added 10 commits June 11, 2026 12:14

use workspace dep for cc

b8d9c49

Merge branch 'dc/exp/jetson_encode_0527' of github.com:livekit/rust-s…

bb53785

…dks into dc/exp/jetson_encode_0527

move argus helper functions into argus module

e607973

only compile for linux

ed8c35d

clean up & remove debug code

60ac10d

remove timing debug

4f5fda7

enum as uint8_t

57f8b58

cleanup & refactor

2a04801

fix build

2682746

add copyright

31790ac

chenosaurus merged commit 527aba5 into main Jun 12, 2026
23 checks passed

chenosaurus deleted the dc/exp/jetson_encode_0527 branch June 12, 2026 19:12

Uh oh!

Conversation

chenosaurus commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Changeset

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants