Skip to content

Commit f86bcaa

Browse files
mykola-mokhnachmykola-mokhnach
authored
docs: Add MJPEG guide (#987)
1 parent 8da3da0 commit f86bcaa

2 files changed

Lines changed: 117 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -154,10 +154,12 @@ appium:unlockSuccessTimeout | Maximum number of milliseconds to wait until the d
154154

155155
### MJPEG
156156

157+
Read the [MJPEG guide](docs/mjpeg.md) for capabilities, settings, and usage details.
158+
157159
Capability Name | Description
158160
--- | ---
159-
appium:mjpegServerPort | The number of the port on the host machine that UiAutomator2 server starts the MJPEG server on. If not provided then the screenshots broadcasting service on the remote device does not get exposed to a local port (e.g. no adb port forwarding is happening)
160-
appium:mjpegScreenshotUrl | The URL of a service that provides realtime device screenshots in MJPEG format. If provided then the actual command to retrieve a screenshot will be requesting pictures from this service rather than directly from the server
161+
appium:mjpegServerPort | The number of the port on the host machine to which the device MJPEG server port is forwarded. If not provided then the screenshots broadcasting service on the remote device does not get exposed to a local port (e.g. no adb port forwarding is happening).
162+
appium:mjpegScreenshotUrl | The URL of a service that provides realtime device screenshots in MJPEG format. If provided then the actual command to retrieve a screenshot will be requesting pictures from this service rather than directly from the server.
161163

162164
### Web Context
163165

@@ -327,7 +329,7 @@ mjpegServerPort | int | The number of the port on the remote device to start MJP
327329
mjpegServerFramerate | int | The maximum count of screenshots per second taken by the MJPEG screenshots broadcaster. Must be in range 1..60. `10` by default
328330
mjpegScalingFactor | int | The percentage value used to apply downscaling on the screenshots generated by the MJPEG screenshots broadcaster. Must be in range 1..100. `50` is by default, which means that screenshots are downscaled to the half of their original size keeping their original proportions.
329331
mjpegServerScreenshotQuality | int | The percentage value used to apply lossy JPEG compression on the screenshots generated by the MJPEG screenshots broadcaster. Must be in range 1..100. `50` is by default, which means that screenshots are compressed to the half of their original quality.
330-
mjpegBilinearFiltering | boolean | Controls whether (`true`) or not (`false`, the default value) to apply bilinear filtering to MJPEG screenshots broadcaster resize algorithm. Enabling this flag may improve the quality of the resulting scaled bitmap, but may introduce a small performance hit.
332+
mjpegBilinearFiltering | boolean | Controls whether (`true`) or not (`false`, the default value) to apply bilinear filtering to MJPEG screenshots broadcaster resize algorithm. Enabling this flag may improve the quality of the resulting scaled bitmap, but may introduce a small performance hit. See the [MJPEG guide](docs/mjpeg.md) for details on all MJPEG settings.
331333
useResourcesForOrientationDetection | boolean | Defines the strategy used by UiAutomator2 server to detect the original device orientation. By default (`false` value) the server uses device rotation value for this purpose. Although, this approach may not work for some devices and a portrait orientation may erroneously be detected as the landscape one (and vice versa). In such case it makes sense to play with this setting.
332334
enforceXPath1 | boolean | Since UiAutomator2 driver version `4.25.0` XPath2 is set as the default and the recommended interpreter for the corresponding element locators. This interpreter is based on [Psychopath XPath2](https://wiki.eclipse.org/PsychoPathXPathProcessor) implementation, which is now a part of the Eclipse foundation. In most of the cases XPath1 locators are also valid XPath2 locators, so there should be no issues while locating elements. Although, since the XPath2 standard is much more advanced in comparison to the previous version, some [issues](https://github.com/appium/appium/issues/16142) are possible for more sophisticated locators, which cannot be fixed easily, as we depend on the third-party library mentioned above. Then try to workaround such issues by enforcing XPath1 usage (whose implementation is a part of the Android platform itself) and assigning this setting to `true`. Note, this setting is actually applied at the time when the element lookup by XPath is executed, so you could switch it on or off whenever needed throughout your automated testing session.
333335
limitXPathContextScope | boolean | Due to historical reasons UiAutomator2 driver limits scopes of element context-based searches to the parent element. This means a request like `findElement(By.xpath, "//root").findElement(By.xpath, "./..")` would always fail, because the driver only collects descendants of the `root` element for the destination XML source. The `limitXPathContextScope` setting being set to `false` changes that default behavior, so the collected page source includes the whole page source XML where `root` node is set as the search context. With that setting disabled the search query above should not fail anymore. Although, you must still be careful while building XPath requests for context-based searches with the `limitXPathContextScope` setting set to `false`. A request like `findElement(By.xpath, "//root").findElement(By.xpath, "//element")` would ignore the current context and search for `element` trough the whole page source. Use `.` notation to correct that behavior and only find `element` nodes which are descendants of the `root` node: `findElement(By.xpath, "//root").findElement(By.xpath, ".//element")`.
@@ -685,7 +687,7 @@ action | string | yes | The name or an integer code of the editor action to be e
685687

686688
### mobile: startScreenStreaming
687689

688-
Starts device screen broadcast by creating MJPEG server. Multiple calls to this method have no effect unless the previous streaming session is stopped. This method only works if the `adb_screen_streaming` feature is enabled on the server side. It is also required that [GStreamer](https://gstreamer.freedesktop.org/) with `gst-plugins-base`, `gst-plugins-good` and `gst-plugins-bad` packages are installed and available in PATH on the server machine.
690+
Starts device screen broadcast by creating MJPEG server. Multiple calls to this method have no effect unless the previous streaming session is stopped. This method only works if the `adb_screen_streaming` feature is enabled on the server side. It is also required that [GStreamer](https://gstreamer.freedesktop.org/) with `gst-plugins-base`, `gst-plugins-good` and `gst-plugins-bad` packages are installed and available in PATH on the server machine. For the built-in device MJPEG stream (no GStreamer on host), see the [MJPEG guide](docs/mjpeg.md).
689691

690692
#### Arguments
691693

‎docs/mjpeg.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# MJPEG Screen Streaming
2+
3+
This guide describes the MJPEG (Motion JPEG) screen streaming support in the UiAutomator2 driver.
4+
5+
## Overview
6+
7+
The UiAutomator2 **server** (running on the Android device) includes a built-in MJPEG broadcaster. It captures the device screen at a configurable rate, encodes frames as JPEG, and serves them over TCP using the `multipart/x-mixed-replace` format. This allows:
8+
9+
- **Live viewing** of the device screen by connecting to the stream (e.g. in a browser or a custom client).
10+
- **Screenshot from stream**: if the driver is configured to use the MJPEG URL, the standard screenshot command returns the latest frame from the stream instead of calling the server’s screenshot API.
11+
12+
The MJPEG server on the device is started automatically when the UiAutomator2 instrumentation starts and listens on a fixed port on the device (default `7810`).
13+
14+
## Use Cases
15+
16+
| Use case | Description |
17+
|----------|-------------|
18+
| **Screen recording** | The UiAutomator2 driver can use the MJPEG stream as input for host-side screen recording tooling (for example, via `ffmpeg`) to produce MP4 (or other) video files. |
19+
| **Screenshots from stream** | If you set the `appium:mjpegScreenshotUrl` capability, the driver uses the **latest frame** from that MJPEG stream when you call the screenshot command instead of calling the server’s regular screenshot API. |
20+
| **Live viewing / custom tools** | Any HTTP client (browser, script, or custom app) can connect to the MJPEG URL (after port forwarding, if needed) to view or process the live screen. |
21+
22+
## Configuration
23+
24+
### Capabilities
25+
26+
#### `appium:mjpegServerPort`
27+
28+
- **Type:** integer (port number on the **host**)
29+
- **Description:** The port on the host machine to which the device MJPEG server port is forwarded via ADB. If not set, no port forwarding is done, so the device MJPEG service is not reachable from the host.
30+
- **Use case:** Set this when you want to connect to the device MJPEG stream from the host (e.g. `http://localhost:<mjpegServerPort>`) or when using screen recording so the host can receive the stream.
31+
- **Parallel sessions:** Use a **unique** `mjpegServerPort` per session when running multiple devices or parallel tests to avoid port conflicts.
32+
33+
#### `appium:mjpegScreenshotUrl`
34+
35+
- **Type:** string (URL)
36+
- **Description:** URL of a service that provides real-time device screenshots in MJPEG format. If set, the driver creates an MJPEG client that consumes this URL; the standard **screenshot** command then returns the latest frame from that stream instead of calling the UiAutomator2 server’s `/screenshot` endpoint.
37+
- **Typical value:** When you forward the device MJPEG port with `appium:mjpegServerPort`, you can set this to `http://localhost:<mjpegServerPort>` (or the same host/port your client uses to reach the stream) so that screenshot uses the MJPEG stream.
38+
- **Note:** The driver starts the MJPEG stream client after the session is created and stops it on session end. If the stream has not yet produced a frame, the screenshot command may fall back to the regular server screenshot.
39+
40+
### Settings (device-side MJPEG broadcaster)
41+
42+
These settings control the MJPEG broadcaster that runs **on the device** (in the UiAutomator2 server). They can be changed at runtime via the [Settings API](https://appium.io/docs/en/latest/guides/settings/) (e.g. `driver.update_settings({...})`).
43+
44+
| Setting | Type | Range / values | Default | Description |
45+
|--------|------|----------------|---------|-------------|
46+
| `mjpegServerPort` | int | 1024..65535 | 7810 | Port on the **device** where the MJPEG server listens. The driver forwards this to the host when `appium:mjpegServerPort` capability is set. |
47+
| `mjpegServerFramerate` | int | 1..60 | 10 | Maximum frames per second. Higher values increase CPU load on the device. |
48+
| `mjpegScalingFactor` | int | 1..100 | 50 | Scale factor in percent (100 = no scaling). Lower values reduce size and CPU use. |
49+
| `mjpegServerScreenshotQuality` | int | 1..100 | 50 | JPEG quality (100 = best). Higher values improve quality but use more CPU. |
50+
| `mjpegBilinearFiltering` | boolean | true / false | false | Use bilinear filtering when scaling. Can improve scaled image quality with a small performance cost. |
51+
52+
## How it Works
53+
54+
1. When the UiAutomator2 server starts on the device, it starts an **MJPEG server** thread that listens on the configured device port (default 7810).
55+
2. For each TCP client that connects, the server sends an HTTP-like response with `Content-Type: multipart/x-mixed-replace; boundary=--BoundaryString` and then a continuous stream of JPEG frames (each frame preceded by a boundary and `Content-Length`).
56+
3. Frames are produced by the device at the configured framerate; each frame is taken via `UiAutomation.takeScreenshot()`, optionally scaled and compressed according to the settings above, then sent to all connected clients.
57+
4. If no clients are connected, the server does not capture at full rate (it sleeps) to save CPU.
58+
59+
## Using the Stream
60+
61+
### Port forwarding and connecting
62+
63+
1. Set the capability **`appium:mjpegServerPort`** to a free port on the host (e.g. `7810` or any port you choose).
64+
2. The driver will run `adb forward <host port> 7810` (or the device port you set via the `mjpegServerPort` setting) so that the device MJPEG server is reachable at `http://localhost:<host port>` (or your host IP if you need remote access).
65+
3. Open that URL in a browser or use any MJPEG-capable client; you should see the live device screen.
66+
4. Optionally set **`appium:mjpegScreenshotUrl`** to the same URL (e.g. `http://localhost:7810`) so that the driver’s screenshot command uses the latest frame from this stream.
67+
68+
### Screenshot behavior
69+
70+
- **Without `appium:mjpegScreenshotUrl`:** The screenshot command is sent to the UiAutomator2 server (`GET /screenshot`), which takes a single screenshot on demand.
71+
- **With `appium:mjpegScreenshotUrl`:** The driver keeps an MJPEG client connected to the given URL. The screenshot command returns the latest frame received from that stream (as base64 PNG). If no frame is available yet, the driver falls back to the regular server screenshot and logs a warning.
72+
73+
## Parallel Testing
74+
75+
When running multiple sessions (e.g. parallel tests or multiple devices):
76+
77+
- Set a **unique** `appium:mjpegServerPort` per session so each device’s MJPEG port is forwarded to a different host port.
78+
- This is especially important if you use screen recording or any tool that connects to the MJPEG stream.
79+
80+
## Performance Considerations
81+
82+
- Higher `mjpegServerFramerate` values increase the number of screenshots taken per second and therefore the CPU load.
83+
- Higher `mjpegServerScreenshotQuality` values increase JPEG encoding cost and network bandwidth.
84+
- Higher `mjpegScalingFactor` values (larger images) also increase encoding time and payload size.
85+
- Enabling `mjpegBilinearFiltering` can improve scaled image quality with a small extra performance cost.
86+
87+
For long-running or parallel streaming, start with moderate defaults (for example, framerate `10`, scaling factor `50`, quality `50`) and then adjust as needed.
88+
89+
## Relationship to `mobile: startScreenStreaming` / `stopScreenStreaming`
90+
91+
The driver also supports **`mobile: startScreenStreaming`** and **`mobile: stopScreenStreaming`**. These are **different** from the built-in device MJPEG server:
92+
93+
- They start or stop a **host-side** MJPEG server that uses **GStreamer** (and optionally ADB) to capture and broadcast the device screen. They require the `adb_screen_streaming` feature and GStreamer with `gst-plugins-base`, `gst-plugins-good`, and `gst-plugins-bad` on the **host**.
94+
- The **device MJPEG server** described in this guide runs entirely on the device, is built into the UiAutomator2 server, and does not require GStreamer.
95+
96+
Use the **device MJPEG server** (with `appium:mjpegServerPort` and optionally `appium:mjpegScreenshotUrl`) when you only need a simple TCP MJPEG stream from the device. Use **`mobile: startScreenStreaming`** when you need the host-based GStreamer pipeline (e.g. for specific encoding or network options).
97+
98+
## Requirements
99+
100+
- For **device-side MJPEG streaming** itself, no extra tools are required beyond a compatible Android device and the UiAutomator2 server; the broadcaster runs entirely on the device.
101+
- For **tools that consume the stream** (for example, screen recording on the host), you must ensure whatever tool you use (such as `ffmpeg` or a custom recorder) understands MJPEG over HTTP and can connect to the forwarded MJPEG URL.
102+
- For **`mobile: startScreenStreaming` / `stopScreenStreaming`**, which are host-side and GStreamer-based, you must have GStreamer with `gst-plugins-base`, `gst-plugins-good`, and `gst-plugins-bad` installed and available on the host `PATH`.
103+
104+
## Summary
105+
106+
| What you want | What to use |
107+
|---------------|-------------|
108+
| Expose device screen as MJPEG on the host | Set `appium:mjpegServerPort` to a host port; connect to `http://localhost:<port>`. |
109+
| Screenshot from the MJPEG stream | Also set `appium:mjpegScreenshotUrl` to that URL (e.g. `http://localhost:<port>`). |
110+
| Tune framerate, size, or quality on the device | Use Settings API: `mjpegServerFramerate`, `mjpegScalingFactor`, `mjpegServerScreenshotQuality`, `mjpegBilinearFiltering`. |
111+
| Host-side GStreamer-based streaming | Use `mobile: startScreenStreaming` / `mobile: stopScreenStreaming` (see README). |

0 commit comments

Comments
 (0)