Commit 41faf96
authored
Enable LiteRT export tests on the torch backend (keras-team#2698)
* Enable LiteRT export tests for the TensorFlow backend
The LiteRT export tests were globally skipped (TODO keras-team#2572) because the
legacy `tf.lite.Interpreter` is deprecated/removed in recent TensorFlow
releases. Re-enable them for the TensorFlow backend on top of the
rewritten Keras 3.15 export path (ExportArchive -> SavedModel ->
`tf.lite.TFLiteConverter.from_saved_model`):
- `run_litert_export_test`: remove the unconditional skip, gate on
keras >= 3.15, and use the ai-edge-litert interpreter exclusively
(no `tf.lite.Interpreter` fallback).
- The exporter enables `SELECT_TF_OPS`, so detect FLEX ops in the
converted `.tflite` via the flatbuffer schema and verify numerics
only when the model is FLEX-free (ai-edge-litert cannot execute FLEX
ops). Export and signature structure are still validated either way.
- Re-enable LiteRT tests for models that now pass: VGG, Moonshine, SAM,
StableDiffusion3, GPTNeoX.
- Keep BASNet, DeepLabV3 and SAM3 skipped, with accurate TODO reasons
describing the real TFLite conversion failures.
Tested with keras 3.15 + tensorflow 2.20 + ai-edge-litert:
68 passed, 5 skipped, 0 failed.
* Harden FLEX-op flatbuffer parsing (review feedback)
Address Gemini review: defensively handle `memoryview` buffers, decode
custom op codes with `errors="replace"`, and guard `.startswith()` with
an `isinstance(str)` check in `_litert_flex_ops`.
* Require keras>=3.15 for the LiteRT export tests
The LiteRT export path (TensorFlow backend, and the torch backend via
litert-torch) was rewritten in Keras 3.15, and run_litert_export_test skips
on older Keras. Bump the declared keras dependency from >=3.13 to >=3.15 so
the rewritten export API is available and the tests run instead of silently
skipping.
* Add ai-edge-litert dependency for the LiteRT export tests
The LiteRT export tests require the ai-edge-litert interpreter. Without it
the tests skip ("requires the 'ai-edge-litert' package"), so on CI they
were silently skipped on the TensorFlow backend. Add it as a dependency so
the tests actually run.
* Enable LiteRT export tests on the torch backend
Builds on the TensorFlow enablement to also run the LiteRT export tests on
the PyTorch backend via the Keras 3.15 torch export path (litert-torch).
- test_case.py: run_litert_export_test supports the torch backend. It builds
a concrete input signature (torch export has no dynamic shapes), gates on
the optional litert-torch package, maps litert-torch's signature inputs
back to the original keys (handles both positional `args_N` and
`args_N_<key>` naming across litert-torch versions), and verifies through
the ai-edge-litert interpreter.
- requirements.txt: add litert-torch.
- Add LiteRT export tests for d_fine, f_net, flux, gpt_oss, segformer, vae,
vit_det and whisper.
- xfail(torch) for models with documented upstream torch.export / litert-torch
limitations (d_fine, f_net, flux, gpt_oss, sam3, vae, stable_diffusion_3,
segformer, and the MoE models mixtral, qwen3_moe, qwen_moe which emit
aten._assert_async); enable on torch where it already works.
Model source fixes for the xfailed models are out of scope here and tracked
in separate PRs.
* test(litert): gate export tests to the torch backend; disable TF backend
The per-model test_litert_export tests currently run on both the torch
(litert-torch) and TensorFlow (ExportArchive -> SavedModel ->
tf.lite.TFLiteConverter) backends. The TensorFlow path OOM-kills CI: each
convert() leaks memory (tensorflow/tensorflow#122598), and converting the
whole model zoo in one process exhausts the runner.
Skip these tests on the TensorFlow backend for now so this torch-backend
work is independent of the broken TF converter path. The torch backend is
unaffected and its tests continue to run. Re-enable TensorFlow once the
upstream leak is fixed.
* Simplify LiteRT test utilities using keras ops and standardize_dtype
Replace manual tensor->numpy and dtype conversion chains in:
- _build_litert_torch_input_signature: use ops.convert_to_numpy + standardize_dtype
- convert_for_tflite: use ops.convert_to_numpy + standardize_dtype + dtype_map
Reduces duplication and unifies on existing keras APIs.
* ci: re-trigger checks after upstream PyTorch CDN SSL issue
* ci: trigger workflow run
* ci: remove trigger file
* ci: re-trigger after upstream CDN retry
* Restore TensorFlow LiteRT test skips to match master branch and format API imports
* Combine LiteRT export check and simplify backend requirement
* Remove TensorFlow/FLEX-specific code from the torch-only LiteRT export path
TensorFlow-backend LiteRT export is out of scope for this PR: it's blocked
by an upstream tf.lite.TFLiteConverter memory leak (OOM-kills CI when
converting many models in one process, tensorflow/tensorflow#122598) with
no code-side fix available. This PR only ever runs on the torch backend
(gated by run_litert_export_test's existing keras>=3.15 + backend=="torch"
check), so remove the TF-only code that was carried over from the
TensorFlow-enablement PR:
- Delete _litert_flex_ops and its call site (FLEX/SelectTf op detection,
the run_inference/verify_numerics gating dance, and the early return
when FLEX ops are present). FLEX ops only ever came from the TF
ExportArchive -> SavedModel -> TFLiteConverter path with SELECT_TF_OPS;
litert-torch export never produces them.
- Collapse three "if backend == torch: ... else: ..." branches (the
input_signature gate, SignatureDef input verification, and interpreter
runner-kwargs mapping) to their torch-only bodies, since backend is
always torch by the time this code runs.
- Drop the TF-only allow_custom_ops/enable_select_tf_ops mention from the
run_litert_export_test docstring.
- basnet_test.py, sam3_pc_image_segmenter_test.py,
stable_diffusion_3_text_to_image_test.py: remove redundant
TensorFlow-backend skip guards and the tf_only_kwargs conditional
(run_litert_export_test's own gate already skips non-torch backends).
* Address remaining review feedback on LiteRT torch export tests
- Drop the redundant keras>=3.15 runtime check in run_litert_export_test:
pyproject.toml already pins keras>=3.15, so the version guard can never
fail and only obscured the actual backend check.
- Remove strict=False from all torch-backend xfail markers so these tests
start failing loudly (instead of silently passing) once the underlying
litert-torch/torch.export limitation is fixed.
- Shorten the xfail reason strings to a single line each.
The litert-torch/ai-edge-litert import-skip guards in
run_litert_export_test are intentionally kept: removing them would mask a
failed package install inside environments where these optional
dependencies aren't available.1 parent 4136e38 commit 41faf96
22 files changed
Lines changed: 175 additions & 67 deletions
File tree
- keras_hub/src
- models
- basnet
- d_fine
- deeplab_v3
- f_net
- flux
- gpt_neo_x
- gpt_oss
- mixtral
- moonshine
- qwen3_moe
- qwen_moe
- sam3
- sam
- segformer
- stable_diffusion_3
- vae
- vgg
- vit_det
- whisper
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
57 | | - | |
58 | 57 | | |
59 | 58 | | |
60 | 59 | | |
| |||
Lines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
177 | 181 | | |
178 | 182 | | |
179 | 183 | | |
| |||
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
| 75 | + | |
| 76 | + | |
76 | 77 | | |
77 | 78 | | |
78 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
57 | 58 | | |
58 | 59 | | |
59 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
60 | 65 | | |
61 | 66 | | |
62 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
| |||
84 | 85 | | |
85 | 86 | | |
86 | 87 | | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
87 | 92 | | |
88 | 93 | | |
89 | 94 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
115 | | - | |
116 | 115 | | |
117 | 116 | | |
118 | 117 | | |
119 | 118 | | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
| 119 | + | |
124 | 120 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
113 | 114 | | |
114 | 115 | | |
115 | 116 | | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
116 | 121 | | |
117 | 122 | | |
118 | 123 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
107 | 108 | | |
108 | 109 | | |
109 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
110 | 115 | | |
111 | 116 | | |
112 | 117 | | |
| |||
Lines changed: 0 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | 148 | | |
152 | 149 | | |
153 | 150 | | |
| |||
Lines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
124 | 125 | | |
125 | 126 | | |
126 | 127 | | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
127 | 132 | | |
128 | 133 | | |
129 | 134 | | |
| |||
0 commit comments