Skip to content

Commit 41faf96

Browse files
authored
Enable LiteRT export tests on the torch backend (keras-team#2698)
* Enable LiteRT export tests for the TensorFlow backend The LiteRT export tests were globally skipped (TODO keras-team#2572) because the legacy `tf.lite.Interpreter` is deprecated/removed in recent TensorFlow releases. Re-enable them for the TensorFlow backend on top of the rewritten Keras 3.15 export path (ExportArchive -> SavedModel -> `tf.lite.TFLiteConverter.from_saved_model`): - `run_litert_export_test`: remove the unconditional skip, gate on keras >= 3.15, and use the ai-edge-litert interpreter exclusively (no `tf.lite.Interpreter` fallback). - The exporter enables `SELECT_TF_OPS`, so detect FLEX ops in the converted `.tflite` via the flatbuffer schema and verify numerics only when the model is FLEX-free (ai-edge-litert cannot execute FLEX ops). Export and signature structure are still validated either way. - Re-enable LiteRT tests for models that now pass: VGG, Moonshine, SAM, StableDiffusion3, GPTNeoX. - Keep BASNet, DeepLabV3 and SAM3 skipped, with accurate TODO reasons describing the real TFLite conversion failures. Tested with keras 3.15 + tensorflow 2.20 + ai-edge-litert: 68 passed, 5 skipped, 0 failed. * Harden FLEX-op flatbuffer parsing (review feedback) Address Gemini review: defensively handle `memoryview` buffers, decode custom op codes with `errors="replace"`, and guard `.startswith()` with an `isinstance(str)` check in `_litert_flex_ops`. * Require keras>=3.15 for the LiteRT export tests The LiteRT export path (TensorFlow backend, and the torch backend via litert-torch) was rewritten in Keras 3.15, and run_litert_export_test skips on older Keras. Bump the declared keras dependency from >=3.13 to >=3.15 so the rewritten export API is available and the tests run instead of silently skipping. * Add ai-edge-litert dependency for the LiteRT export tests The LiteRT export tests require the ai-edge-litert interpreter. Without it the tests skip ("requires the 'ai-edge-litert' package"), so on CI they were silently skipped on the TensorFlow backend. Add it as a dependency so the tests actually run. * Enable LiteRT export tests on the torch backend Builds on the TensorFlow enablement to also run the LiteRT export tests on the PyTorch backend via the Keras 3.15 torch export path (litert-torch). - test_case.py: run_litert_export_test supports the torch backend. It builds a concrete input signature (torch export has no dynamic shapes), gates on the optional litert-torch package, maps litert-torch's signature inputs back to the original keys (handles both positional `args_N` and `args_N_<key>` naming across litert-torch versions), and verifies through the ai-edge-litert interpreter. - requirements.txt: add litert-torch. - Add LiteRT export tests for d_fine, f_net, flux, gpt_oss, segformer, vae, vit_det and whisper. - xfail(torch) for models with documented upstream torch.export / litert-torch limitations (d_fine, f_net, flux, gpt_oss, sam3, vae, stable_diffusion_3, segformer, and the MoE models mixtral, qwen3_moe, qwen_moe which emit aten._assert_async); enable on torch where it already works. Model source fixes for the xfailed models are out of scope here and tracked in separate PRs. * test(litert): gate export tests to the torch backend; disable TF backend The per-model test_litert_export tests currently run on both the torch (litert-torch) and TensorFlow (ExportArchive -> SavedModel -> tf.lite.TFLiteConverter) backends. The TensorFlow path OOM-kills CI: each convert() leaks memory (tensorflow/tensorflow#122598), and converting the whole model zoo in one process exhausts the runner. Skip these tests on the TensorFlow backend for now so this torch-backend work is independent of the broken TF converter path. The torch backend is unaffected and its tests continue to run. Re-enable TensorFlow once the upstream leak is fixed. * Simplify LiteRT test utilities using keras ops and standardize_dtype Replace manual tensor->numpy and dtype conversion chains in: - _build_litert_torch_input_signature: use ops.convert_to_numpy + standardize_dtype - convert_for_tflite: use ops.convert_to_numpy + standardize_dtype + dtype_map Reduces duplication and unifies on existing keras APIs. * ci: re-trigger checks after upstream PyTorch CDN SSL issue * ci: trigger workflow run * ci: remove trigger file * ci: re-trigger after upstream CDN retry * Restore TensorFlow LiteRT test skips to match master branch and format API imports * Combine LiteRT export check and simplify backend requirement * Remove TensorFlow/FLEX-specific code from the torch-only LiteRT export path TensorFlow-backend LiteRT export is out of scope for this PR: it's blocked by an upstream tf.lite.TFLiteConverter memory leak (OOM-kills CI when converting many models in one process, tensorflow/tensorflow#122598) with no code-side fix available. This PR only ever runs on the torch backend (gated by run_litert_export_test's existing keras>=3.15 + backend=="torch" check), so remove the TF-only code that was carried over from the TensorFlow-enablement PR: - Delete _litert_flex_ops and its call site (FLEX/SelectTf op detection, the run_inference/verify_numerics gating dance, and the early return when FLEX ops are present). FLEX ops only ever came from the TF ExportArchive -> SavedModel -> TFLiteConverter path with SELECT_TF_OPS; litert-torch export never produces them. - Collapse three "if backend == torch: ... else: ..." branches (the input_signature gate, SignatureDef input verification, and interpreter runner-kwargs mapping) to their torch-only bodies, since backend is always torch by the time this code runs. - Drop the TF-only allow_custom_ops/enable_select_tf_ops mention from the run_litert_export_test docstring. - basnet_test.py, sam3_pc_image_segmenter_test.py, stable_diffusion_3_text_to_image_test.py: remove redundant TensorFlow-backend skip guards and the tf_only_kwargs conditional (run_litert_export_test's own gate already skips non-torch backends). * Address remaining review feedback on LiteRT torch export tests - Drop the redundant keras>=3.15 runtime check in run_litert_export_test: pyproject.toml already pins keras>=3.15, so the version guard can never fail and only obscured the actual backend check. - Remove strict=False from all torch-backend xfail markers so these tests start failing loudly (instead of silently passing) once the underlying litert-torch/torch.export limitation is fixed. - Shorten the xfail reason strings to a single line each. The litert-torch/ai-edge-litert import-skip guards in run_litert_export_test are intentionally kept: removing them would mask a failed package install inside environments where these optional dependencies aren't available.
1 parent 4136e38 commit 41faf96

22 files changed

Lines changed: 175 additions & 67 deletions

keras_hub/src/models/basnet/basnet_test.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,6 @@ def test_saved_model(self):
5454
input_data=self.images,
5555
)
5656

57-
@pytest.mark.skip(reason="TODO: Bug with BASNet liteRT export")
5857
def test_litert_export(self):
5958
self.run_litert_export_test(
6059
cls=BASNetImageSegmenter,

keras_hub/src/models/d_fine/d_fine_object_detector_test.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,10 @@ def test_saved_model(self):
174174
input_data=self.images,
175175
)
176176

177+
@pytest.mark.xfail(
178+
condition=keras.backend.backend() == "torch",
179+
reason="D-FINE's multi-scale features hit a torch.export shape guard.",
180+
)
177181
def test_litert_export(self):
178182
backbone = DFineBackbone(**self.base_backbone_kwargs)
179183
init_kwargs = {

keras_hub/src/models/deeplab_v3/deeplab_v3_segmenter_test.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,8 @@ def test_saved_model(self):
7272
)
7373

7474
@pytest.mark.skip(
75-
reason="TODO: Bug with DeepLabV3ImageSegmenter liteRT export"
75+
reason="TODO: DeepLabV3ImageSegmenter LiteRT export fails with "
76+
"'symbolic tf.Tensor used as a Python bool' while tracing for export."
7677
)
7778
def test_litert_export(self):
7879
self.run_litert_export_test(

keras_hub/src/models/f_net/f_net_text_classifier_test.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import os
22

33
import pytest
4+
from keras import backend
45

56
from keras_hub.src.models.f_net.f_net_backbone import FNetBackbone
67
from keras_hub.src.models.f_net.f_net_text_classifier import FNetTextClassifier
@@ -57,6 +58,10 @@ def test_saved_model(self):
5758
input_data=self.input_data,
5859
)
5960

61+
@pytest.mark.xfail(
62+
condition=backend.backend() == "torch",
63+
reason="litert-torch has no lowering for aten.complex (from ops.fft2).",
64+
)
6065
def test_litert_export(self):
6166
# F-Net does NOT use padding_mask - it only uses token_ids and
6267
# segment_ids. Don't add padding_mask to input_data.

keras_hub/src/models/flux/flux_backbone_test.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import pytest
2+
from keras import backend
23
from keras import ops
34

45
from keras_hub.src.models.clip.clip_text_encoder import CLIPTextEncoder
@@ -84,6 +85,10 @@ def test_saved_model(self):
8485
input_data=self.input_data,
8586
)
8687

88+
@pytest.mark.xfail(
89+
condition=backend.backend() == "torch",
90+
reason="torch.export guard from Flux's dynamic num_heads reshape.",
91+
)
8792
def test_litert_export(self):
8893
self.run_litert_export_test(
8994
cls=FluxBackbone,

keras_hub/src/models/gpt_neo_x/gpt_neo_x_causal_lm_test.py

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -112,13 +112,9 @@ def test_saved_model(self):
112112
)
113113

114114
def test_litert_export(self):
115-
pytest.skip(reason="TODO: Fix TFLite export bug for GPTNeoX")
116115
self.run_litert_export_test(
117116
cls=GPTNeoXCausalLM,
118117
init_kwargs=self.init_kwargs,
119118
input_data=self.input_data,
120-
output_thresholds={
121-
"max": 1e-3,
122-
"mean": 1e-4,
123-
}, # More lenient thresholds for numerical differences
119+
output_thresholds={"*": {"max": 1e-3, "mean": 1e-4}},
124120
)

keras_hub/src/models/gpt_oss/gpt_oss_causal_lm_test.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
from unittest.mock import patch
22

33
import pytest
4+
from keras import backend
45
from keras import ops
56

67
from keras_hub.src.models.gpt_oss.gpt_oss_backbone import GptOssBackbone
@@ -113,6 +114,10 @@ def test_saved_model(self):
113114
input_data=self.input_data,
114115
)
115116

117+
@pytest.mark.xfail(
118+
condition=backend.backend() == "torch",
119+
reason="litert-torch NHWC rewriter has no lowering for aten.amax.",
120+
)
116121
def test_litert_export(self):
117122
self.run_litert_export_test(
118123
cls=GptOssCausalLM,

keras_hub/src/models/mixtral/mixtral_causal_lm_test.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import os
22
from unittest.mock import patch
33

4+
import keras
45
import pytest
56
from keras import ops
67

@@ -107,6 +108,10 @@ def test_saved_model(self):
107108
input_data=self.input_data,
108109
)
109110

111+
@pytest.mark.xfail(
112+
condition=keras.backend.backend() == "torch",
113+
reason="litert-torch cannot lower aten._assert_async from MoE routing.",
114+
)
110115
def test_litert_export(self):
111116
self.run_litert_export_test(
112117
cls=MixtralCausalLM,

keras_hub/src/models/moonshine/moonshine_audio_to_text_test.py

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -145,9 +145,6 @@ def test_saved_model(self):
145145
input_data=self.input_data,
146146
)
147147

148-
@pytest.mark.skip(
149-
reason="TODO: Bug with MoonshineAudioToText liteRT export"
150-
)
151148
def test_litert_export(self):
152149
self.run_litert_export_test(
153150
cls=MoonshineAudioToText,

keras_hub/src/models/qwen3_moe/qwen3_moe_causal_lm_test.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
os.environ["KERAS_BACKEND"] = "jax"
55

6+
import keras
67
import pytest
78
from keras import ops
89

@@ -124,6 +125,10 @@ def test_saved_model(self):
124125
input_data=self.input_data,
125126
)
126127

128+
@pytest.mark.xfail(
129+
condition=keras.backend.backend() == "torch",
130+
reason="litert-torch cannot lower aten._assert_async from MoE routing.",
131+
)
127132
def test_litert_export(self):
128133
self.run_litert_export_test(
129134
cls=Qwen3MoeCausalLM,

0 commit comments

Comments
 (0)