Update testcase.py (keras-team#2512)

* Update gemma3_causal_lm_preprocessor.py

Added checks for invalid inputs

* Update gemma3_causal_lm_preprocessor.py

* Update gemma3_causal_lm_preprocessor_test.py

Added tests to check invalid inputs

* Update reversible_embedding.py

Fix for model not loading when using numpy behaviour with tensorflow

* upadated Gemma3InterleaveEmbeddings

* Update gemma3_interleave_embeddings.py

* Revert "Update reversible_embedding.py"

This reverts commit 3fdc7fd.

* Update gemma3_interleave_embeddings.py

* Update backbone.py

* Update backbone.py

* Update task.py

* Revert "Update task.py"

This reverts commit 62d2484.

* Revert "Update backbone.py"

This reverts commit de830b1.

* export

export working 1st commit

* refactoring

* refactor

* Update registry.py

* Refactor export logic and improve error handling

Refactored exporter and registry logic for better type safety and error handling. Improved input signature methods in config classes by extracting sequence length logic. Enhanced LiteRT exporter with clearer verbose handling and stricter error reporting. Registry now conditionally registers LiteRT exporter and extends export method only if dependencies are available.

* reformat

* Add export submodule to keras_hub API

Introduces the keras_hub.api.export submodule and updates the main API to expose it. The new export module imports various exporter configs and functions from the internal export package, making them available through the public API.

* reformat

* now supporting export for objectDetectors

* Add and refine image model exporter configs

Added ImageClassifierExporterConfig, ImageSegmenterExporterConfig, and ObjectDetectorExporterConfig to the export API. Improved input shape inference and dummy input generation for image-related exporter configs. Refactored LiteRTExporter to better handle model type checks and input signature logic, with improved error handling for input mapping.

* Refactor: move keras import to module level

Moved the 'import keras' statement to the top of the module and removed redundant local imports within class methods. This improves code clarity and avoids repeated imports.

* Remove debug_object_detection.py script

Deleted the debug_object_detection.py script, which was used for testing object detection model outputs and export issues. This cleanup removes unused debugging code from the repository.

* Rename LiteRT to Litert and update exporter configs

Renames all references of 'LiteRT' to 'Litert' across the codebase, including file names, class names, and function names. Updates exporter registry and API imports to use the new 'litert' naming. Also improves image model exporter configs to dynamically determine input dtype from the model, enhancing flexibility for different input types. Adds support for ImageSegmenter model type detection in the exporter registry.

* Refactor InputSpec formatting and fix import path

Refactored InputSpec definitions in exporter configs for improved readability by placing each argument on a separate line. Updated import path in litert.py to import from keras.src.export.litert instead of keras.src.export.litert_exporter.

* Refactor exporter configs and model building logic

Simplifies and unifies input signature and dummy input generation for text and image models by removing redundant helper methods and centralizing logic. Updates model building in KerasHubExporter to use input signatures and improves error handling. Refactors LiteRT exporter to use the new parameterized input signature and model building approach, reducing code duplication and improving maintainability.

* Refactor export initialization and improve warnings

Removed redundant registry initialization in export_model and clarified model building comments in KerasHubExporter. Switched to using warnings.warn for import errors in models/__init__.py instead of print statements for better error reporting.

* Improve dtype handling and verbose output in exporters

Refined dtype extraction logic in image and object model exporter configs to better handle different dtype representations. Updated LiteRT exporter to use Keras io_utils for progress messages and improved verbose flag handling. Added ObjectDetector and ImageSegmenter to export registry model type checks. Enhanced docstrings for clarity and consistency in base exporter classes.

* Remove get_dummy_inputs methods from exporter configs

Eliminates the get_dummy_inputs methods from KerasHubExporterConfig and its subclasses. Model building now relies solely on shape-based initialization, simplifying the export process and removing fallback logic for dummy data.

* Rename LitertExporter to LiteRTExporter

Refactored all references and class names from LitertExporter to LiteRTExporter for consistency with Keras naming conventions. This affects imports, class definitions, and usage throughout the export modules.

* Update registry.py

* Refactor exporter registry to use model classes

Changed exporter registry and config registration to use model classes instead of string type names for improved type safety and clarity. Updated input signature methods to use isinstance checks and standardized padding_mask dtype to int32. Enhanced LiteRTExporter to dynamically determine input signature parameters based on model type and preprocessor attributes.

* Remove conditional import for keras

Replaces the try-except block for importing keras with a direct import, assuming keras is always available. Simplifies the code and removes the KERAS_AVAILABLE flag.

* Add comprehensive export test suites for Keras Hub

Introduce new test modules for export base classes, configuration classes, LiteRT export functionality, registry logic, and production model export verification. Also update TensorFlow CUDA requirements to include ai-edge-litert for LiteRT export support.

* Refactor LiteRT exporter model adapters

Replaces the previous wrapper with type-specific adapter classes for text and image models in the LiteRT exporter, improving input conversion logic and maintainability. Also updates docstrings and return type annotations for consistency across exporter config classes.

* Clarify type annotations in docstrings for export modules

Updated docstrings in base.py, configs.py, and litert.py to specify explicit type annotations for function arguments and return values. This improves code readability and helps developers understand expected input types for exporter configuration and usage.

* testing refactor

* refactor test

* Fix LiteRT export filepath and mask argument usage

Ensure exported model filepath ends with '.tflite' and update verbose message to reflect correct path. Remove unused 'mask' argument from model calls in LiteRTExporter adapters. Update test to use input dtypes from interpreter for test inputs.

* Refactor LiteRTExporter model adapter calls

Simplifies calls to keras_hub_model in TextModelAdapter and ImageModelAdapter by removing unnecessary line breaks and grouping arguments. Also updates string quotes for consistency and improves formatting in litert_test.py for readability.

* Add warning for private TensorFlow API usage

Added a warning comment about using the private _DictWrapper API from tensorflow.python.trackable.data_structures in Backbone. This highlights potential instability and suggests considering alternatives or stricter TensorFlow version pinning if issues arise.

* Refactor exporter configs and remove TextModelExporterConfig

Removed the unused TextModelExporterConfig and its imports. Refactored sequence length and image dtype inference into helper functions for reuse and clarity. Updated LiteRTExporter to pass parameters to the export wrapper and simplified model type checks. Cleaned up Keras-Hub model detection logic in registry.

* Refactor trackable children filtering logic

Replaced explicit for-loops with list and dict comprehensions for filtering trackable children in lists and dicts. This improves code readability and conciseness in the Backbone model.

* Refactor ExporterRegistry model config lookup

Replaces hardcoded model class list with iteration over registered configs in ExporterRegistry. This improves maintainability and extensibility by removing direct imports and manual class checks.

* Update litert.py

* Refactor tests to remove try/except and improve clarity

Removed unnecessary try/except blocks and test skipping logic from multiple test files, making test failures more explicit and improving readability. Updated registry initialization to ensure Seq2SeqLM is registered before CausalLM. Simplified _is_keras_hub_model logic to use isinstance(Task) for more robust model type detection.

* Fix docstring in TextClassifierExporterConfig

Corrected the docstring in _is_model_compatible to refer to text classifier instead of image classifier.

* Update base.py

* Create litert_export_design.md

* Refactor LiteRT export tests for consistency and efficiency

Refactored LiteRT export tests to use a standardized helper method for model export and numerical accuracy verification, reducing code duplication and improving maintainability. Removed direct file and interpreter management in favor of a unified approach, tightened numerical accuracy thresholds, and ensured proper resource cleanup. Updated test cases to dynamically determine input shapes and handle model-specific requirements, improving robustness and reliability of export validation.

* Refactor LiteRT export tests to support per-output thresholds

Adds support for specifying per-output numerical thresholds and input ranges in LiteRT export tests. Refactors test utilities to handle output mapping, threshold configuration, and input preparation for improved accuracy and flexibility across model types.

* Update litert_models_test.py

* Delete litert_export_design.md

* Update litert_models_test.py

* Refactor LiteRT export tests to use pytest parametrization

Replaces class-based test cases with pytest parameterized functions for CausalLM, ImageClassifier, ObjectDetector, and ImageSegmenter LiteRT export tests. This improves test readability, reduces code duplication, and ensures each model configuration is tested independently with clearer output. Cleans up helper methods and consolidates numerical verification logic.

* Refactor export registry and add direct export to Task

Removed the export registry and related initialization logic, replacing it with a direct model type detection via `get_exporter_config`. The `Task` class now provides its own `export` method for specialized Keras-Hub model export, supporting dictionary inputs and LiteRT export. Cleaned up imports and removed registry-related tests and files.

* Update litert.py

* Update task.py

* Update test_case.py

* Enable dynamic input shapes for LiteRT export

Updates LiteRT exporter and related configs to support dynamic input shapes by default for text and image models, allowing runtime resizing via TFLite's interpreter.resize_tensor_input(). Removes static sequence length inference, adapts input signature logic, and updates tests to verify dynamic shape support and runtime resizing. Also improves multimodal model handling and input mapping for TFLite export.

* Improve SignatureDef handling in LiteRT export tests

Adds tests to verify that SignatureDef preserves input names for ImageClassifier and CausalLM models. Refactors test utilities to use SignatureDef for input/output mapping, ensuring meaningful names and robust output verification. Updates numerical accuracy checks to compare outputs by name using SignatureDef, and adds validation for expected input/output names in exported models.

* Refactor LiteRT test utilities for clarity and robustness

Consolidates LiteRT input preparation, inference, and output verification into clearer helper methods. Improves handling of dynamic shapes, input/output name matching via SignatureDef, and output comparison logic. Updates docstrings and argument names for consistency and readability.

bug fix in inference:
- there was a bug, that used corrupted the results of invoke, during getting the SignatureDef fixed it.

* Refactor TFLite inference to use signature runner

Updated test utilities to use TFLite's signature runner for inference, simplifying input handling and output extraction. Also updated model creation in numerical accuracy tests to use explicit Input layers for clarity and consistency.

* Add exporter configs for Multimodal models

Now supporting gemma3 multimodal and other models too.

* Add LiteRT export tests for model test suites

Introduces `test_litert_export` methods to numerous model test files, enabling automated testing of LiteRT export functionality. Also updates SAMPromptEncoder mask handling for LiteRT compatibility and adjusts test input shapes for SAMImageSegmenter. These changes improve coverage and reliability of LiteRT export across text, image, audio, and multimodal models.

* Refactor and expand exporter configs, update tests

Added new exporter configs for AudioToText, SAMImageSegmenter, and TextToImage models. Refactored input signature logic for multimodal and text/image models in configs.py and litert.py. Updated test files to remove unused keras imports and improve input data handling for LiteRT export compatibility.

* Add DepthEstimator export support and improve tests

Introduces DepthEstimatorExporterConfig for depth estimation model export, updates LiteRT exporter to support DepthEstimator, and refines input signature logic for object detectors. Test files now skip LiteRT export tests for non-TensorFlow backends, and output thresholds/statistical comparison modes are adjusted for several models. The base TestCase class is refactored to remove TensorFlow dependency, improving backend compatibility. The run_litert_tests.py script is enhanced for backend selection, parametrized test parsing, and markdown reporting.

* Delete run_litert_tests.py

* Update test_case.py

* Add @pytest.mark.large to LiteRT export tests

Marked LiteRT export tests with @pytest.mark.large in object detector, causal LM, and image segmenter test files to better categorize resource-intensive tests. Also simplified dtype handling in TestCase for TensorFlow tensors.

* Update model test cases for statistical comparison

Test cases for DepthAnything, FNet, PaliGemma, RetinaNet, and SAM models now use statistical comparison modes and output thresholds for LiteRT export tests. The input size for RetinaNet tests was increased to 800. The TestCase utility was improved to extract comparison_mode from export_kwargs and to allow SignatureDef to have additional optional inputs, only failing if expected inputs are missing.

* Refactor input handling for export and tests

Moved the Dictionary handling to keras Core, Also moved Wrapper creation and handling to keras.
Removed Adapter stuff.

Simplifies input signatures for image and object detection models to use single tensor inputs instead of dictionaries. Removes support for 'image_shape' as an input for object detectors and updates related tests and exporter logic to match. Cleans up legacy model building code and adapts tests to new input formats for classifiers, detectors, and segmenters. Delegates LiteRT export to Keras Core's exporter for improved input handling.

* Refactor LiteRT export to use Keras Core directly

Removed LiteRTExporter class and its tests, updating all usages to call Keras Core's export_litert directly with domain-specific input signatures from exporter configs. The base exporter classes were moved into configs.py, and input signature logic now defaults to preprocessor's sequence_length if available. Updated documentation and tests to reflect the new export flow.

* Add @pytest.mark.large to LiteRT export tests

Marked LiteRT export tests as large in multiple model test files to improve test categorization. Also updated statistical comparison parameters in PARSeq test and clarified docstring in Task class. Refactored single tensor input handling in TestCase for consistency.

* Update d_fine_object_detector_test.py

* Update retinanet_object_detector_test.py

* Add @pytest.mark.large to saved model tests

Applied the @pytest.mark.large decorator to the test_saved_model methods in d_fine_object_detector_test.py, retinanet_object_detector_test.py, and sam_image_segmenter_test.py to categorize these tests as large.

* Remove TensorFlow export compatibility shims

Deleted the _get_save_spec and _trackable_children methods from the Backbone class, which provided compatibility for TensorFlow SavedModel/TFLite export and filtered problematic _DictWrapper objects.
Moved this to core keras.

* Update test configs and refactor export verbose handling

Replaces 'tolerances' with 'output_thresholds' in depth estimator and GPTNeoX tests, adds target_spec to SAM and StableDiffusion3 tests, and refactors verbose handling in Task export to avoid passing it to core export. These changes improve test configuration clarity and address runtime issues with delegate selection.

* Fix ESM attention for TFLite compatibility

* Fix PARSeq decoder for TFLite compatibility

Use ops.cond() instead of Python if-statement in PARSeqDecoder.call()
to ensure graph mode compatibility during TFLite conversion.

- Changed 'if tokens_length > 1:' to ops.cond(tokens_length > 1, ...)
- This allows TensorFlow to properly trace the graph with symbolic tensors

This fixes the 'Using a symbolic tf.Tensor as a Python bool is not allowed'
error during TFLite conversion when sequence length is dynamic (None).

* Add LiteRT export tests to model test suites

Introduces `test_litert_export` to multiple model test files, enabling automated testing of LiteRT export functionality. Some tests include statistical comparison modes and output thresholds for validation. The FNet test was updated to clarify input requirements for LiteRT export.

* Remove export module and clean up API imports

Deleted the keras_hub/api/export and keras_hub/src/export modules and their related files. Updated __init__.py files in api and submodules to remove references to export and reformat import statements for improved readability and maintainability.

* Refactor content construction for backend consistency

Replaces ops.cond-based content building with a branchless approach to ensure consistent tensor shapes across graph and JAX backends. This change improves compatibility and simplifies the logic for handling token positions.

* Skip LiteRT export tests for non-TensorFlow backends

Added a pytest skipif marker to LiteRT export tests in multiple model test files to ensure these tests only run when the Keras backend is TensorFlow, as LiteRT export is not supported on other backends.

* Skip LiteRT export tests for non-TensorFlow backends

Added @pytest.mark.skipif to LiteRT export tests in multiple model test files to skip these tests when the Keras backend is not TensorFlow, as LiteRT export only supports TensorFlow. Also standardized import of keras in affected test files.

* Add TensorFlow-only LiteRT export test skip to model tests

Added @pytest.mark.skipif to LiteRT export tests in all model test files to skip tests when the Keras backend is not TensorFlow, as LiteRT export only supports TensorFlow. Also standardized import order for keras and pytest in several files. Removed unused __init__.py docstring in models module.

* Remove TensorFlow-only LiteRT export test skips

Deleted @pytest.mark.skipif decorators that restricted LiteRT export tests to TensorFlow backend across all model test files.
it is handled inside `run_litert_export_test` in test_case.py

* Update retinanet_object_detector_test.py

* Update retinanet_object_detector_test.py

* Move ai-edge-litert to requirements-common.txt

The ai-edge-litert dependency was moved from requirements-tensorflow-cuda.txt to requirements-common.txt to ensure it is included for all environments. This change simplifies dependency management and avoids duplication.

* Update qwen3_moe_causal_lm_test.py

* Replace tf.cast with ops.cast for padding_mask

Updated tests in Gemma3 and Llama3 causal LM modules to use ops.cast instead of tf.cast for converting padding_mask to int32, improving compatibility with LiteRT and reducing TensorFlow dependency.

* Remove trailing blank line from requirements files

Cleaned up formatting in requirements-tensorflow-cuda.txt and requirements.txt by removing unnecessary trailing blank lines after including requirements-common.txt.

* Refactor type casting to use ops.cast and rename test method

Replaced tf.cast with ops.cast for type conversions in test files to ensure consistency and compatibility. Renamed _verify_outputs to _verify_litert_numerics in test_case.py for clarity and updated all relevant references.

* Update test_case.py

* Update retinanet_object_detector_test.py

* Skip TFLite export tests for GPTNeoX and SAM models

Temporarily skip the TFLite export tests in GPTNeoX and SAM image segmenter test files due to known bugs. Added TODO comments to revisit and fix the issues before re-enabling these tests.

* Update requirements.txt

* Update CI workflows to use Python 3.11

All GitHub Actions workflows now use Python 3.11 instead of 3.10 for consistency and to leverage the latest Python features and improvements.

* Revert "Update CI workflows to use Python 3.11"

This reverts commit 79cd5e8.

* Reapply "Update CI workflows to use Python 3.11"

This reverts commit dec7688.

* Update CI to use Python 3.10 and adjust test skip

Changed GitHub Actions workflows to use Python 3.10 instead of 3.11 for consistency across actions, nightly, and PyPI publishing. Also refactored the TFLite export test skip in GPTNeoXCausalLMTest to be within the test method.

* update python version

* update version

* update keras version

* Fix reversible embedding quantization (keras-team#2476)

* Update gemma3_causal_lm_preprocessor.py

Added checks for invalid inputs

* Update gemma3_causal_lm_preprocessor.py

* Update gemma3_causal_lm_preprocessor_test.py

Added tests to check invalid inputs

* Update reversible_embedding.py

Fix for model not loading when using numpy behaviour with tensorflow

* upadated Gemma3InterleaveEmbeddings

* Update gemma3_interleave_embeddings.py

* Revert "Update reversible_embedding.py"

This reverts commit 3fdc7fd.

* Update gemma3_interleave_embeddings.py

* Fix ReversibleEmbedding quantization for Keras 3.13 compatibility

- Update _int8_build method to accept config parameter
- Ensure backward compatibility with Keras 3.12
- Resolves TypeError in quantization tests

* Models should reference ReversibleEmbedding from Keras core (keras-team#2482)

* Models should reference ReversibleEmbedding from Keras core

* backwards compat

* remove keras 3.9 from test matrix

* python version changes

* disable cache when fetching nightly

* update keras version

* update gpu python version

* update python version

* resolve gpu errors

* check gpu tests

* update python version

* fix gpu tests

* sigh

* Added more tests

* fix error

* Skip liteRT export tests due to known bugs

Added @pytest.skip decorators to the liteRT export tests in MoonshineAudioToTextTest and StableDiffusion3TextToImageTest due to known bugs. This prevents test failures while the issues are being addressed.

* Use pytest.mark.skip for export test skips

Replaces @pytest.skip with @pytest.mark.skip in MoonshineAudioToText and StableDiffusion3TextToImage test files for consistency with pytest's recommended usage.

* Update build script to use Python 3.11

Switched the default Python binary from 3.10 to 3.11 in the GPU build script. Commented out the temporary relaxation of the Python version requirement, as 3.11 is now standard.

* Remove @pytest.mark.large from model export tests

Eliminated the @pytest.mark.large decorator from all LiteRT export test methods across model test files to streamline test categorization and avoid marking these export tests as large. Also added missing LiteRT export test for MobileNetBackbone.

* Switch GPU build to Python 3.10

Changed the Python binary from 3.11 to 3.10 and updated pyproject.toml requirement to >=3.10 for GPU tests. This temporarily relaxes the Python version constraint to support GPU testing.

* Remove extra blank lines before test_litert_export methods

Deleted unnecessary blank lines preceding the test_litert_export method in multiple model test files for consistency and code cleanliness.

* Revert infrastructure files to master state: remove Python/Keras version bumps

* Revert GitHub Actions workflows to master state: remove Python/Keras version bumps

* Update to Python 3.11 and Keras 3.13, drop 3.9 support

Bump minimum Python version to 3.11 and update Keras dependency to >=3.13 in pyproject.toml and CI workflows. Remove Python 3.9 references and ai-edge-litert from requirements. Add temporary workaround in GPU build script to allow Python 3.10 for GPU tests.

* Skip LiteRT export test for Keras < 3.13

Added a version check to skip the LiteRT export test if the installed Keras version is less than 3.13, as this feature requires Keras 3.13 or higher.

* Skip VGGImageClassifier LiteRT export test

Marked the test_litert_export test as skipped due to a known bug with VGGImageClassifier LiteRT export. Added a TODO note for future investigation.

* Remove redundant TensorFlow import in test_case.py

Eliminates an unnecessary import of TensorFlow within the ImportError exception block, as tf is already available in the module context.

---------

Co-authored-by: divyashreepathihalli <divyashreepathihalli@gmail.com>
Co-authored-by: Jyotinder Singh <33001894+JyotinderSingh@users.noreply.github.com>

Author: 3 people

keras_hub/src/tests/test_case.py

@@ -605,8 +605,6 @@ def run_litert_export_test(
         try:
             from ai_edge_litert.interpreter import Interpreter
         except ImportError:
-            import tensorflow as tf
-
             Interpreter = tf.lite.Interpreter
 
         if output_thresholds is None: