Add pet clustering with feedback, naming, and search by Amrithesh-Kakkoth · Pull Request #9665 · ente-io/ente

Amrithesh-Kakkoth · 2026-03-15T04:12:59Z

Summary

Add ML-based pet detection, clustering, and recognition with user feedback support, pet naming, and search integration

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 536ec02e71

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

web/packages/new/photos/components/utils/use-app-lock.ts

web/packages/accounts-rs/components/RecoveryKey.tsx

3-phase Rust clustering pipeline (face cluster, body rescue, cross-merge) using average-linkage agglomerative clustering, validated against Python reference with 100% agreement (NMI=1.0). Includes Dart service layer, DB migrations, API bindings, and test binary for comparison.

Call PetClusteringService after face clustering in runAllML so pet clusters are formed automatically after indexing completes.

- LazySession was forcing CPU-only for ALL models; now only pet models use CPU-only policy (to avoid NNAPI/CoreML issues with FP16 models) - Restore preferNnapi and preferXnnpack to Platform.isAndroid - Restore GraphOptimizationLevel::Level3 (was downgraded to Level1) - Restore arena allocator on CPUExecutionProvider

Sessions stay loaded across images and get cleaned up when the isolate disposes via release_runtime(). Avoids expensive repeated ONNX session load/unload cycles on every image.

This was debug-only code that altered production indexing logic to re-process all files for pet-only mode. Removes the flag, the forceAll parameter threading through ml_util, and the ClipText skip hack.

- Inline embedding columns into CREATE TABLE (remove ALTER TABLE migrations) - Store landmarks inside detection JSON (remove separate column) - Remove migration tracking from PetVectorDB (tables are new) - Rename pet_bodies table to detected_objects with generic column names - Change species column from TEXT to INTEGER (store raw COCO class) - Revert shouldAutomaticDispose back to true - Add separate petMlVersion constant - Move embedding storage methods from MLService to MLDataDB

Add pet_indexed_files tracking table so files already indexed for faces+CLIP are re-queued when pet recognition is enabled. Also fix MLResult.ranML to include petsRan, which was causing pet-only analysis results to be silently discarded.

…sc bugs - Migrate ML runtime from Mutex to RwLock + OnceCell for concurrent inference (CLIP text no longer blocks image indexing) - Fix cat face detection: model outputs per-class scores, not a class ID; use argmax instead of truncating float to u8 - Fix species normalization: store 0/1 (dog/cat) not COCO 15/16 - Fix markPetIndexed guard to check shouldRunPets && rustPets - Add pet model paths to runtime config cache key - Batch pet body embeddings by species (matching face pattern) - Rename objectIDColumn to detectedObjectIDColumn to avoid ambiguity - Implement deletePetDataForFiles in DB layer - Fix design system usage in pet thumbnail widget

…ing column Co-Authored-By: Amrithesh <amritheshkakkoth10@gmail.com>

… missing schema tables

Amrithesh-Kakkoth · 2026-03-15T09:23:57Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2def4bb422

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mobile/apps/photos/lib/services/machine_learning/pet_ml/pet_clustering_service.dart

mobile/apps/photos/lib/ui/viewer/people/pet_cluster_page.dart

Extract shared clustering algorithm from pet module into reusable ml::cluster module. Add face-specific batch and incremental clustering functions with rejection support. Wire as fallback in ml_service when Dart linear fails and useRustForML is enabled. Also remove dead petIndexedFilesTable and redundant markPetIndexed calls.

Amrithesh-Kakkoth · 2026-03-15T14:06:08Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1f9c592313

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mobile/apps/photos/rust/src/ml/pet/cluster.rs

mobile/apps/photos/lib/ui/viewer/people/pet_cluster_page.dart

…esh, species-filtered moves

Amrithesh-Kakkoth · 2026-03-15T16:13:53Z

@codex review

- Move cross-model human face suppression into pet detection loop for single-pass, zero-allocation filtering - Raise pet face detection threshold from 0.3 to 0.5 - Add toggle flag to disable cross-model filtering - Show only user-given names for pets, hide auto-generated labels like "Dog 1", "Cat 2"; show "Add name" prompt instead - Resolve merge conflict in feature flag service

Amrithesh-Kakkoth · 2026-04-02T12:12:37Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd3b0297a7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mobile/apps/photos/lib/services/machine_learning/pet_ml/pet_service.dart

- Swap reconcileClusters order: push local changes to remote before fetching remote feedback, preventing fresh local assignments from being deleted by the stale-cleanup pass - Clear remote assigned list for pets that lost all local clusters, preventing stale assignments from persisting on the server

laurenspriem · 2026-04-07T10:05:38Z

@codex review

Offline mode is view-only raw clustering output. Skip not-pet feedback table creation/deletion and rejected-assignment overrides in offline mode. Disable name editing, cluster view, merge, and selection bar in pet cluster UI when offline.

mobile/apps/photos/lib/db/ml/db.dart

laurenspriem · 2026-04-07T10:17:18Z

mobile/apps/photos/lib/db/ml/db.dart

-        _logger.warning("Failed to delete pet body vectors", e, s);
+    // Collect which clusters will lose members so we can recompute their
+    // summaries and centroids after the deletion.
+    final affectedClusterIds = <String>{};


instead of recomputing, let's delete and recompute when actually needed? (assuming we keep centroid)

laurenspriem · 2026-04-07T10:33:29Z

mobile/apps/photos/lib/db/ml/schema.dart

+    'pet_cluster_centroid_vector_id_map';
+const petClusterCentroidVectorIdColumn = 'pet_cluster_vector_id';
+
+const createPetClusterCentroidVectorIdMappingTable = '''


Check if we're using this for clustering or if this can be removed

It is used in clustering. It maps cluster_id to integer key

mobile/apps/photos/lib/ui/viewer/people/merge_pet_sheet.dart

laurenspriem · 2026-04-07T11:06:32Z

mobile/apps/photos/rust/Cargo.toml


 [lib]
-crate-type = ["cdylib", "staticlib"]
+crate-type = ["cdylib", "staticlib", "rlib"]


Bump. check other old comments too

rust/photos/src/ml/pet/detect.rs

rust/photos/src/ml/indexing.rs

laurenspriem · 2026-04-07T11:57:35Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0fe3d34071

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mobile/apps/photos/lib/services/machine_learning/pet_ml/pet_service.dart

chatgpt-codex-connector · 2026-04-07T12:08:17Z

mobile/apps/photos/lib/services/machine_learning/pet_ml/pet_service.dart

+  }
+
+  Future<void> deletePet(String petID) async {
+    await entityService.deleteEntry(petID);


Remove cluster mappings when deleting a pet

Deleting a pet entity here does not clear pet_cluster_pet rows for that pet, so clusters remain locally mapped to a now-missing pet until a later sync pass cleans orphans. During that period, search/grouping still treats those clusters as merged under the deleted pet ID, which leaves inconsistent UI state and can propagate stale mapping behavior.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-04-07T12:08:17Z

rust/photos/src/ml/pet/detect.rs

+    eprintln!(
+        "[ml][pet] face detect: output_shape={:?}, row_len={}, detection_rows={}, is_2class={}",
+        output_shape,
+        row_len,
+        detection_rows,
+        row_len >= 13
+    );


Remove verbose stderr logging from pet detection loop

These unconditional eprintln! calls run in the inference hot path for every analyzed image (and per detection), which can flood log output and add avoidable overhead on-device. In production builds this can noticeably hurt ML throughput and make logs noisy; these statements should be gated behind debug-only logging or removed.

Useful? React with 👍 / 👎.

# Conflicts: # mobile/apps/photos/pubspec.yaml # mobile/apps/photos/pubspec_overrides.yaml

chatgpt-codex-connector bot reviewed Mar 15, 2026

View reviewed changes

web/packages/new/photos/components/utils/use-app-lock.ts Show resolved Hide resolved

web/packages/accounts-rs/components/RecoveryKey.tsx Show resolved Hide resolved

Amrithesh-Kakkoth added 18 commits March 15, 2026 09:57

Wire pet clustering into ML pipeline

c4ae159

Call PetClusteringService after face clustering in runAllML so pet clusters are formed automatically after indexing completes.

Add FP16 model support, lazy session loading, and pet recognition types

096b952

Add pet face and body detection, alignment, and embedding pipeline

dc69aa3

Expose pet recognition pipeline through Rust API bridge

ed5889e

Integrate pet recognition into ML indexing pipeline with feature flag

5834bab

feat: Implement pet detection and display pet faces in file details.

064e1a9

Remove libc crate dependency, use raw extern C declaration for dlsym

c8cb1fa

Remove per-image session unloading from ML pipeline

3205d5b

Sessions stay loaded across images and get cleaned up when the isolate disposes via release_runtime(). Avoids expensive repeated ONNX session load/unload cycles on every image.

Remove debugPetOnlyMode and forceAll debug scaffolding

1d0fd11

This was debug-only code that altered production indexing logic to re-process all files for pet-only mode. Removes the flag, the forceAll parameter threading through ml_util, and the ClipText skip hack.

Rename detected_objects schema to pet_bodies and remove unused embedd…

b9c0bbe

…ing column Co-Authored-By: Amrithesh <amritheshkakkoth10@gmail.com>

Retrigger CI checks

545a779

Add pet cluster feedback, naming, and search integration

d76e3f1

Fix mutable runtime API after rebase

a3b74d4

Amrithesh-Kakkoth force-pushed the ml/pet-clustering branch from 536ec02 to a3b74d4 Compare March 15, 2026 08:46

Fix review issues: parameterize SQL, guard clustering memory, restore…

2def4bb

… missing schema tables

chatgpt-codex-connector bot reviewed Mar 15, 2026

View reviewed changes

mobile/apps/photos/lib/services/machine_learning/pet_ml/pet_clustering_service.dart Show resolved Hide resolved

mobile/apps/photos/lib/ui/viewer/people/pet_cluster_page.dart Outdated Show resolved Hide resolved

Amrithesh-Kakkoth added 3 commits March 15, 2026 18:51

Fix clippy lints: collapsible ifs, is_multiple_of, doc formatting

f198e7d

Remove unused deletePetName method

1f9c592

chatgpt-codex-connector bot reviewed Mar 15, 2026

View reviewed changes

mobile/apps/photos/rust/src/ml/pet/cluster.rs Outdated Show resolved Hide resolved

mobile/apps/photos/lib/ui/viewer/people/pet_cluster_page.dart Outdated Show resolved Hide resolved

Fix incremental pet clustering: stale summary deletion, centroid refr…

258797e

…esh, species-filtered moves

Amrithesh-Kakkoth added 6 commits April 2, 2026 14:11

Merge main into ml/pet-clustering

ab64ba6

Fix rustfmt formatting in pet clustering

f62338c

Fix rustfmt formatting in indexing.rs

70ddeb6

Fix clippy warnings in pet clustering

d35aa8b

Remove redundant null checks on LatLng from latlngAsync

bd3b029

Amrithesh-Kakkoth closed this Apr 2, 2026

Amrithesh-Kakkoth reopened this Apr 2, 2026

chatgpt-codex-connector bot reviewed Apr 2, 2026

View reviewed changes

mobile/apps/photos/lib/services/machine_learning/pet_ml/pet_service.dart Show resolved Hide resolved

mobile/apps/photos/lib/services/machine_learning/pet_ml/pet_service.dart Show resolved Hide resolved

Amrithesh-Kakkoth added 6 commits April 2, 2026 18:00

Update pet clustering and service layer after main merge

49feb77

Merge upstream main, resolve import conflict in ml_service.dart

82457e2

Optimize agglomerative clustering

2fa25d0

Format cluster.rs

2e1a93b

Fix clippy lint in cluster loop

8301f6f

Amrithesh-Kakkoth added 3 commits April 7, 2026 16:30

chore: Apply dart format

7fafbd4

refactor: remove pet human-face cross-check

0fe3d34

laurenspriem reviewed Apr 7, 2026

View reviewed changes

chatgpt-codex-connector bot reviewed Apr 7, 2026

View reviewed changes

Amrithesh-Kakkoth added 7 commits April 7, 2026 17:55

refactor: remove redundant pet search params

90636cd

refactor: remove pet cluster move flow

3ee48e1

refactor: drop pet cluster merge helper

5ceff27

refactor: remove incomplete pet merge flows

c8c06f9

Merge remote-tracking branch 'origin/main' into ml/pet-clustering

f3d16a5

# Conflicts: # mobile/apps/photos/pubspec.yaml # mobile/apps/photos/pubspec_overrides.yaml

build: drop photos rust rlib crate type

d94c576

fix: await pet assignment sync updates

bc52511

Uh oh!

Conversation

Amrithesh-Kakkoth commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Amrithesh-Kakkoth commented Mar 15, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Amrithesh-Kakkoth commented Mar 15, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Amrithesh-Kakkoth commented Mar 15, 2026

Uh oh!

Amrithesh-Kakkoth commented Apr 2, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

laurenspriem commented Apr 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

laurenspriem Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

laurenspriem Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Amrithesh-Kakkoth Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

laurenspriem Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

laurenspriem commented Apr 7, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Amrithesh-Kakkoth commented Mar 15, 2026 •

edited

Loading