[MODFQMMGR-1034] Handle entity type inheritance when migrating queries, custom entities (#1144)

Author: ncovercash

docs/Migration.md

@@ -1,9 +1,10 @@
-# Query Migration
+# Migration
 
 Entity types and their fields change over time, be it adding fields, moving them between entity types, or completely rethinking the way some fields are handled. As such, we have a robust migration system to ensure that consuming apps will not break, and their queries will continue to work despite any internal FQM changes.
 
-- [Query versions](#query-versions)
+- [Versions](#versions)
 - [Updating a query](#updating-a-query)
+- [Updating an entity type](#updating-an-entity-type)
 - [Writing migrations](#writing-migrations)
   - [Changes](#changes)
     - [Entity type changes](#entity-type-changes)
@@ -26,8 +27,10 @@ Entity types and their fields change over time, be it adding fields, moving them
     - [State](#state)
     - [Extra magic](#extra-magic)
     - [Advanced migration tips](#advanced-migration-tips)
+- [Custom entity types support](#custom-entity-types-support)
+  - [Recovery](#recovery)
 
-## Query versions
+## Versions
 
 The version of a query is stored inside the FQL string:
 
@@ -38,14 +41,28 @@ The version of a query is stored inside the FQL string:
 }
 ```
 
-These are arbitrary strings, and consuming applications should make no assumptions about them (they are currently integers, but may be changed in the future to commit hashes, module versions, or anything else).
+And versions of custom entity types are stored inside the entity definition:
+
+```json
+{
+  "id": "d41130e9-0302-5ef3-a6b2-70f6ae1678ce",
+  "name": "my_custom_entity",
+  "_version": "3"
+}
+```
+
+These are arbitrary strings, and consuming applications should make no assumptions about them (they are currently semver-adjacent, but may be changed in the future to commit hashes, module versions, or anything else).
 
 Queries from Quesnelia or earlier will have no version associated with them and will be considered version `"0"`.
 
 ## Updating a query
 
 To update a query, send it, the entity type ID, and a list of fields (if desired) to `/fqm/migrate`. See our [API documentation](https://dev.folio.org/reference/api/#mod-fqm-manager) for more information about this endpoint. Our module will return the updated query, entity type ID, and list of fields, all of which should be saved. Additionally, the response may contain [warnings](#warnings), meaning that some parts of the query or field list was unable to be migrated.
 
+## Updating an entity type
+
+Custom entity types will be migrated when the module is installed. No additional action is required; for more information see [custom entity types support](#custom-entity-types-support).
+
 ## Writing migrations
 
 Any change to an entity type that results in a field being removed or renamed should result in a migration script. The easiest way to do this is to do the following:
@@ -108,7 +125,20 @@ public Map<UUID, UUID> getEntityTypeChanges() {
 
 ### Defining source maps
 
-<!-- TODO: describe this (in next PR) -->
+Source maps are used in migrations to define relations between composite and simple entity types. For example, if your migration alters `simple_instance_status`, it's necessary for the migration system to know that `composite_instances`'s `inst_stat` source points to `simple_instance_status`. To define these relationships, override `getEntityTypeSourceMaps` (note that the inner keys are the source aliases used by the composite):
+
+```java
+public Map<UUID, Map<String, UUID>> getEntityTypeSourceMaps() {
+  return Map.of(
+    COMPOSITE_INSTANCES_ID, Map.of("inst_stat", SIMPLE_INSTANCE_STATUS_ID),
+    COMPOSITE_ITEM_DETAILS_ID, Map.of("instance_status", SIMPLE_INSTANCE_STATUS_ID)
+  );
+}
+```
+
+> [!NOTE]
+>
+> Only references from all inheriting composites to the migrated entities need to be defined here — other sources used in parent entities do not need to be explicitly stated.
 
 ### Warnings
 
@@ -346,3 +376,38 @@ public MigratableQueryInformation additionalChanges(Void v, MigratableQueryInfor
   - `{entityTypeId=composite-users-et, fieldPrefix=outer_entity., field=users.id}`
   - `{entityTypeId=simple-user-et, fieldPrefix=outer_entity.users., field=id}`
   - Iterations are done in this order (from the outermost entity to the simplest) and will stop either when a transformation **does** occur (field/condition changes, warning emitted, etc) or when there's no more levels to process.
+
+## Custom entity types support
+
+> “With great power comes great responsibility”
+>
+> _- Uncle Ben, Spider-Man comics_
+
+Custom entity types are incredibly powerful, however, this very power limits the ability for the entities and their queries to be automatically migrated.
+
+Currently, FQM will migrate custom entity types based on changes to FQM itself. **No migration is supported for changes made by users to custom entity types.** Here is what FQM will migrate on the entities:
+
+- Source entity type ID changes,
+- Source/target join field changes,
+- Default sort order, and
+- Group by definitions.
+
+Queries will be migrated just like any other, with the exception of:
+
+- If a source's entity type ID changes, queries may not have migrations applicable to that source performed.
+
+> [!WARNING]
+>
+> Custom entity migration is done on a “best effort” basis and may not cover all edge cases, nor will it necessarily guarantee a working entity type or query after migration. In the event that something could not be automatically handled (for example, a source's `targetField` is no longer available), a warning will be emitted in the custom entity's `description`. Be sure to check these descriptions and the migration warnings after performing a migration to ensure everything is still as expected.
+>
+> For additional validation, or if you experience issues, follow the [recovery](#recovery) steps below.
+
+### Recovery
+
+In the event that migration results in a ”broken” entity type (for example, a source no longer exists), it can be easily repaired. To do so, follow these steps:
+
+1. `GET` the migrated entity type via `/entity-types/custom/{id}`,
+2. Fix any noticed issues,
+3. `PUT` it back to `/entity-types/custom/{id}`,
+4. If validation fails, go back to step 2.
+5. Success! 🎉

src/main/java/org/folio/fqm/migration/MigratableQueryInformation.java

@@ -16,11 +16,10 @@ public record MigratableQueryInformation(
   @CheckForNull String fqlQuery,
   List<String> fields,
   @Singular List<Warning> warnings,
-  String version,
   boolean hadBreakingChanges
 )
   implements Serializable {
   public MigratableQueryInformation(UUID entityTypeId, String fqlQuery, List<String> fields) {
-    this(entityTypeId, fqlQuery, fields, List.of(), null, false);
+    this(entityTypeId, fqlQuery, fields, List.of(), false);
   }
 }

src/main/java/org/folio/fqm/migration/MigrationUtils.java

@@ -41,16 +41,21 @@ public class MigrationUtils {
    * Helper function to transform an FQL query where each field gets turned into a new quantity of fields.
    * This runs a given function on each field's condition in the query, potentially adding or removing $and as needed.
    *
+   * Note that, for nested fields, the handler function may be called multiple times for the same field,
+   * once for each level of nesting. If the handler modifies the field/condition, no further unwrapping will be attempted.
+   *
    * @param entityTypeId The entity type ID of the query being migrated
    * @param fqlQuery The root query to migrate
    * @param handler  something that takes an {@link MigratableFqlFieldAndCondition} and returns a list of
    *                 {@link SingleFieldMigrationResult} indicating the new field(s), warnings, and whether
    *                 a breaking change occurred
+   * @param sourceMappings A map of entity type IDs to their sources (alias -> source target ID), used for unwrapping nested fields
    */
   public static MigrationResult<String> migrateFql(
     UUID entityTypeId,
     String fqlQuery,
-    Function<MigratableFqlFieldAndCondition, SingleFieldMigrationResult<MigratableFqlFieldAndCondition>> handler
+    Function<MigratableFqlFieldAndCondition, SingleFieldMigrationResult<MigratableFqlFieldAndCondition>> handler,
+    Map<UUID, Map<String, UUID>> sourceMappings
   ) {
     try {
       ObjectNode fql = (ObjectNode) objectMapper.readTree(fqlQuery);
@@ -68,7 +73,9 @@ public static MigrationResult<String> migrateFql(
 
       List<SingleFieldMigrationResult<MigratableFqlFieldAndCondition>> transformed = startingFields
         .stream()
-        .map(f -> handleSingleFieldWithNesting(f, handler, MigrationUtils::didMigrationModifyFieldAndCondition))
+        .map(f ->
+          handleSingleFieldWithNesting(f, handler, MigrationUtils::didMigrationModifyFieldAndCondition, sourceMappings)
+        )
         .toList();
       List<MigratableFqlFieldAndCondition> resultingFields = transformed
         .stream()
@@ -105,19 +112,22 @@ public static MigrationResult<String> migrateFql(
    * @param handler something that takes an {@link MigratableFqlFieldOnly} and returns a list of
    *                {@link SingleFieldMigrationResult} indicating the new field(s), warnings, and
    *                whether a breaking change occurred
+   * @param sourceMappings A map of entity type IDs to their sources (alias -> source target ID), used for unwrapping nested fields
    */
   public static MigrationResult<List<String>> migrateFieldNames(
     UUID entityTypeId,
     List<String> fields,
-    Function<MigratableFqlFieldOnly, SingleFieldMigrationResult<MigratableFqlFieldOnly>> handler
+    Function<MigratableFqlFieldOnly, SingleFieldMigrationResult<MigratableFqlFieldOnly>> handler,
+    Map<UUID, Map<String, UUID>> sourceMappings
   ) {
     List<SingleFieldMigrationResult<MigratableFqlFieldOnly>> transformed = fields
       .stream()
       .map(f ->
         handleSingleFieldWithNesting(
           new MigratableFqlFieldOnly(entityTypeId, "", f),
           handler,
-          MigrationUtils::didMigrationModifyFieldOnly
+          MigrationUtils::didMigrationModifyFieldOnly,
+          sourceMappings
         )
       )
       .toList();
@@ -135,17 +145,37 @@ public static MigrationResult<List<String>> migrateFieldNames(
     );
   }
 
+  /**
+   * Iteratively calls `handler` on each level of nesting for the given field, until either:
+   * - the handler modifies the field (as determined by `didModify`), or
+   * - there is no further nesting to unwrap
+   */
   private static <F extends MigratableFqlField<F>> SingleFieldMigrationResult<F> handleSingleFieldWithNesting(
     F original,
     Function<F, SingleFieldMigrationResult<F>> handler,
-    BiPredicate<F, SingleFieldMigrationResult<F>> didModify
+    BiPredicate<F, SingleFieldMigrationResult<F>> didModify,
+    Map<UUID, Map<String, UUID>> sourceMappings
   ) {
     SingleFieldMigrationResult<F> transformed = handler.apply(original);
+    int fieldDelimiterIndex = original.field().indexOf('.');
 
-    // stub for follow-up ticket
-    didModify.test(original, transformed);
+    if (didModify.test(original, transformed) || fieldDelimiterIndex == -1) {
+      return transformed;
+    }
+
+    Map<String, UUID> sourceMap = sourceMappings.get(original.entityTypeId());
+    String source = original.field().substring(0, fieldDelimiterIndex);
+    String remainder = original.field().substring(fieldDelimiterIndex + 1);
+    if (sourceMap == null || sourceMap.get(source) == null) {
+      return transformed;
+    }
 
-    return transformed;
+    return handleSingleFieldWithNesting(
+      original.dereferenced(sourceMap.get(source), source, remainder),
+      handler,
+      didModify,
+      sourceMappings
+    );
   }
 
   public static boolean didMigrationModifyFieldAndCondition(

src/main/java/org/folio/fqm/migration/strategies/AbstractRegularMigrationStrategy.java

@@ -1,8 +1,11 @@
 package org.folio.fqm.migration.strategies;
 
 import java.util.Collection;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 import java.util.Objects;
+import java.util.UUID;
 import java.util.stream.Stream;
 import org.folio.fqm.migration.MigratableQueryInformation;
 import org.folio.fqm.migration.MigrationUtils;
@@ -23,6 +26,27 @@ public S getStartingState() {
     return null;
   }
 
+  /**
+   * Defines the relationship of composite entities to the entity being migrated here. For example,
+   * a migration which alters `simple_instance_status` would need to define that
+   * `composite_instances` and `composite_item_details` inherited it. This must be provided in the
+   * migration as, without this, we cannot guarantee that a future `composite_instances` will refer
+   * to `simple_instance_status` in the same way.
+   *
+   * Note that this map needs to only contain relevant sources; it is not necessary to define every
+   * other source/composite.
+   *
+   * To define these relationships, return a map like:
+   * @example
+   * Map.of(
+   *  COMPOSITE_INSTANCES_ID, Map.of("inst_stat", SIMPLE_INSTANCE_STATUS_ID),
+   *  COMPOSITE_ITEM_DETAILS_ID, Map.of("instance_status", SIMPLE_INSTANCE_STATUS_ID)
+   * )
+   */
+  public Map<UUID, Map<String, UUID>> getEntityTypeSourceMaps() {
+    return Map.of();
+  }
+
   /**
    * Perform changes to fields and conditions within the FQL. This enables settings values for
    * the field's name, operator, and value together. Note that renaming fields must be done both
@@ -77,18 +101,26 @@ public MigratableQueryInformation additionalChanges(S state, MigratableQueryInfo
   }
 
   @Override
-  public final MigratableQueryInformation apply(MigratableQueryInformation query) {
+  public final MigratableQueryInformation apply(
+    MigratableQueryInformation query,
+    Map<UUID, Map<String, UUID>> customEntityTypeMappings
+  ) {
     S state = getStartingState();
 
+    Map<UUID, Map<String, UUID>> sourceMappings = new HashMap<>(getEntityTypeSourceMaps());
+    sourceMappings.putAll(customEntityTypeMappings);
+
     MigrationResult<String> fqlMigration = MigrationUtils.migrateFql(
       query.entityTypeId(),
       query.fqlQuery(),
-      f -> this.migrateFql(state, f)
+      f -> this.migrateFql(state, f),
+      sourceMappings
     );
     MigrationResult<List<String>> fieldsMigration = MigrationUtils.migrateFieldNames(
       query.entityTypeId(),
       query.fields(),
-      f -> this.migrateFieldName(state, f)
+      f -> this.migrateFieldName(state, f),
+      sourceMappings
     );
 
     return additionalChanges(

src/main/java/org/folio/fqm/migration/strategies/MigrationStrategy.java

@@ -1,5 +1,7 @@
 package org.folio.fqm.migration.strategies;
 
+import java.util.Map;
+import java.util.UUID;
 import org.folio.fqm.migration.MigratableQueryInformation;
 
 /**
@@ -21,5 +23,8 @@ public interface MigrationStrategy {
   /**
    * Migrate a query.
    */
-  MigratableQueryInformation apply(MigratableQueryInformation migratableQueryInformation);
+  MigratableQueryInformation apply(
+    MigratableQueryInformation migratableQueryInformation,
+    Map<UUID, Map<String, UUID>> customEntityTypeMappings
+  );
 }

src/main/java/org/folio/fqm/migration/strategies/impl/V23UserCreatedUpdatedDateFieldDeprecation.java

@@ -31,6 +31,14 @@ public String getLabel() {
     return "V23 -> V24 Legacy user created/updated dates deprecation (part of MODFQMMGR-1006)";
   }
 
+  @Override
+  public Map<UUID, Map<String, UUID>> getEntityTypeSourceMaps() {
+    return Map.ofEntries(
+      Map.entry(COMPOSITE_USERS_ID, Map.of("users", SIMPLE_USERS_ID)),
+      Map.entry(COMPOSITE_LOAN_DETAILS_ID, Map.of("users", SIMPLE_USERS_ID))
+    );
+  }
+
   @Override
   public Map<UUID, Map<String, String>> getFieldChanges() {
     return Map.ofEntries(

src/main/java/org/folio/fqm/migration/warnings/RemovedFieldWarning.java

@@ -6,6 +6,7 @@
 import lombok.EqualsAndHashCode;
 import lombok.RequiredArgsConstructor;
 import lombok.ToString;
+import org.folio.fqm.service.LocalizationService;
 import org.folio.spring.i18n.service.TranslationService;
 
 @Builder
@@ -31,7 +32,20 @@ public WarningType getType() {
 
   @Override
   public String getDescription(TranslationService translationService) {
-    return Warning.getDescriptionByAlternativeAndFql(translationService, this.getType(), field, fql, alternative);
+    String translationKey = LocalizationService.MIGRATION_WARNING_TRANSLATION_TEMPLATE.formatted(TYPE.toString());
+
+    if (fql == null) {
+      translationKey += ".field";
+    } else {
+      translationKey += ".query";
+    }
+    if (alternative == null) {
+      translationKey += ".withoutAlternative";
+    } else {
+      translationKey += ".withAlternative";
+    }
+
+    return translationService.format(translationKey, "name", field, "alternative", alternative, "fql", fql);
   }
 
   public static FieldWarningFactory withoutAlternative() {