Create valid temporal ranges when updating collection extents

[anayeaye]

Note I initially drafted this issue thinking that the temporal ranges were causing stac-browser incompatibility for the dev disasters instance but it turns out there is also a cloudfront behavior problem. If the cloudfront behavior resolution makes the root stac api compatible with stac-browser, we can close this issue. Related issue: https://github.com/NASA-IMPACT/veda-architecture/issues/754

What

Automated updates to stac collection temporal extent is creating invalid stac metadata. This is currently happening in a custom SQL function which should be improved or replaced with a different technique for updating the temporal range of a collection to agree with the true range of item dates.

    "temporal": {
      "interval": [
        [
          "2023-01-15 00:00:00+00",
          "2024-08-08 23:59:59+00"
        ]
      ]
    }

should be

    "temporal": {
      "interval": [
        [
          "2023-01-15T00:00:00Z",
          "2024-08-08T23:59:59Z"
        ]
      ]
    }

update extents current SQL

veda-backend/database/runtime/handler.py

Lines 148 to 190 in 7eaa5b3

	def create_collection_extents_functions(cursor) -> None:
	"""
	Functions to update spatial and temporal extents off all items in a collection
	This is slightly different from the inbuilt pgstac.update_collection_extents function which describes the range of nominal datetimes,
	here we capture the maximum range which must include max end datetime.
	"""
	collection_temporal_extent_max_sql = """
	CREATE OR REPLACE FUNCTION dashboard.collection_temporal_extent_max(id text) RETURNS jsonb
	LANGUAGE sql
	IMMUTABLE PARALLEL SAFE
	SET search_path TO 'pgstac', 'public'
	AS $function$
	SELECT to_jsonb(array[array[min(datetime)::text, max(end_datetime)::text]])
	FROM items WHERE collection=$1;
	$function$
	;
	"""
	cursor.execute(sql.SQL(collection_temporal_extent_max_sql))
	update_collection_extents_max_sql = """
	CREATE OR REPLACE FUNCTION dashboard.update_collection_extents_max(id text)
	RETURNS void
	LANGUAGE sql
	SET search_path TO 'pgstac', 'public'
	AS $function$
	UPDATE collections SET
	content = content ||
	jsonb_build_object(
	'extent', jsonb_build_object(
	'spatial', jsonb_build_object(
	'bbox', collection_bbox(collections.id)
	),
	'temporal', jsonb_build_object(
	'interval', dashboard.collection_temporal_extent_max(collections.id)
	)
	)
	)
	WHERE collections.id=$1;
	$function$
	;
	"""
	cursor.execute(sql.SQL(update_collection_extents_max_sql))

datetime formatting in datetime summries

veda-backend/database/runtime/handler.py

Lines 204 to 208 in 7eaa5b3

	SELECT to_jsonb(
	array[
	to_char(min(datetime) at time zone 'Z', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
	to_char(max(end_datetime) at time zone 'Z', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
	])​

ingest-api extent update event

Collection temporal and spatial extents are updated when the ingestions api is used to enqueue and publish items to an existing collection

veda-backend/ingest_api/runtime/src/vedaloader.py

Lines 23 to 36 in 7eaa5b3

	# First update the spatial and temporal extents for all item records for the collection
	logger.info(f"Updating extents for collection: {collection_id}.")
	cur.execute(
	"SELECT dashboard.update_collection_extents_max(%s)",
	(collection_id,),
	)
	# Next update default summaries which use the collection temporal extent for summaries of periodic items
	logger.info(
	f"Updating dashboard summaries for collection: {collection_id}."
	)
	cur.execute(
	"SELECT dashboard.update_collection_default_summaries(%s)",
	(collection_id,),

AC

• first check to see if upgrading pgstac version will resolve this problem!
• when ingestions/ endpoint enqueued veda data load completes, the temporal extent update produces valid RFC3339 dates
• propose and ticket solution for identifying and correcting existing invalid collection temporal ranges

[anayeaye]

This format may actually be fine, I just read the fine print (some RFC3339 checkers still fail when a space is used instead of 'T'). I am going to close this for now but we may find some STAC ecosystem tools require the T.

https://datatracker.ietf.org/doc/html/rfc3339#section-5.6

NOTE: ISO 8601 defines date and time separated by "T".
Applications using this syntax may choose, for the sake of
readability, to specify a full-date and full-time separated by
(say) a space character.

[anayeaye]

@vgeorge @gadomski I can re-open this ticket to add the T to the temporal extents if my interpretation above (#590 (comment)) is wrong--on reread it sounds more like it is OK to not show the T but perhaps it is still required in the record.

[anayeaye]

Background: we added this function because the default collection extents for collections items that spanned a full year did not extend across the full year but this may have been updated in pgstac since we created a max end datetime extent UDF (where the T was mistakenly dropped).

[gadomski]

I can re-open this ticket to add the T to the temporal extents if my interpretation above (#590 (comment)) is wrong--on reread it sounds more like it is OK to not show the T but perhaps it is still required in the record.

I would recommend having a T. The json-schema date-time type requires it, so many validators will fail if you don't.

I agree that this text is confusing from RFC 3339:

NOTE: ISO 8601 defines date and time separated by "T".
Applications using this syntax may choose, for the sake of
readability, to specify a full-date and full-time separated by
(say) a space character.

However, most readings of of the RFC stop here and required the T: date-time = full-date "T" full-time.