URL signing secret: extra polish and clarificatoin

Author: ErykKul

conf/keycloak/docker-compose-dev.yml

@@ -31,6 +31,7 @@ services:
       DATAVERSE_FEATURE_API_BEARER_AUTH_USE_BUILTIN_USER_ON_ID_MATCH: "1"
       DATAVERSE_MAIL_SYSTEM_EMAIL: "dataverse@localhost"
       DATAVERSE_MAIL_MTA_HOST: "smtp"
+      DATAVERSE_API_SIGNING_SECRET: "dev-only-signing-secret-change-me"
       DATAVERSE_AUTH_OIDC_ENABLED: "1"
       DATAVERSE_AUTH_OIDC_CLIENT_ID: test
       DATAVERSE_AUTH_OIDC_CLIENT_SECRET: 94XHrfNRwXsjqTqApRrwWmhDLDHpIYV8

doc/release-notes/fix-url-signing-special-characters.md

@@ -19,6 +19,12 @@ they did before 6.10 keep working unchanged, signatures are computed the same wa
 regression, and URLs containing special characters validate again. No client-side changes are
 required.
 
+Note that, as before 6.10, the `url` submitted to `/api/admin/requestSignedUrl` (and any URL consumed
+as a signed URL) must be in its URL-decoded form: the signature is computed over the URL exactly as
+provided, but the server validates it against the URL-decoded request. Passing a percent-encoded URL
+(e.g. `persistentId=doi%3A10.5072%2FFK2%2FABC` instead of `persistentId=doi:10.5072/FK2/ABC`) and then
+using it verbatim will therefore fail validation.
+
 ### A signing secret is now required for signed URLs
 
 Separately from the fix above, Dataverse no longer falls back to a weak signing key when
@@ -31,9 +37,13 @@ URLs are signed with a key based on a user's API token:
   guestbook-response download endpoints under `/api/access` (`datafile/{id}`, `datafiles/{ids}`,
   `dataset/{id}` and `dataset/{id}/versions/{versionId}`) - return an error instead of issuing a
   weakly-signed URL.
-- Signed callbacks built internally (external tool launches, Globus transfers) are sent unsigned,
-  with a warning logged, rather than weakly signed; such unsigned URLs cannot be used to access
-  draft datasets or restricted files.
+- External tool launches send their callback unsigned (with a warning logged) rather than weakly
+  signed. Such an unsigned callback only allows anonymous access to public data; it cannot be used to
+  access draft datasets or restricted files.
+- Globus transfers are effectively disabled without a signing secret: the callbacks the
+  dataverse-globus app relies on require an authenticated, signed request, so Globus upload (and
+  download of restricted or unpublished files) fails with an authorization error. Installations that
+  use Globus must set `dataverse.api.signing-secret`.
 - The permissions-history CSV download links on the permission management pages are not offered
   (a warning is logged).
 

doc/sphinx-guides/source/api/native-api.rst

@@ -9079,7 +9079,7 @@ Signed URLs were developed to support External Tools but may be useful in other
 This API call allows a Dataverse superUser to generate a signed URL for such scenarios.
 The JSON input parameter required is an object with the following keys:
 
-- ``url`` - the exact URL to sign, including api version number and all query parameters
+- ``url`` - the exact URL to sign, including api version number and all query parameters. Provide it in its URL-decoded form (for example ``persistentId=doi:10.5072/FK2/ABC``, not ``persistentId=doi%3A10.5072%2FFK2%2FABC``): the signature is computed over the URL exactly as provided, but Dataverse validates it against the URL-decoded request, so a percent-encoded URL used verbatim will fail validation.
 - ``timeOut`` - how long in minutes the signature should be valid for, default is 10 minutes
 - ``httpMethod`` - which HTTP method is required, default is GET
 - ``user`` - the user identifier for the account associated with this signature, the default is the superuser making the call. The API call will succeed/fail based on whether the specified user has the required permissions. 

doc/sphinx-guides/source/developers/globus-api.rst

@@ -78,7 +78,7 @@ Note that while Dataverse will not add files that violate the size or quota rule
 they intend to transfer before submitting a transfer request to Globus.
 
 The getDatasetMetadata and getFileListing URLs are just signed versions of the standard Dataset metadata and file listing API calls. The other two are Globus specific.
-Note that these URLs are only signed when a non-empty :ref:`dataverse.api.signing-secret` is configured on the Dataverse installation; without it, they are sent unsigned (a warning is logged) and cannot be used to access draft datasets or restricted files.
+Note that these URLs are only signed when a non-empty :ref:`dataverse.api.signing-secret` is configured on the Dataverse installation. Without it the callbacks are sent unsigned (a warning is logged), and because the Globus callback endpoints (``globusUploadParameters``, ``requestGlobusUploadPaths``, ``addGlobusFiles``, and the download equivalents for restricted or unpublished files) require an authenticated, signed request, the Globus transfer flow fails with an authorization error; only anonymous download of public files can proceed. In practice, Globus transfer support requires ``dataverse.api.signing-secret`` to be set.
 
 If called for a dataset using a store that is configured with a remote Globus endpoint(s), the return response is similar but the response includes a
 the "managed" parameter will be false, the "endpoint" parameter is replaced with a JSON array of "referenceEndpointsWithPaths" and the

docker/compose/demo/compose.yml

@@ -16,6 +16,9 @@ services:
       DATAVERSE_FEATURE_API_BEARER_AUTH: "1"
       DATAVERSE_MAIL_SYSTEM_EMAIL: "Demo Dataverse <dataverse@example.org>"
       DATAVERSE_MAIL_MTA_HOST: "smtp"
+      # Required for signed URLs (external tools, Globus transfers, signed guestbook downloads,
+      # rdm-integration). Change this to a private value; see the Configuration Guide.
+      DATAVERSE_API_SIGNING_SECRET: "demo-only-signing-secret-change-me"
       JVM_ARGS: -Ddataverse.files.storage-driver-id=file1
         -Ddataverse.files.file1.type=file
         -Ddataverse.files.file1.label=Filesystem

src/main/java/edu/harvard/iq/dataverse/api/Access.java

@@ -548,6 +548,13 @@ private Map<Long, DataFile> getDatafilesMap(DataverseRequest req, String fileIds
     }
 
     private Response returnSignedUrl(ContainerRequestContext crc, UriInfo uriInfo, User user, String id, String gbrids) {
+        // Record the guestbook-response id for the side-effect-only callers
+        // (datafileBundleWithGuestbookResponse, postDownloadDatafiles) that discard the Response we
+        // return and then read this property to drive the actual download. This must happen even when
+        // no signing secret is configured and we bail out below, otherwise those downloads lose the
+        // guestbook response that processDatafileWithGuestbookResponse just saved (leading to a
+        // spurious "guestbookResponseMissing" 400 or a duplicate guestbook/MakeDataCount entry).
+        crc.setProperty("gbrids", gbrids);
         // The signed-URL endpoints already check this before any guestbook side effects (see
         // requireSigningSecretForSignedUrl). This check protects the callers that invoke
         // processDatafileWithGuestbookResponse only for its guestbook side effects and discard this
@@ -587,7 +594,6 @@ private Response returnSignedUrl(ContainerRequestContext crc, UriInfo uriInfo, U
             builder.replaceQueryParam("gbrids", gbrids);
         }
         builder.replaceQueryParam("persistentId", null); // remove this as a parm and add the id to the path
-        crc.setProperty("gbrids", gbrids);
         String baseUrlEncoded = builder.build().toString();
         String baseUrl = URLDecoder.decode(baseUrlEncoded, StandardCharsets.UTF_8);
         baseUrl = baseUrl.replace(":persistentId", id);

src/main/java/edu/harvard/iq/dataverse/api/Admin.java

@@ -2459,30 +2459,37 @@ public Response getSignedUrl(@Context ContainerRequestContext crc, JsonObject ur
                     "Requesting signed URLs requires a signing secret to be configured. Please set the dataverse.api.signing-secret JVM option.");
         }
 
-        String userId = urlInfo.getString("user");
-        String key=null;
-        if (userId != null) {
-            AuthenticatedUser user = authSvc.getAuthenticatedUser(userId);
-            // If a user param was sent, we sign the URL for them, otherwise on behalf of
-            // the superuser who made this api call
-            if (user != null) {
-                ApiToken apiToken = authSvc.findApiTokenByUser(user);
-                if (apiToken != null && !apiToken.isExpired() && !apiToken.isDisabled()) {
-                    key = apiToken.getTokenString();
-                }
-            } else {
-                userId = superuser.getUserIdentifier();
-                // We ~know this exists - the superuser just used it and it was unexpired/not
-                // disabled. (ToDo - if we want this to work with workflow tokens (or as a
-                // signed URL), we should do more checking as for the user above))
-                key = authSvc.findApiTokenByUser(superuser).getTokenString();
+        // "url" is required; "user" defaults to the superuser making the call (see the docs for this
+        // endpoint). Use the defaulted accessors: JsonObject.getString(name) throws NullPointerException
+        // when the key is absent.
+        String baseUrl = urlInfo.getString("url", null);
+        if (baseUrl == null) {
+            return error(Response.Status.BAD_REQUEST, "Required parameter 'url' is missing.");
+        }
+        String userId = urlInfo.getString("user", null);
+
+        String key = null;
+        AuthenticatedUser signingUser = (userId != null) ? authSvc.getAuthenticatedUser(userId) : null;
+        if (signingUser != null) {
+            // A known user was requested: sign the URL for them.
+            ApiToken apiToken = authSvc.findApiTokenByUser(signingUser);
+            if (apiToken != null && !apiToken.isExpired() && !apiToken.isDisabled()) {
+                key = apiToken.getTokenString();
             }
-            if (key == null) {
-                return error(Response.Status.CONFLICT, "Do not have a valid user with apiToken");
+        } else {
+            // No user param, or an unknown one: sign on behalf of the superuser who made this API call.
+            userId = superuser.getUserIdentifier();
+            // The superuser just authenticated, but that does not guarantee an API token exists (e.g.
+            // bearer-token or session auth), so null-check rather than dereference blindly.
+            ApiToken apiToken = authSvc.findApiTokenByUser(superuser);
+            if (apiToken != null) {
+                key = apiToken.getTokenString();
             }
         }
+        if (key == null) {
+            return error(Response.Status.CONFLICT, "Do not have a valid user with apiToken");
+        }
 
-        String baseUrl = urlInfo.getString("url");
         int timeout = urlInfo.getInt(URLTokenUtil.TIMEOUT, 10);
         String method = urlInfo.getString(URLTokenUtil.HTTP_METHOD, "GET");
 

src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java

@@ -4,7 +4,6 @@
 import edu.harvard.iq.dataverse.authorization.users.*;
 import edu.harvard.iq.dataverse.privateurl.PrivateUrl;
 import edu.harvard.iq.dataverse.privateurl.PrivateUrlServiceBean;
-import edu.harvard.iq.dataverse.settings.JvmSettings;
 import edu.harvard.iq.dataverse.util.UrlSignerUtil;
 
 import jakarta.inject.Inject;
@@ -52,6 +51,14 @@ private String getSignedUrlRequestParameter(ContainerRequestContext containerReq
 
     private User getAuthenticatedUserFromSignedUrl(ContainerRequestContext containerRequestContext) {
         User user = null;
+        // Without a signing secret we never issue signed URLs (signUrlWithApiKey refuses on the sign
+        // side), so we must not accept them here either. Otherwise a bare API token - or, for a guest,
+        // the public request URL - would be enough to forge a URL whose signature validates against the
+        // "" + token key computed below. Reject so findUserFromRequest returns the standard 401.
+        if (!UrlSignerUtil.isSigningSecretConfigured()) {
+            logger.warning("Rejecting signed URL authentication: no signing secret configured (dataverse.api.signing-secret).");
+            return null;
+        }
         // The signedUrl contains a param telling which user this is supposed to be for.
         // We don't trust this. So we look up that user, and get their API key, and use
         // that as a secret in validating the signedURL. If the signature can't be
@@ -60,6 +67,11 @@ private User getAuthenticatedUserFromSignedUrl(ContainerRequestContext container
         // If User is Guest we can return a generic guest user with key made from URI
         UriInfo uriInfo = containerRequestContext.getUriInfo();
         String userId = uriInfo.getQueryParameters().getFirst(SIGNED_URL_USER);
+        if (userId == null) {
+            // A token param was present (that is why this mechanism ran) but no user param: this can
+            // never be a URL we signed, and dereferencing userId below would throw a NullPointerException.
+            return null;
+        }
         User targetUser = null;
         ApiToken userApiToken = null;
         if (userId.equalsIgnoreCase("guest")) {
@@ -90,7 +102,7 @@ private User getAuthenticatedUserFromSignedUrl(ContainerRequestContext container
             }
 
             String requestMethod = containerRequestContext.getMethod();
-            String signedUrlSigningKey = JvmSettings.API_SIGNING_SECRET.lookupOptional().orElse("") + userApiToken.getTokenString();
+            String signedUrlSigningKey = UrlSignerUtil.getApiSigningKey(userApiToken.getTokenString());
             boolean isSignedUrlValid = UrlSignerUtil.isValidUrl(signedUrl, userId, requestMethod, signedUrlSigningKey);
             if (isSignedUrlValid) {
                 user = targetUser;

src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java

@@ -110,12 +110,8 @@ public String handleRequest(boolean preview) {
                             + externalTool.getId();
                 }
                 if (apiToken != null) {
-                    if (UrlSignerUtil.isSigningSecretConfigured()) {
-                        callback = UrlSignerUtil.signUrlWithApiKey(callback, 5, apiToken.getAuthenticatedUser().getUserIdentifier(),
-                                HttpMethod.GET, apiToken.getTokenString());
-                    } else {
-                        logger.warning("Cannot sign external tool callback: no signing secret configured (dataverse.api.signing-secret). Sending an unsigned callback.");
-                    }
+                    callback = UrlSignerUtil.trySignUrlWithApiKey(callback, 5, apiToken.getAuthenticatedUser().getUserIdentifier(),
+                            HttpMethod.GET, apiToken.getTokenString(), "external tool callback");
                 }
                 paramsString= "?callback=" + Base64.getEncoder().encodeToString(StringUtils.getBytesUtf8(callback));
                 if (getLocaleCode() != null) {

src/main/java/edu/harvard/iq/dataverse/globus/GlobusServiceBean.java

@@ -779,11 +779,12 @@ public String getGlobusAppUrlForDataset(Dataset d, boolean upload, List<DataFile
         if (apiToken == null) {
             // Shouldn't happen
             logger.warning("Unable to get api token for user: " + user.getIdentifier());
-        } else if (UrlSignerUtil.isSigningSecretConfigured()) {
-            callback = UrlSignerUtil.signUrlWithApiKey(callback, 5, apiToken.getAuthenticatedUser().getUserIdentifier(),
-                    HttpMethod.GET, apiToken.getTokenString());
         } else {
-            logger.warning("Cannot sign Globus callback: no signing secret configured (dataverse.api.signing-secret). Sending an unsigned callback.");
+            // Note: without a signing secret the callback is sent unsigned, which the dataverse-globus
+            // app cannot use - the Globus upload/download callback endpoints require an authenticated,
+            // signed request. Globus transfers therefore require dataverse.api.signing-secret.
+            callback = UrlSignerUtil.trySignUrlWithApiKey(callback, 5, apiToken.getAuthenticatedUser().getUserIdentifier(),
+                    HttpMethod.GET, apiToken.getTokenString(), "Globus callback");
         }
         appUrl = appUrl + "&callback=" + Base64.getEncoder().encodeToString(StringUtils.getBytesUtf8(callback));