RHCS-6003: Revert "Use synchronous LWCA KeyRetriever"

This reverts commit 451cc4d.

Making LWCA key retrieval causes hangs on servers to which
lightweight CA keys have not yet been successfully retrieved (e.g.
due to transient issue).  The problem arises as follows:

1. AuthorityMonitor thread observes creation of new CA via LDAP
   replication.
2. AuthorityMonitor initialises the CertificateAuthority object.
3. At SigningUnit initialisation, observes absence of keys and
   initiates key retrieval (synchronously)
4. Key retrieval fails; enters exponential backoff loop
5. CertificateAuthority initialisation is stuck in key retrieval
   backoff loop; it does not get added to the CA registry.
6. Separately, `AuthorityService` / `AuthorityServlet` receives a
   request related to the CA (e.g. retrieve authority info or cert
   chain).
7. Lookup via `AuthorityRepository` locates the data, but no
   `CertificateAuthority` object exists for it.  Starts the
   initialisation process afresh (and hangs).
8. Rinse, repeat.  Eventually even unrelated functions can be
   impacted due to resource starvation (e.g. no available HTTP
   worker threads).

The fix is to revert to asynchronous key retrieval.  Happily, the
earlier patch reverted cleanly.  The change to synchronous retrieval
was intended to resolve an [issue] with nondeterministic failures
when creating many LWCAs in rapid succession.  Further investigation
of that issue hints that the issue may be resource limit related
because LDAP connection failures affect both LWCA creation and key
retrieval.  We may need to do further investigation and
experimentation to solve that issue in a different way.

[issue]: dogtagpki#4677.

Fixes: https://issues.redhat.com/browse/RHCS-6003

Author: frasertweedale

base/ca/src/main/java/com/netscape/ca/CertificateAuthority.java

@@ -219,7 +219,7 @@ public class CertificateAuthority extends Subsystem implements IAuthority, IOCSP
     private ResponderID mResponderIDByName = null;
     private ResponderID mResponderIDByHash = null;
 
-    private KeyRetrieverRunner keyRetrieverRunner;
+    private Thread keyRetrieverThread;
 
     /**
      * Internal constants
@@ -649,7 +649,7 @@ public synchronized void startKeyRetriever() throws EBaseException {
             return;
         }
 
-        if (keyRetrieverRunner != null) {
+        if (keyRetrieverThread != null) {
             logger.info("CertificateAuthority: KeyRetriever already running for authority " + authorityID);
             return;
         }
@@ -687,16 +687,14 @@ public synchronized void startKeyRetriever() throws EBaseException {
             throw new EBaseException(e);
         }
 
-        // Use a synchronous KeyRetriever to ensure that the LWCA has
-        // the signing key before it can be used.
-        // https://github.com/dogtagpki/pki/issues/4677
+        KeyRetrieverRunner runner = new KeyRetrieverRunner(keyRetriever, this);
 
-        keyRetrieverRunner = new KeyRetrieverRunner(keyRetriever, this);
-        keyRetrieverRunner.run();
+        keyRetrieverThread = new Thread(runner, "KeyRetriever-" + authorityID);
+        keyRetrieverThread.start();
     }
 
     public synchronized void removeKeyRetriever() {
-        keyRetrieverRunner = null;
+        keyRetrieverThread = null;
     }
 
     public void initSigningUnits() throws Exception {