I believe that pipelines being directly connected to the same database that the ArchiveBot backend is running on is actually harming ArchiveBot being able to be updated.
As such, I propose that instead of pipelines getting ssh keys, they get some kind of other X25519 based key that would be used to have them do signed HTTPS requests to a endpoint like archivebot.com/api/v/ which would be able to be properly secured, and have more decoupling logic, so it makes updates to the handing out jobs logic and similar easier.
For this we will need to document where specifically pipelines need the data from the database, and making three endpoints, one for getting a work item, one for pushing the current update messages back, and one for receiving job control messages.
We will need to redesign logging to make it not flow trough Redis, tracked in #613.
I believe that pipelines being directly connected to the same database that the ArchiveBot backend is running on is actually harming ArchiveBot being able to be updated.
As such, I propose that instead of pipelines getting ssh keys, they get some kind of other X25519 based key that would be used to have them do signed HTTPS requests to a endpoint like archivebot.com/api/v/ which would be able to be properly secured, and have more decoupling logic, so it makes updates to the handing out jobs logic and similar easier.
For this we will need to document where specifically pipelines need the data from the database, and making three endpoints, one for getting a work item, one for pushing the current update messages back, and one for receiving job control messages.
We will need to redesign logging to make it not flow trough Redis, tracked in #613.