Prose Pod API as part of Prose Pod Server

[RemiBardon]

Context

Up until now, the Prose Pod API has always been a separate Prose Pod component, with its own repository and Docker image. For reasons explained in “Prose Pod Server architecture: Server API vs XMPP server”, some of the Prose Pod API logic was recently moved to the Prose Pod Server. The argument for keeping both HTTP APIs separate was mainly to reduce XMPP server downtime when running updates (for higher availability). This new architecture came with a lot of great benefits, but it also came with downsides:

TL;DR: It’s annoying for devs, annoying for ops, it will likely cause more downtime… there’s very few positive aspects in fact. Here is an overview:

Summary:

• Changes in the Pod API need changes in the Pod Server. Updating one requires updating the other. Instead of reducing downtime it increases the potential for operational error which would cause prolongated downtime. → Bad UX, bad OX, more time spent on customer support, bad reviews…
• Implementing features is so much more annoying. → Lower velocity, bad DX…
• Both projects have the same tech stack, which requires 2 hours to build. Each Pod API release takes at least 4h30, which is unacceptable. → Lower incident response time, bad DX, wasted resources…
• And a lot more problems.

Details (more exhaustive): [tap to reveal]

• At the moment, most features in the Prose Pod API are still unique/innovative. They require changes on the Pod Server almost every time.
  • In order for a new Pod API version to work, we therefore need to publish a new Pod Server version. Because bugs can happen, we should also make the release through a Pull Request to ensure all integration tests pass. But for it to pass we need to wait for the Pod Server release to be published first. Rust code takes a long time to compile so now we’re talking about:
    1. Pod Server release build (~2h!)
    2. Pod API CI tests in PR (~12min)
    3. Pod API release build (~2h!)
  • So every time we need to make a small change, we have to wait around 4h30, and be present to merge the PRs, make commits and push to trigger the builds… that’s too much.
  • Every new commit on master also triggers an edge release. It’s built with less optimizations so it’s faster but it still takes around 1h10 for the Pod API. That’s wasted computing power and energy.
• The Prose Pod API and the Prose Pod Server API use the same tech stack, which means the same libraries are compiled twice in two different projects. Last time I checked, and if I remember correctly, GitHub did not provide a way to share build cache across repositories. Most of our compile time comes from dependencies (HTTP server and encryption); that’s a lot of wasted time and resources for something that could be cached.
  • Cache might not work as expected with builds of -lite images because Rust busts cache if a single feature flag changes.
• In addition to those changes which take time, we also suffer from numerous layers of API. The XMPP server exposes multiple APIs, which the Pod Server API uses to expose its own HTTP API, that the Pod API uses to expose its own HTTP API, which the Dashboard finally uses! There are both historical and technical reasons for it, but maybe this should be flattened until real-world issues require us to go back to this split (if ever!).
  • Each API requires design, type definitions, authorization, implementation, client implementation, parsing, tests, etc. This adds development time and reduces velocity.
  • Each layer adds serialization and deserialization overhead, which uses more resources and slows down request processing. To reduce HTTP requests, we sometimes implement caches, but then we have to manage those caches properly, which introduces potential bugs and security issues.
  • It doesn’t add much of a security layer as the Server API is also authorized properly (defense in depth, no fortress) and uses the same tech stack. If anything, it reduces security as security fixes might be implemented in one project but not the other.
  • For example, analytics and software update forwarding means: a request to the Pod API, which proxies to Server API, which proxies to Cloud API. Thankfully we are using Rust and can do all of that in streaming fashion, but it adds overhead and latency. We’d remove one step and have a single-node proxy if the Pod API was exposed by the Pod Server.
• Lifecycle operations are quadratically more complex, as Pod Server and Pod API instances can be restarted/reloaded independently. For better UX, both are configured with the same configuration file, which makes inconsistent states possible (i.e. config reloaded in one bot not the other). This configuration file has to be parsed by both projects and some keys are read by both so there has to be some shared library somewhere to avoid discrepancies. This is one more annoying thing to do that takes dev time and energy.
• Backup & restore is harder to implement, as the Pod API has its own source of truth (database) and lifecycle. Ideally we would make the Pod API stateless by using the Pod Server as source of truth, but that would mean either having an abstract storage system there or having to make changes to the Pod Server every time we change a feature in the Pod API.
• It’s harder to know what is handled where, and know where to implement new features. Documentation also gets duplicated/split.
• The Pod API is tightly coupled to Pod Server versions, and updating one without the other will likely result in downtime. Because there are multiple parts involved, debugging will take more time, resulting in more XMPP downtime than if the Server was itself updated more often but alone. In addition, this kind of debugging is a very bad UX for Prose Pod operators.
• The error handling logic has been improved in the Pod Server API compared to the Pod API, but the Pod API hasn’t benefited from those changes yet for API stability and coherence reasons.

Decision

Over the next months, we will progressively move logic from the Pod API to the Pod Server API. The architecture won’t change, the Pod API will act as a relay. The prose-pod-api repository has a lot of tooling related to testing and demo; this will take time to merge in prose-pod-server.

After all routes have been moved to the Pod Server API, we will move all of the tooling out of prose-pod-api, and create a clean “mono repository”. The prose-pod-api repository will be archived, and its open issues and discussions will be moved over to the prose-pod-server repository. When that’s done, we will archive the prose-pod-api repository and mention where it lives now.

Consequences

This new architecture will fix all of the issues listed in “Context”. It will greatly improve the overall developer experience of the Prose backend and make code more readable/auditable. Prose Pods will also become easier to deploy and operate. Here are some other consequences (non-exhaustive):

• I expect a massive reduction in number of Lines of Code
  • Less code to understand, less code to maintain, less code to build
  • No more duplicated code, no more glue code
• All Prose Pod API updates will require a restart of the Prose Pod Server, but as explained earlier this is an acceptable tradeoff.
• The Prose Pod Server used to expose internal-only routes; those will disappear (less breaking API changes).
• People deploying Prose without a Dashboard will now be forced to run the Pod API.
  • Deploying a Pod Server without having a Pod API to interact with it doesn’t make much sense. Why use Prose in the first place then?

[RemiBardon]

Would fix #326 (not relevant anymore).

[RemiBardon]

When upgrading to the version where the Pod API is part of the Pod Server, ops would have to move Prose data around in order for the Pod Server to have access to the Pod API’s database. While we could do it in a backward-compatible manner, this would lead to confusion about what data goes where. To simplify deployments, backups, etc., we should put all data in the /var/lib/prose-pod-server directory.

At startup, if the Pod Server doesn’t find database.sqlite it means data hasn’t been migrated correctly. It should then print a clear error message explaining what to do and pointing to the docs, then crash. We don’t cant to add extra logic in the Pod Server, just a plain old crash with instructions for the operator. They’ll do it better (e.g. if they have a non-standard setup).