Autograder is currenlty set up to report all it's logs to a Loki container running on the EC2 instance. @leesjensen and I have decided that it's a bit overkill. It's only capturing logs at the moment, not metrics. Logs can be alternatively accessed with docker commands, and the Grafana image is on the same image as the autograder so if the EC2 has issues, Grafana is probably not going to catch it. We don't teach Grafana until CS329 so the odds of the new TAs understanding what it's doing are slim. Plus, the thing hasn't been updated in two years.
Furthermore, the logs for the Loki instances were not configured to rotate, which caused our disk space to fill up. While that has been fixed, I believe if the instance was remade for some reason the Docker configuration would reset. Configurating the rotating of the logs in the compose file would probably be the defense against this.
I plan to have this change ready by the Fall.
Autograder is currenlty set up to report all it's logs to a Loki container running on the EC2 instance. @leesjensen and I have decided that it's a bit overkill. It's only capturing logs at the moment, not metrics. Logs can be alternatively accessed with docker commands, and the Grafana image is on the same image as the autograder so if the EC2 has issues, Grafana is probably not going to catch it. We don't teach Grafana until CS329 so the odds of the new TAs understanding what it's doing are slim. Plus, the thing hasn't been updated in two years.
Furthermore, the logs for the Loki instances were not configured to rotate, which caused our disk space to fill up. While that has been fixed, I believe if the instance was remade for some reason the Docker configuration would reset. Configurating the rotating of the logs in the compose file would probably be the defense against this.
I plan to have this change ready by the Fall.