feat: Switch to devx-logs - #1631
Merged
Merged
Conversation
The `cdk-base` role enables https://github.com/guardian/devx-logs. See also https://github.com/guardian/amigo/tree/main/roles/cdk-base.
These logs will be picked up by fluentbit (via https://github.com/guardian/devx-logs) and forwarded to Kinesis for ingestion into Central ELK.
Contributor
Contributor
Contributor
Contributor
https://github.com/guardian/devx-logs requires additional ASG tags and the `ec2:DescribeInstances` IAM permission.
Logs are now sent to Central ELK via https://github.com/guardian/devx-logs.
akash1810
force-pushed
the
aa/devx-logs
branch
from
January 20, 2026 16:48
51a1e97 to
7b4c2bc
Compare
Contributor
Contributor
Contributor
Contributor
https://github.com/guardian/mobile-logstash-encoder hydrates logs with Stack, Stage, App markers before writing to Kinesis. This functionality is now covered by https://github.com/guardian/devx-logs so we can remove the dependency.
akash1810
force-pushed
the
aa/devx-logs
branch
from
January 20, 2026 18:02
7b4c2bc to
0b298b0
Compare
akash1810
commented
Jan 20, 2026
| amiEncrypted: true | ||
| amiTags: | ||
| Recipe: mobile-java11-jammy-ARM | ||
| Recipe: mobile-java11-jammy-ARM-with-cdk-base |
| templateStageParameters: | ||
| CODE: | ||
| SloSqsQueueArn: /notifications/CODE/mobile-notifications/notifications.queues.sloMonitoringArn | ||
| LoggingStreamName: /account/services/logging.stream.name.code |
Member
Author
There was a problem hiding this comment.
The CloudFormation template now has a LoggingStreamName parameter without a default set. This change tells Riff-Raff what value to use when deploying and means the entire change can be deployed with Riff-Raff.
| <encoder class="com.gu.mobile.logback.MobileLogstashEncoder"> | ||
| <defaultAppName>notifications</defaultAppName> | ||
| <encoder> | ||
| <pattern>%date [%thread] %-5level {%mdc} %logger{36}:%L - %msg%n%xException{15}</pattern> |
Member
Author
There was a problem hiding this comment.
This format was copied from guardian/prism's.
| { groupId = "com.typesafe.play", artifactId = "sbt-plugin", version = "2.9.1" }, | ||
|
|
||
| # Pin logstash to 1.8 until we have bumped simple-configuration | ||
| { groupId = "com.gu", artifactId = "mobile-logstash-encoder", version = "1.1.8" }, |
Member
Author
There was a problem hiding this comment.
This dependency has been removed.
| "org.postgresql" % "postgresql" % "42.7.7", | ||
| "ch.qos.logback" % "logback-core" % logbackVersion, | ||
| "ch.qos.logback" % "logback-classic" % logbackVersion, | ||
| "net.logstash.logback" % "logstash-logback-encoder" % "8.1" |
Member
Author
There was a problem hiding this comment.
Required to produce JSON formatted logs, as required by fluentbit (via devx-logs). The latest version (9.0) requires Java 17+. We're currently using Java 11.
Member
Author
|
These have all deployed to PROD now. Confirming that logs are appearing: 
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Easiest to review commit by commit.
What does this change?
This change switches the EC2 services' log shipping from the aws-kinesis-agent to fluentbit via https://github.com/guardian/devx-logs via AMIgo's
cdk-baserole. This has a few benefits, including:cloud-init-output.logfile, which includes logs from the user data. These can be helpful to understand why a service hasn't started, for example.Note
To enable this, I've created a new recipe in AMIgo as the existing recipe is also used by
guardian/mobile-entry, so we can't update it directly.How to test
reportregistrationnotificationdevx-logsadds aShippedByfield to each log, so we can use that to understand which logs come from this change, vs another build.Footnotes
https://github.com/guardian/mobile-logstash-encoder has a field name of
ec2_instance, howeverdevx-logsusesec2_instance_id. I'm not sure if any dashboards explicitly surface this field, so cannot be certain if the change is an issue. ↩