Skip to content

feat: Switch to devx-logs #1631

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
akash1810 merged 5 commits into from
Jan 21, 2026
Merged

feat: Switch to devx-logs #1631

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
29abd4b
feat: Use AMI that includes `cdk-base` role
akash1810 Jan 20, 2026
8df1738
feat: Update EC2 apps to log to stdout
akash1810 Jan 20, 2026
1106378
feat: Meet devx-logs requirements
akash1810 Jan 20, 2026
2e3471a
feat: Remove `aws-kinesis-agent`
akash1810 Jan 20, 2026
0b298b0
chore(deps): Remove unused `mobile-logstash-encoder`
akash1810 Jan 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .scala-steward.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,4 @@ updates.pin = [

# Pin Play framework to 2.9 until we've migrated the code from Akka to Pekko
{ groupId = "com.typesafe.play", artifactId = "sbt-plugin", version = "2.9.1" },

# Pin logstash to 1.8 until we have bumped simple-configuration
{ groupId = "com.gu", artifactId = "mobile-logstash-encoder", version = "1.1.8" },

@akash1810 akash1810 Jan 20, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dependency has been removed.

]
2 changes: 1 addition & 1 deletion build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,11 +118,11 @@ lazy val common = project
"org.tpolecat" %% "doobie-specs2" % doobieVersion % Test,
"org.tpolecat" %% "doobie-scalatest" % doobieVersion % Test,
"org.tpolecat" %% "doobie-h2" % doobieVersion % Test,
"com.gu" %% "mobile-logstash-encoder" % "1.1.8",
"com.gu" %% "simple-configuration-ssm" % simpleConfigurationVersion,
"org.postgresql" % "postgresql" % "42.7.7",
"ch.qos.logback" % "logback-core" % logbackVersion,
"ch.qos.logback" % "logback-classic" % logbackVersion,
"net.logstash.logback" % "logstash-logback-encoder" % "8.1"

@akash1810 akash1810 Jan 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required to produce JSON formatted logs, as required by fluentbit (via devx-logs). The latest version (9.0) requires Java 17+. We're currently using Java 11.

),
fork := true,
startDynamoDBLocal := startDynamoDBLocal.dependsOn(Test / compile).value,
Expand Down
52 changes: 48 additions & 4 deletions cdk/lib/__snapshots__/registration.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,14 @@ exports[`The Registration stack matches the snapshot for CODE 1`] = `
"Description": "The HostedZone, should contain the trailing dot zone.example.com.",
"Type": "String",
},
"LoggingStreamName": {
"AllowedValues": [
"/account/services/logging.stream.name.code",
"/account/services/logging.stream.name",
],
"Description": "SSM parameter containing the Name (not ARN) on the kinesis stream",
"Type": "AWS::SSM::Parameter::Value<String>",
},
"NotEnough200sPerDayThreshold": {
"Description": "Alarm if less than too many 200s. This value was based on just below 2 standard deviations from the mean over 6 weeks of data.",
"Type": "Number",
Expand Down Expand Up @@ -156,7 +164,10 @@ exports[`The Registration stack matches the snapshot for CODE 1`] = `
},
},
{
"Action": "ec2:DescribeTags",
"Action": [
"ec2:DescribeTags",
"ec2:DescribeInstances",
],
"Effect": "Allow",
"Resource": "*",
},
Expand Down Expand Up @@ -729,6 +740,13 @@ exports[`The Registration stack matches the snapshot for CODE 1`] = `
"PropagateAtLaunch": true,
"Value": "guardian/mobile-n10n",
},
{
"Key": "LogKinesisStreamName",
"PropagateAtLaunch": true,
"Value": {
"Ref": "LoggingStreamName",
},
},
{
"Key": "Stack",
"PropagateAtLaunch": true,
Expand All @@ -739,6 +757,11 @@ exports[`The Registration stack matches the snapshot for CODE 1`] = `
"PropagateAtLaunch": true,
"Value": "CODE",
},
{
"Key": "SystemdUnit",
"PropagateAtLaunch": true,
"Value": "registration.service",
},
],
"VPCZoneIdentifier": {
"Ref": "PrivateSubnets",
Expand Down Expand Up @@ -781,7 +804,6 @@ exports[`The Registration stack matches the snapshot for CODE 1`] = `
"#!/bin/bash -ev
aws --region \${AWS::Region} s3 cp s3://\${DistBucket}/\${Stack}/\${Stage}/\${App}/\${App}_1.0-latest_all.deb /tmp
dpkg -i /tmp/\${App}_1.0-latest_all.deb
/opt/aws-kinesis-agent/configure-aws-kinesis-agent \${AWS::Region} mobile-log-aggregation-\${Stage} /var/log/\${App}/application.log
",
{
"App": {
Expand Down Expand Up @@ -907,6 +929,14 @@ exports[`The Registration stack matches the snapshot for PROD 1`] = `
"Description": "The HostedZone, should contain the trailing dot zone.example.com.",
"Type": "String",
},
"LoggingStreamName": {
"AllowedValues": [
"/account/services/logging.stream.name.code",
"/account/services/logging.stream.name",
],
"Description": "SSM parameter containing the Name (not ARN) on the kinesis stream",
"Type": "AWS::SSM::Parameter::Value<String>",
},
"NotEnough200sPerDayThreshold": {
"Description": "Alarm if less than too many 200s. This value was based on just below 2 standard deviations from the mean over 6 weeks of data.",
"Type": "Number",
Expand Down Expand Up @@ -987,7 +1017,10 @@ exports[`The Registration stack matches the snapshot for PROD 1`] = `
},
},
{
"Action": "ec2:DescribeTags",
"Action": [
"ec2:DescribeTags",
"ec2:DescribeInstances",
],
"Effect": "Allow",
"Resource": "*",
},
Expand Down Expand Up @@ -1560,6 +1593,13 @@ exports[`The Registration stack matches the snapshot for PROD 1`] = `
"PropagateAtLaunch": true,
"Value": "guardian/mobile-n10n",
},
{
"Key": "LogKinesisStreamName",
"PropagateAtLaunch": true,
"Value": {
"Ref": "LoggingStreamName",
},
},
{
"Key": "Stack",
"PropagateAtLaunch": true,
Expand All @@ -1570,6 +1610,11 @@ exports[`The Registration stack matches the snapshot for PROD 1`] = `
"PropagateAtLaunch": true,
"Value": "PROD",
},
{
"Key": "SystemdUnit",
"PropagateAtLaunch": true,
"Value": "registration.service",
},
],
"VPCZoneIdentifier": {
"Ref": "PrivateSubnets",
Expand Down Expand Up @@ -1612,7 +1657,6 @@ exports[`The Registration stack matches the snapshot for PROD 1`] = `
"#!/bin/bash -ev
aws --region \${AWS::Region} s3 cp s3://\${DistBucket}/\${Stack}/\${Stage}/\${App}/\${App}_1.0-latest_all.deb /tmp
dpkg -i /tmp/\${App}_1.0-latest_all.deb
/opt/aws-kinesis-agent/configure-aws-kinesis-agent \${AWS::Region} mobile-log-aggregation-\${Stage} /var/log/\${App}/application.log
",
{
"App": {
Expand Down
16 changes: 12 additions & 4 deletions cdk/lib/__snapshots__/report.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,11 @@ exports[`The Report stack matches the snapshot for CODE 1`] = `
"PropagateAtLaunch": true,
"Value": "CODE",
},
{
"Key": "SystemdUnit",
"PropagateAtLaunch": true,
"Value": "report.service",
},
],
"TargetGroupARNs": [
{
Expand Down Expand Up @@ -1616,8 +1621,7 @@ aws s3 cp 's3://",
"Ref": "DistributionBucketName",
},
"/mobile-notifications/CODE/report/report_1.0-latest_all.deb' '/report/report_1.0-latest_all.deb'
dpkg -i /report/report_1.0-latest_all.deb
/opt/aws-kinesis-agent/configure-aws-kinesis-agent eu-west-1 mobile-log-aggregation-CODE /var/log/report/application.log",
dpkg -i /report/report_1.0-latest_all.deb",
],
],
},
Expand Down Expand Up @@ -1821,6 +1825,11 @@ exports[`The Report stack matches the snapshot for PROD 1`] = `
"PropagateAtLaunch": true,
"Value": "PROD",
},
{
"Key": "SystemdUnit",
"PropagateAtLaunch": true,
"Value": "report.service",
},
],
"TargetGroupARNs": [
{
Expand Down Expand Up @@ -3287,8 +3296,7 @@ aws s3 cp 's3://",
"Ref": "DistributionBucketName",
},
"/mobile-notifications/PROD/report/report_1.0-latest_all.deb' '/report/report_1.0-latest_all.deb'
dpkg -i /report/report_1.0-latest_all.deb
/opt/aws-kinesis-agent/configure-aws-kinesis-agent eu-west-1 mobile-log-aggregation-PROD /var/log/report/application.log",
dpkg -i /report/report_1.0-latest_all.deb",
],
],
},
Expand Down
9 changes: 4 additions & 5 deletions cdk/lib/report.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,10 @@ export class Report extends GuStack {
timeout: Duration.seconds(10), // The default is 5s
unhealthyThresholdCount: 10, // This also seems unusually high - the default is 2
},
applicationLogging: {
enabled: true,
systemdUnitName: app,
},
});

//TODO check if this customisation is really necessary (it has been copied across from
Expand All @@ -93,11 +97,6 @@ export class Report extends GuStack {
const cfnAsg = autoScalingGroup.node.defaultChild as CfnAutoScalingGroup;
cfnAsg.healthCheckGracePeriod = Duration.seconds(400).toSeconds();

//TODO replace configure-aws-kinesis-agent with devx-logs?
autoScalingGroup.userData.addCommands(
`/opt/aws-kinesis-agent/configure-aws-kinesis-agent ${region} mobile-log-aggregation-${stage} /var/log/${app}/application.log`,
);

adjustCloudformationParameters(this);

new GuCname(this, 'DnsRecordForReport', {
Expand Down
9 changes: 7 additions & 2 deletions notification/conf/logback.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,22 @@
<maxHistory>30</maxHistory>
</rollingPolicy>

<encoder class="com.gu.mobile.logback.MobileLogstashEncoder">
<defaultAppName>notifications</defaultAppName>
<encoder>
<pattern>%date [%thread] %-5level {%mdc} %logger{36}:%L - %msg%n%xException{15}</pattern>

@akash1810 akash1810 Jan 20, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This format was copied from guardian/prism's.

</encoder>
</appender>

<appender name="ASYNCFILE" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
</appender>

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>

<root level="INFO">
<appender-ref ref="ASYNCFILE"/>
Comment thread
jorgeazevedo marked this conversation as resolved.
<appender-ref ref="STDOUT"/>
</root>

</configuration>
18 changes: 15 additions & 3 deletions notification/conf/notification.yaml
View file
Open in desktop

@akash1810 akash1810 Jan 20, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the docs for a summary of these changes.

Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,12 @@ Parameters:
RunbookCopy:
Type: String
Default: <<<Runbook|https://docs.google.com/document/d/1aJMytnPGeWH8YLpD2_66doxqyr8dPvAVonYIOG-zmOA>>>

LoggingStreamName:
Type: AWS::SSM::Parameter::Value<String>
Description: SSM parameter containing the Name (not ARN) on the kinesis stream
AllowedValues:
- /account/services/logging.stream.name.code
- /account/services/logging.stream.name
Resources:
DnsRecord:
Type: AWS::Route53::RecordSet
Expand Down Expand Up @@ -238,7 +243,9 @@ Resources:
Resource:
- !Sub arn:aws:s3:::${DistBucket}/*
- !Sub arn:aws:s3:::${S3TopicCountBucket}/*
- Action: ec2:DescribeTags
- Action:
- ec2:DescribeTags
- ec2:DescribeInstances
Effect: Allow
Resource: '*'
- Action:
Expand Down Expand Up @@ -324,6 +331,12 @@ Resources:
PropagateAtLaunch: true
Value:
!FindInMap [Constants, App, Value]
- Key: SystemdUnit
Value: notification.service
PropagateAtLaunch: true
- Key: LogKinesisStreamName
Value: !Ref LoggingStreamName
PropagateAtLaunch: true
VPCZoneIdentifier: !Ref PrivateSubnets
NotificationInstanceProfile:
Type: AWS::IAM::InstanceProfile
Expand All @@ -350,7 +363,6 @@ Resources:
#!/bin/bash -ev
aws --region ${AWS::Region} s3 cp s3://${DistBucket}/${Stack}/${Stage}/${App}/${App}_1.0-latest_all.deb /tmp
dpkg -i /tmp/${App}_1.0-latest_all.deb
/opt/aws-kinesis-agent/configure-aws-kinesis-agent ${AWS::Region} mobile-log-aggregation-${Stage} /var/log/${App}/application.log
- Stack: !FindInMap [Constants, Stack, Value]
App: !FindInMap [Constants, App, Value]
ScaleDownPolicy:
Expand Down
4 changes: 3 additions & 1 deletion notification/conf/riff-raff.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,16 @@ deployments:
parameters:
amiEncrypted: true
amiTags:
Recipe: mobile-java11-jammy-ARM
Recipe: mobile-java11-jammy-ARM-with-cdk-base

@akash1810 akash1810 Jan 20, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-with-cdk-base appears to be a convention...

AmigoStage: PROD
templatePath: notification.yaml
templateStageParameters:
CODE:
SloSqsQueueArn: /notifications/CODE/mobile-notifications/notifications.queues.sloMonitoringArn
LoggingStreamName: /account/services/logging.stream.name.code

@akash1810 akash1810 Jan 20, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CloudFormation template now has a LoggingStreamName parameter without a default set. This change tells Riff-Raff what value to use when deploying and means the entire change can be deployed with Riff-Raff.

PROD:
SloSqsQueueArn: /notifications/PROD/mobile-notifications/notifications.queues.sloMonitoringArn
LoggingStreamName: /account/services/logging.stream.name
notification:
type: autoscaling
parameters:
Expand Down
9 changes: 7 additions & 2 deletions registration/conf/logback.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,22 @@
<maxHistory>30</maxHistory>
</rollingPolicy>

<encoder class="com.gu.mobile.logback.MobileLogstashEncoder">
<defaultAppName>notifications</defaultAppName>
<encoder>
<pattern>%date [%thread] %-5level {%mdc} %logger{36}:%L - %msg%n%xException{15}</pattern>
</encoder>
</appender>

<appender name="ASYNCFILE" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
</appender>

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>

<root level="INFO">
<appender-ref ref="ASYNCFILE"/>
<appender-ref ref="STDOUT"/>
</root>

</configuration>
17 changes: 15 additions & 2 deletions registration/conf/registration.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ Parameters:
RunbookCopy:
Type: String
Default: <<<Runbook|https://docs.google.com/document/d/1aJMytnPGeWH8YLpD2_66doxqyr8dPvAVonYIOG-zmOA>>>
LoggingStreamName:
Type: AWS::SSM::Parameter::Value<String>
Description: SSM parameter containing the Name (not ARN) on the kinesis stream
AllowedValues:
- /account/services/logging.stream.name.code
- /account/services/logging.stream.name
Resources:
DistributionInstanceProfile:
Type: AWS::IAM::InstanceProfile
Expand All @@ -105,7 +111,9 @@ Resources:
- Action: s3:GetObject
Effect: Allow
Resource: !Sub arn:aws:s3:::${DistBucket}/*
- Action: ec2:DescribeTags
- Action:
- ec2:DescribeTags
- ec2:DescribeInstances
Effect: Allow
Resource: '*'
- Action:
Expand Down Expand Up @@ -267,6 +275,12 @@ Resources:
PropagateAtLaunch: true
Value:
!FindInMap [Constants, App, Value]
- Key: SystemdUnit
Value: registration.service
PropagateAtLaunch: true
- Key: LogKinesisStreamName
Value: !Ref LoggingStreamName
PropagateAtLaunch: true
VPCZoneIdentifier: !Ref PrivateSubnets
RegistrationLaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Expand All @@ -287,7 +301,6 @@ Resources:
#!/bin/bash -ev
aws --region ${AWS::Region} s3 cp s3://${DistBucket}/${Stack}/${Stage}/${App}/${App}_1.0-latest_all.deb /tmp
dpkg -i /tmp/${App}_1.0-latest_all.deb
/opt/aws-kinesis-agent/configure-aws-kinesis-agent ${AWS::Region} mobile-log-aggregation-${Stage} /var/log/${App}/application.log
- Stack: !FindInMap [Constants, Stack, Value]
App: !FindInMap [Constants, App, Value]
ScaleDownPolicy:
Expand Down
7 changes: 6 additions & 1 deletion registration/conf/riff-raff.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,16 @@ deployments:
parameters:
amiEncrypted: true
amiTags:
Recipe: mobile-java11-jammy-ARM
Recipe: mobile-java11-jammy-ARM-with-cdk-base
AmigoStage: PROD
templateStagePaths:
CODE: Registration-CODE.template.json
PROD: Registration-PROD.template.json
templateStageParameters:
CODE:
LoggingStreamName: /account/services/logging.stream.name.code
PROD:
LoggingStreamName: /account/services/logging.stream.name
registration:
type: autoscaling
parameters:
Expand Down
Loading