Neuron 2.29.0.0 bump

[qianxjcraig]

Description of changes:
Bumps the Neuron driver (aws-neuronx-dkms) from 2.26.10.0 to 2.29.0.0 for kernel-6.1, kernel-6.12, and kernel-6.18.

Testing done:

Built AMIs for all three kernels from a locally-published kernel-kit and validated the neuron driver on real hardware. inf2 (Inferentia2) is covered by the automated aws-k8s-tester single-node case; inf1 was validated manually with the BERT inference container.

Kernel	Variant	inf2 e2e	inf1 BERT
6.1	aws-k8s-1.32	PASS	—
6.12	aws-k8s-1.34	PASS	PASS
6.18	aws-k8s-1.36	PASS	—

modinfo neuron on inf2.xlarge / kernel-6.18 — driver 2.29.0.0, signed, under lockdown

bash-5.2# modinfo neuron | head -12
filename:       /lib/modules/6.18.38/kernel/drivers/neuron/neuron.ko
alias:          pci:v00001d0fd00007064sv*sd*bc*sc*i*
version:        2.29.0.0
license:        GPL
description:    Neuron Driver, built from SHA: c64fcb38034831fa1d30e95aca96ff5ed5005726
import_ns:      DMA_BUF
srcversion:     99E7EA212A04EC1B5E7F8BB
depends:
name:           neuron
retpoline:      Y
vermagic:       6.18.38 SMP preempt mod_unload modversions
sig_id:         PKCS#7

bash-5.2# apiclient get settings.kernel
{
  "settings": {
    "kernel": {
      "lockdown": "integrity"
    }
  }
}

modinfo neuron on inf2.xlarge / kernel-6.1 — driver 2.29.0.0, signed, under lockdown

bash-5.2# modinfo neuron | head -12
filename:       /lib/modules/6.1.176/kernel/drivers/neuron/neuron.ko.gz
alias:          pci:v00001d0fd00007064sv*sd*bc*sc*i*
version:        2.29.0.0
license:        GPL
description:    Neuron Driver, built from SHA: c64fcb38034831fa1d30e95aca96ff5ed5005726
import_ns:      DMA_BUF
srcversion:     4731D31EB0EAD9EB76F0B54
depends:
retpoline:      Y
name:           neuron
vermagic:       6.1.176 SMP preempt mod_unload modversions
sig_id:         PKCS#7

bash-5.2# apiclient get settings.kernel
{
  "settings": {
    "kernel": {
      "lockdown": "integrity"
    }
  }
}

modinfo neuron on inf2.xlarge / kernel-6.12 — driver 2.29.0.0, signed, under lockdown

bash-5.2# modinfo neuron | head -12
filename:       /lib/modules/6.12.94/kernel/drivers/neuron/neuron.ko
alias:          pci:v00001d0fd00007064sv*sd*bc*sc*i*
version:        2.29.0.0
license:        GPL
description:    Neuron Driver, built from SHA: c64fcb38034831fa1d30e95aca96ff5ed5005726
import_ns:      DMA_BUF
srcversion:     4731D31EB0EAD9EB76F0B54
depends:
name:           neuron
retpoline:      Y
vermagic:       6.12.94 SMP preempt mod_unload modversions
sig_id:         PKCS#7

bash-5.2# apiclient get settings.kernel
{
  "settings": {
    "kernel": {
      "lockdown": "integrity"
    }
  }
}

Neuron device plugin reports the expected resources on inf2.xlarge

$ kubectl get nodes -o jsonpath='{.items[0].status.allocatable.aws\.amazon\.com/neuroncore}'
2
$ kubectl get nodes -o jsonpath='{.items[0].status.allocatable.aws\.amazon\.com/neuron}'
1

inf2 single-node e2e — kernel-6.18 / aws-k8s-1.36 (EKS 1.36)

Ran the aws-k8s-tester neuron single-node case against an EKS 1.36 cluster with a Bottlerocket inf2.xlarge nodegroup:

go test -v -tags=e2e ./test/cases/neuron/... \
  -timeout=0 -test.v -args \
  --feature="single-node" -efaEnabled=false \
  --neuronTestImage=$IMAGE \
  --nodeType=inf2.xlarge

The test harness saw the hardware, and the MNIST model compiled and trained on both
NeuronCores:

[INFO] Total Nodes: 1
[INFO] Total Neuron Count: 1, Neuron Per Node: 1
[INFO] Total Neuron Core Count: 2, Neuron Core Per Node: 2

Compiler status PASS
[INFO]: Compilation Successfully Completed for model.MODULE_1389654516982237585+e30acd3a.hlo_module.pb
Compiler status PASS
[INFO]: Compilation Successfully Completed for model.MODULE_5247849761025925942+e30acd3a.hlo_module.pb

Train throughput (iter/sec): 155.50979826816217
Train throughput (iter/sec): 155.44042443120804
Final loss is 0.3252
Final loss is 0.0950
----------End Training ---------------

The Job completed and was cleaned up by the harness.

inf2 single-node e2e — kernel-6.1 / aws-k8s-1.32 (EKS 1.32)

Same aws-k8s-tester single-node case, EKS 1.32 cluster, Bottlerocket inf2.xlarge nodegroup:

Compiler status PASS
Using a cached neff at /var/tmp/neuron-compile-cache/neuronxcc-2.16.372.0+4a9b2326/MODULE_11349347541938798069+e30acd3a/model.neff
Train throughput (iter/sec): 172.69821496592647
Train throughput (iter/sec): 172.71514554904022
Final loss is 0.0942
Final loss is 0.3240
----------End Training ---------------

--- PASS: TestNeuronNodes (350.07s)
    --- PASS: TestNeuronNodes/single-node (350.07s)
        --- PASS: TestNeuronNodes/single-node/Single_node_test_Job_succeeds (350.01s)
    --- SKIP: TestNeuronNodes/multi-node (0.00s)
PASS
ok      github.com/aws/aws-k8s-tester/test/cases/neuron    421.423s

inf2 single-node e2e — kernel-6.12 / aws-k8s-1.34 (EKS 1.34)

Same aws-k8s-tester single-node case, EKS 1.34 cluster, Bottlerocket inf2.xlarge nodegroup:

Compiler status PASS
Using a cached neff at /var/tmp/neuron-compile-cache/neuronxcc-2.16.372.0+4a9b2326/MODULE_11349347541938798069+e30acd3a/model.neff
Train throughput (iter/sec): 167.43307489486853
Train throughput (iter/sec): 167.4326745834489
Final loss is 0.3240
Final loss is 0.0942
----------End Training ---------------

--- PASS: TestNeuronNodes (340.06s)
    --- PASS: TestNeuronNodes/single-node (340.06s)
        --- PASS: TestNeuronNodes/single-node/Single_node_test_Job_succeeds (340.01s)
    --- SKIP: TestNeuronNodes/multi-node (0.00s)
PASS
ok      github.com/aws/aws-k8s-tester/test/cases/neuron    411.591s

inf1 BERT inference — kernel-6.12 / aws-k8s-1.34 (inf1.xlarge)

inf1 has no automated coverage, so it was validated manually. Ran the HuggingFace pretrained BERT inference container (torch-neuron) as a pod on a Bottlerocket inf1.xlarge node, with direct device access (/dev/neuron0 + neuron RTD socket):

Compiler status PASS
INFO:Neuron:Neuron successfully compiled 1 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 100.0%
BERT says that "The company HuggingFace is based in New York City" and
  "HuggingFace's headquarters are situated in Manhattan" are paraphrase
BERT says that "The company HuggingFace is based in New York City" and
  "Apples are especially bad for your health" are not paraphrase

Maximum throughput = 215 sentences/sec
Average throughput = 212 sentences/sec

Latency Percentiles:
P50 = 9 ms   P90 = 9 ms   P95 = 9 ms   P99 = 9 ms   P100 = 13 ms

Accuracy = 84.56%

Sanity test: Processed - num batches 12209, batch size 1, num cores 2

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[vigh-m]

Suggested change

	Source401: https://yum.repos.neuron.amazonaws.com/aws-neuronx-dkms-2.29.0.0.noarch.rpm
	Source401: https://yum.repos.neuron.amazonaws.com/aws-neuronx-dkms-%{neuron_ver}.0.noarch.rpm

This must have been missed at some point. We can fix it

[qianxjcraig]

Fixed