← 返回首页
Driver Upgrade Guide — AMD GPU Operator 1.5.0
Back to top Ctrl+K
  • Driver Upgrade Guide

Driver Upgrade Guide

Driver Upgrade Guide#

This guide walks through the process of upgrading AMD GPU drivers on worker nodes.

Overview#

The driver can be upgraded in the following methods.

  1. Automatic Upgrade Process

  2. Manual Upgrade Process

1. Automatic Upgrade Process#

Automatic upgrade is enabled if the deviceconfig has field upgradePolicy enabled. If this field is not configured, then user has to follow the manual steps as shown in next section If this field is configured and the version field is changed in driver spec, automatic driver upgrade progress is initiated.

The following operations are sequentially executed by the gpu operator for each selected node

  1. The node is cordoned so that no pods can be scheduled on this node

  2. The existing pods (that require amd gpus) are drained/deleted based on the config in the upgrade policy.

  3. The desired driver version label is updated as shown below.

    kmm.node.kubernetes.io/version-module.<namespace>.<config-name>=<new-version>

  4. KMM operator unloads the old driver version and loads the new driver version.

  5. If the node requires reboot post installation (configurable in upgradePolicy), the node is rebooted

  6. Once the node is rebooted and the desired driver is loaded, the node is uncordoned and available for scheduling.

The following are the steps to perform the automatic driver upgrade

  1. Set the desired driver version and configure upgrade policy

  2. Track the upgrade status through CR status

1. Set desired driver version and configure upgrade policy#

The following sample config shows the relevant fields to start automatic driver upgrade across the nodes in the cluster with default upgrade configuration.

apiVersion: amd.com/v1alpha1 kind: DeviceConfig metadata: name: test-deviceconfig # use the namespace where AMD GPU Operator is running namespace: kube-amd-gpu spec: driver: # NOTE: Starting from ROCm 7.1 the amdgpu version is using new versioning schema # please refer to https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/user-kernel-space-compat-matrix.html version: 6.3.2 enable: true upgradePolicy: enable: true selector: feature.node.kubernetes.io/amd-gpu: "true"

Upgrade configuration reference

To check the full spec of upgrade configuration run kubectl get crds deviceconfigs.amd.com -oyaml

driver.upgradePolicy Parameters#

Parameter

Description

Default

enable

Enable this upgrade policy

false

maxParallelUpgrades

Maximum number of nodes which will be upgraded in parallel

1

maxUnavailableNodes

Maximum number (or Percentage) of nodes which can be unavailable (cordoned) in the cluster

25%

rebootRequired

Reboot the node after driver upgrade is done. Waits for 60 mins post reboot before declaring as failed

true

Warning: When using ROCm drivers version 6.3 and below, a known issue may prevent the driver upgrade from fully completing unless the node is rebooted. As a workaround, we strongly recommend setting the rebootRequired field to true in your upgrade policy. This ensures that a reboot is triggered after the driver upgrade, allowing the new driver to be fully loaded. This workaround should be applied until a permanent fix is provided in a future release.

driver.upgradePolicy.nodeDrainPolicy Parameters#

Parameter

Description

Default

force

Allow drain to proceed on the node even if there are managed pods such as daemon-sets. In such cases drain will not proceed unless this option is set to true

true

timeout

The length of time to wait before giving up. Zero means infinite

300s

driver.upgradePolicy.podDeletionPolicy Parameters#

Parameter

Description

Default

force

Force delete all pods that use amd gpus

true

timeout

The length of time to wait before giving up. Zero means infinite

300s

2. Track the upgrade status through CR status#

The status.nodeModuleStatus.<worker-node>.status captures the status of the upgrade process for each node

status: nodeModuleStatus: worker-10-11-71-66: containerImage: test.registry.io:5000/drivers:ubuntu-22.04-5.15.0-124-generic-6.3.2 kernelVersion: 5.15.0-124-generic lastTransitionTime: 2024-12-05 05:35:04 +0000 UTC status: Upgrade-Complete worker-10-11-71-67: containerImage: test.registry.io:5000/drivers:ubuntu-22.04-5.15.0-124-generic-6.3.2 kernelVersion: 5.15.0-124-generic lastTransitionTime: 2024-12-05 05:35:04 +0000 UTC status: Upgrade-Complete worker-10-11-71-69: containerImage: test.registry.io:5000/drivers:ubuntu-22.04-5.15.0-124-generic-6.3.2 kernelVersion: 5.15.0-124-generic lastTransitionTime: 2024-12-05 05:34:53 +0000 UTC status: Upgrade-Complete worker-10-11-77-194: containerImage: test.registry.io:5000/drivers:ubuntu-22.04-5.15.0-119-generic-6.3.2 kernelVersion: 5.15.0-119-generic lastTransitionTime: 2024-12-05 05:37:14 +0000 UTC status: Upgrade-Complete

The following are the different node states during the upgrade process

State

Description

Install-In-Progress

Driver is being installed on the node for the first time

Install-Complete

Driver install is complete

Upgrade-Not-Started

Automatic upgrade enabled and driver version change is detected. All nodes move to this state

Upgrade-In-Progress

Selected nodes conforming to upgrade policy will be attempted for driver upgrade

Upgrade-Complete

Driver upgrade is successfully complete on the node

Upgrade-Timed-Out

Driver upgrade couldn’t finish within 2 hours

Cordon-Failed

Cordoning of the node failed

Uncordon-Failed

Uncordoning of the node failed

Drain-Failed

Drain node or Delete pods operation failed

Reboot-In-Progress

Driver upgrade is done and reboot is in progress

Reboot-Failed

Driver upgrade is done and reboot attempt failed

Upgrade-Failed

Driver upgrade failed for any other reasons

The following are considered during the automatic upgrade process

  1. Selection of a node should satisfy both maxUnavailableNodes and maxParallelUpgrades criteria

  2. All nodes in failed state is considered while calculating maxUnavailableNodes

3. Recovery From Upgrade Failure#

If it is observed that the upgrade status is in failed state for a specific node, the user can debug the node, fix it and then add this label to the node to restart upgrade on it. The upgrade state will be reset and it can be tracked as it was before

  • Command: kubectl label node <nodename> operator.amd.com/gpu-driver-upgrade-state=upgrade-required

  • Label: operator.amd.com/gpu-driver-upgrade-state: upgrade-required

2. Manual Upgrade Process#

The manual upgrade process involves the following steps:

  1. Verifying current installation

  2. Updating the driver version

  3. Managing workloads

  4. Updating node labels

  5. Performing the upgrade

1. Check Current Driver Version#

Verify the existing driver version label on your worker nodes:

kubectl get node <worker-node> -o yaml

Look for the label in this format:

kmm.node.kubernetes.io/version-module.<deviceconfig-namespace>.<deviceconfig-name>=<version>

Example:

kmm.node.kubernetes.io/version-module.kube-amd-gpu.test-device-config=6.1.3

2. Update DeviceConfig#

Update the driversVersion field in your DeviceConfig:

kubectl edit deviceconfigs <config-name> -n kube-amd-gpu

The operator will automatically:

  1. Look for the new driver image in the registry

  2. Build the image if it doesn’t exist

  3. Push the built image to your specified registry

Image Tag Format#

The operator uses specific tag formats based on the OS:

OS

Tag Format

Example

Ubuntu

ubuntu-<version>-<kernel>-<driver>

ubuntu-22.04-6.8.0-40-generic-6.1.3

RHEL CoreOS

coreos-<version>-<kernel>-<driver>

coreos-416.94-5.14.0-427.28.1.el9_4.x86_64-6.2.2

Warning: If a node’s ready status changes during upgrade (Ready → NotReady → Ready) before its driver version label is updated, the old driver won’t be reinstalled. Complete the upgrade steps for these nodes to install the new driver.

3. Stop Workloads#

Stop all workloads using the AMD GPU driver on the target node before proceeding.

4. Update Node Labels#

You have two options for updating node labels:

Option A: Direct Update (Recommended)#

If no additional maintenance is needed, directly update the version label:

# Old label format: kmm.node.kubernetes.io/version-module.<namespace>.<config-name>=<old-version> # New label format: kmm.node.kubernetes.io/version-module.<namespace>.<config-name>=<new-version>

Option B: Remove and Add (If maintenance is needed)#

  • Remove old version label:

kubectl label node <worker-node> \ kmm.node.kubernetes.io/version-module.<namespace>.<config-name>-

  • Perform required maintenance

  • Add new version label:

kubectl label node <worker-node> \ kmm.node.kubernetes.io/version-module.<namespace>.<config-name>=<new-version>

5. Restart Workloads#

After the new driver is installed successfully, restart your GPU workloads on the upgraded node.

Verification#

To verify the upgrade, check node labels:

kubectl get node <worker-node> --show-labels | grep kmm.node.kubernetes.io

  • Verify driver version:

kubectl get deviceconfigs <config-name> -n kube-amd-gpu -o yaml

  • Check driver status:

kubectl get deviceconfigs <config-name> -n kube-amd-gpu -o jsonpath='{.status}'

previous

Secure Boot Configuration Guide

next

Kernel Upgrade Guide

© 2025 Advanced Micro Devices, Inc