Skip to content

Commit 71e62c0

Browse files
w13915984028w13915984028
authored
Add troubleshooting about rancher-logging and upgradelogging (harvester#927)
* Add troubleshooting about rancher-logging and upgradelogging Signed-off-by: Jian Wang <jian.wang@suse.com>
1 parent 8511089 commit 71e62c0

9 files changed

Lines changed: 342 additions & 3 deletions

File tree

‎docs/troubleshooting/logging.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
sidebar_position: 5
3+
sidebar_label: Logging
4+
title: "Logging"
5+
---
6+
7+
<head>
8+
<link rel="canonical" href="https://docs.harvesterhci.io/v1.6/troubleshooting/logging"/>
9+
</head>
10+
11+
The following sections contain tips to troubleshoot Harvester Logging.
12+
13+
## Failed to enable `rancher-logging` Add-On
14+
15+
### Issue Description
16+
17+
An error message appears on the Harvester UI when you attempt to enable the [`rancher-logging`](../advanced/addons.md) add-on.
18+
19+
![](/img/v1.6/troubleshooting/failed-to-enable-rancher-logging-addon.png)
20+
21+
Log messages from the `cattle-logging-system/helm-install-rancher-logging` pod confirm that an error has occurred.
22+
23+
Example:
24+
25+
```sh
26+
...
27+
echo 'Installing helm chart'
28+
...
29+
+ helm install --version ... rancher-logging rancher-logging/rancher-logging ...
30+
...
31+
Error: INSTALLATION FAILED: Unable to continue with install: ClusterRole "logging-admin" in namespace "" exists
32+
and cannot be imported into the current release: invalid ownership metadata;
33+
annotation validation error: key "meta.helm.sh/release-name" must equal "rancher-logging":
34+
current value is "hvst-upgrade-md54b-upgradelog-operator"
35+
```
36+
37+
![](/img/v1.6/troubleshooting/logging-installation-failure-due-to-conflict.png)
38+
39+
### Root Cause
40+
41+
**Summary**: When you enable the `rancher-logging` add-on, Harvester attempts to install the [Logging Operator](https://kube-logging.dev/docs/#overview). The installation fails when the operator already exists on the cluster, which may be a stale resource from previous upgrade attempts or created by the current ongoing upgrade.
42+
43+
The **Upgrade Software** screen includes an **Enable Logging** option that you must select to enable Harvester to log upgrade events.
44+
45+
![](/img/v1.6/troubleshooting/upgrade-with-enable-logging.png)
46+
47+
However, the background processing depends on whether the `rancher-logging` add-on is enabled.
48+
49+
When the add-on is disabled, the upgrade controller creates the following `logging` and `managedchart` objects.
50+
51+
```sh
52+
$ kubectl get upgrade.harvesterhci -A
53+
NAMESPACE NAME AGE
54+
harvester-system hvst-upgrade-hpfnw 36s
55+
56+
$ kubectl get logging -A
57+
NAME LOGGINGREF CONTROLNAMESPACE
58+
hvst-upgrade-hpfnw-upgradelog-infra harvester-upgradelog harvester-system // newly created by upgrade controller
59+
hvst-upgrade-hpfnw-upgradelog-operator-root cattle-logging-system // newly created by upgrade controller, acts as logging-operator
60+
61+
$ kubectl get managedchart -A
62+
NAMESPACE NAME AGE
63+
fleet-local hvst-upgrade-hpfnw-upgradelog-operator 18s // newly created by upgrade controller
64+
```
65+
66+
When the add-on is enabled, the upgrade controller creates the following `logging` object.
67+
68+
```sh
69+
$ kubectl get upgrade.harvesterhci -A
70+
NAMESPACE NAME AGE
71+
harvester-system hvst-upgrade-9sn4x 14s
72+
73+
$ kubectl get managedchart -A
74+
NAMESPACE NAME AGE
75+
76+
$ kubectl get logging -A
77+
NAME LOGGINGREF CONTROLNAMESPACE
78+
hvst-upgrade-9sn4x-upgradelog-infra harvester-upgradelog harvester-system // newly created by upgrade controller
79+
rancher-logging-kube-audit harvester-kube-audit-log-ref cattle-logging-system // originally created by rancher-logging addon
80+
rancher-logging-root cattle-logging-system // originally created by rancher-logging addon, acts as logging-operator
81+
```
82+
83+
You may encounter the issue in the following situations:
84+
85+
- The `rancher-logging` add-on is initially disabled. You start the upgrade with the **Enable Logging** option selected. Without waiting for the upgrade to be completed, you enable the `rancher-logging` add-on. The Harvester UI displays an error message.
86+
87+
- The `rancher-logging` add-on is initially disabled. You start the upgrade with the **Enable Logging** option selected and wait until the upgrade is completed. The `managedchart` and `logging` objects are not removed because of a [known issue](https://github.com/harvester/harvester/issues/7654). Next, you enable the `rancher-logging` add-on. The Harvester UI displays an error message.
88+
89+
### Workaround
90+
91+
1. If an upgrade is in progress, wait until it is successfully completed or removed.
92+
93+
:::info important
94+
95+
To avoid logging conflicts, do not enable or disable the `rancher-logging` add-on while an upgrade is in progress. This action is blocked in Harvester v1.7.0 and later versions.
96+
97+
:::
98+
99+
1. If the `rancher-logging` add-on is enabled but in a failed state, disable it.
100+
101+
1. Check the `logging` and `managedchart` objects. If the names of these objects start with `hvst-upgrade-`, manually delete them.
102+
103+
1. Enable the `rancher-logging` add-on.
104+
105+
:::note
106+
107+
All add-ons must be in a healthy state prior to starting an upgrade. This prerequisite is automatically verified in Harvester v1.7.0 and later versions.
108+
109+
:::
110+
111+
### Related Issue
112+
113+
[#9289](https://github.com/harvester/harvester/issues/9289) and [#9644](https://github.com/harvester/harvester/issues/9644)

‎docs/troubleshooting/vm.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
sidebar_position: 5
2+
sidebar_position: 6
33
sidebar_label: VM
44
title: "VM"
55
---

‎static/img/v1.6/troubleshooting/failed-to-enable-rancher-logging-addon.png‎

249 KB
Loading

‎static/img/v1.6/troubleshooting/logging-installation-failure-due-to-conflict.png‎

352 KB
Loading

‎static/img/v1.6/troubleshooting/upgrade-with-enable-logging.png‎

149 KB
Loading

‎versioned_docs/version-v1.5/troubleshooting/logging.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
sidebar_position: 5
3+
sidebar_label: Logging
4+
title: "Logging"
5+
---
6+
7+
<head>
8+
<link rel="canonical" href="https://docs.harvesterhci.io/v1.6/troubleshooting/logging"/>
9+
</head>
10+
11+
The following sections contain tips to troubleshoot Harvester Logging.
12+
13+
## Failed to enable `rancher-logging` Add-On
14+
15+
### Issue Description
16+
17+
An error message appears on the Harvester UI when you attempt to enable the [`rancher-logging`](../advanced/addons.md) add-on.
18+
19+
![](/img/v1.6/troubleshooting/failed-to-enable-rancher-logging-addon.png)
20+
21+
Log messages from the `cattle-logging-system/helm-install-rancher-logging` pod confirm that an error has occurred.
22+
23+
Example:
24+
25+
```sh
26+
...
27+
echo 'Installing helm chart'
28+
...
29+
+ helm install --version ... rancher-logging rancher-logging/rancher-logging ...
30+
...
31+
Error: INSTALLATION FAILED: Unable to continue with install: ClusterRole "logging-admin" in namespace "" exists
32+
and cannot be imported into the current release: invalid ownership metadata;
33+
annotation validation error: key "meta.helm.sh/release-name" must equal "rancher-logging":
34+
current value is "hvst-upgrade-md54b-upgradelog-operator"
35+
```
36+
37+
![](/img/v1.6/troubleshooting/logging-installation-failure-due-to-conflict.png)
38+
39+
### Root Cause
40+
41+
**Summary**: When you enable the `rancher-logging` add-on, Harvester attempts to install the [Logging Operator](https://kube-logging.dev/docs/#overview). The installation fails when the operator already exists on the cluster, which may be a stale resource from previous upgrade attempts or created by the current ongoing upgrade.
42+
43+
The **Upgrade Software** screen includes an **Enable Logging** option that you must select to enable Harvester to log upgrade events.
44+
45+
![](/img/v1.6/troubleshooting/upgrade-with-enable-logging.png)
46+
47+
However, the background processing depends on whether the `rancher-logging` add-on is enabled.
48+
49+
When the add-on is disabled, the upgrade controller creates the following `logging` and `managedchart` objects.
50+
51+
```sh
52+
$ kubectl get upgrade.harvesterhci -A
53+
NAMESPACE NAME AGE
54+
harvester-system hvst-upgrade-hpfnw 36s
55+
56+
$ kubectl get logging -A
57+
NAME LOGGINGREF CONTROLNAMESPACE
58+
hvst-upgrade-hpfnw-upgradelog-infra harvester-upgradelog harvester-system // newly created by upgrade controller
59+
hvst-upgrade-hpfnw-upgradelog-operator-root cattle-logging-system // newly created by upgrade controller, acts as logging-operator
60+
61+
$ kubectl get managedchart -A
62+
NAMESPACE NAME AGE
63+
fleet-local hvst-upgrade-hpfnw-upgradelog-operator 18s // newly created by upgrade controller
64+
```
65+
66+
When the add-on is enabled, the upgrade controller creates the following `logging` object.
67+
68+
```sh
69+
$ kubectl get upgrade.harvesterhci -A
70+
NAMESPACE NAME AGE
71+
harvester-system hvst-upgrade-9sn4x 14s
72+
73+
$ kubectl get managedchart -A
74+
NAMESPACE NAME AGE
75+
76+
$ kubectl get logging -A
77+
NAME LOGGINGREF CONTROLNAMESPACE
78+
hvst-upgrade-9sn4x-upgradelog-infra harvester-upgradelog harvester-system // newly created by upgrade controller
79+
rancher-logging-kube-audit harvester-kube-audit-log-ref cattle-logging-system // originally created by rancher-logging addon
80+
rancher-logging-root cattle-logging-system // originally created by rancher-logging addon, acts as logging-operator
81+
```
82+
83+
You may encounter the issue in the following situations:
84+
85+
- The `rancher-logging` add-on is initially disabled. You start the upgrade with the **Enable Logging** option selected. Without waiting for the upgrade to be completed, you enable the `rancher-logging` add-on. The Harvester UI displays an error message.
86+
87+
- The `rancher-logging` add-on is initially disabled. You start the upgrade with the **Enable Logging** option selected and wait until the upgrade is completed. The `managedchart` and `logging` objects are not removed because of a [known issue](https://github.com/harvester/harvester/issues/7654). Next, you enable the `rancher-logging` add-on. The Harvester UI displays an error message.
88+
89+
### Workaround
90+
91+
1. If an upgrade is in progress, wait until it is successfully completed or removed.
92+
93+
:::info important
94+
95+
To avoid logging conflicts, do not enable or disable the `rancher-logging` add-on while an upgrade is in progress. This action is blocked in Harvester v1.7.0 and later versions.
96+
97+
:::
98+
99+
1. If the `rancher-logging` add-on is enabled but in a failed state, disable it.
100+
101+
1. Check the `logging` and `managedchart` objects. If the names of these objects start with `hvst-upgrade-`, manually delete them.
102+
103+
1. Enable the `rancher-logging` add-on.
104+
105+
:::note
106+
107+
All add-ons must be in a healthy state prior to starting an upgrade. This prerequisite is automatically verified in Harvester v1.7.0 and later versions.
108+
109+
:::
110+
111+
### Related Issue
112+
113+
[#9289](https://github.com/harvester/harvester/issues/9289) and [#9644](https://github.com/harvester/harvester/issues/9644)

‎versioned_docs/version-v1.5/troubleshooting/vm.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
sidebar_position: 5
2+
sidebar_position: 6
33
sidebar_label: VM
44
title: "VM"
55
---

‎versioned_docs/version-v1.6/troubleshooting/logging.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
sidebar_position: 5
3+
sidebar_label: Logging
4+
title: "Logging"
5+
---
6+
7+
<head>
8+
<link rel="canonical" href="https://docs.harvesterhci.io/v1.6/troubleshooting/logging"/>
9+
</head>
10+
11+
The following sections contain tips to troubleshoot Harvester Logging.
12+
13+
## Failed to enable `rancher-logging` Add-On
14+
15+
### Issue Description
16+
17+
An error message appears on the Harvester UI when you attempt to enable the [`rancher-logging`](../advanced/addons.md) add-on.
18+
19+
![](/img/v1.6/troubleshooting/failed-to-enable-rancher-logging-addon.png)
20+
21+
Log messages from the `cattle-logging-system/helm-install-rancher-logging` pod confirm that an error has occurred.
22+
23+
Example:
24+
25+
```sh
26+
...
27+
echo 'Installing helm chart'
28+
...
29+
+ helm install --version ... rancher-logging rancher-logging/rancher-logging ...
30+
...
31+
Error: INSTALLATION FAILED: Unable to continue with install: ClusterRole "logging-admin" in namespace "" exists
32+
and cannot be imported into the current release: invalid ownership metadata;
33+
annotation validation error: key "meta.helm.sh/release-name" must equal "rancher-logging":
34+
current value is "hvst-upgrade-md54b-upgradelog-operator"
35+
```
36+
37+
![](/img/v1.6/troubleshooting/logging-installation-failure-due-to-conflict.png)
38+
39+
### Root Cause
40+
41+
**Summary**: When you enable the `rancher-logging` add-on, Harvester attempts to install the [Logging Operator](https://kube-logging.dev/docs/#overview). The installation fails when the operator already exists on the cluster, which may be a stale resource from previous upgrade attempts or created by the current ongoing upgrade.
42+
43+
The **Upgrade Software** screen includes an **Enable Logging** option that you must select to enable Harvester to log upgrade events.
44+
45+
![](/img/v1.6/troubleshooting/upgrade-with-enable-logging.png)
46+
47+
However, the background processing depends on whether the `rancher-logging` add-on is enabled.
48+
49+
When the add-on is disabled, the upgrade controller creates the following `logging` and `managedchart` objects.
50+
51+
```sh
52+
$ kubectl get upgrade.harvesterhci -A
53+
NAMESPACE NAME AGE
54+
harvester-system hvst-upgrade-hpfnw 36s
55+
56+
$ kubectl get logging -A
57+
NAME LOGGINGREF CONTROLNAMESPACE
58+
hvst-upgrade-hpfnw-upgradelog-infra harvester-upgradelog harvester-system // newly created by upgrade controller
59+
hvst-upgrade-hpfnw-upgradelog-operator-root cattle-logging-system // newly created by upgrade controller, acts as logging-operator
60+
61+
$ kubectl get managedchart -A
62+
NAMESPACE NAME AGE
63+
fleet-local hvst-upgrade-hpfnw-upgradelog-operator 18s // newly created by upgrade controller
64+
```
65+
66+
When the add-on is enabled, the upgrade controller creates the following `logging` object.
67+
68+
```sh
69+
$ kubectl get upgrade.harvesterhci -A
70+
NAMESPACE NAME AGE
71+
harvester-system hvst-upgrade-9sn4x 14s
72+
73+
$ kubectl get managedchart -A
74+
NAMESPACE NAME AGE
75+
76+
$ kubectl get logging -A
77+
NAME LOGGINGREF CONTROLNAMESPACE
78+
hvst-upgrade-9sn4x-upgradelog-infra harvester-upgradelog harvester-system // newly created by upgrade controller
79+
rancher-logging-kube-audit harvester-kube-audit-log-ref cattle-logging-system // originally created by rancher-logging addon
80+
rancher-logging-root cattle-logging-system // originally created by rancher-logging addon, acts as logging-operator
81+
```
82+
83+
You may encounter the issue in the following situations:
84+
85+
- The `rancher-logging` add-on is initially disabled. You start the upgrade with the **Enable Logging** option selected. Without waiting for the upgrade to be completed, you enable the `rancher-logging` add-on. The Harvester UI displays an error message.
86+
87+
- The `rancher-logging` add-on is initially disabled. You start the upgrade with the **Enable Logging** option selected and wait until the upgrade is completed. The `managedchart` and `logging` objects are not removed because of a [known issue](https://github.com/harvester/harvester/issues/7654). Next, you enable the `rancher-logging` add-on. The Harvester UI displays an error message.
88+
89+
### Workaround
90+
91+
1. If an upgrade is in progress, wait until it is successfully completed or removed.
92+
93+
:::info important
94+
95+
To avoid logging conflicts, do not enable or disable the `rancher-logging` add-on while an upgrade is in progress. This action is blocked in Harvester v1.7.0 and later versions.
96+
97+
:::
98+
99+
1. If the `rancher-logging` add-on is enabled but in a failed state, disable it.
100+
101+
1. Check the `logging` and `managedchart` objects. If the names of these objects start with `hvst-upgrade-`, manually delete them.
102+
103+
1. Enable the `rancher-logging` add-on.
104+
105+
:::note
106+
107+
All add-ons must be in a healthy state prior to starting an upgrade. This prerequisite is automatically verified in Harvester v1.7.0 and later versions.
108+
109+
:::
110+
111+
### Related Issue
112+
113+
[#9289](https://github.com/harvester/harvester/issues/9289) and [#9644](https://github.com/harvester/harvester/issues/9644)

‎versioned_docs/version-v1.6/troubleshooting/vm.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
sidebar_position: 5
2+
sidebar_position: 6
33
sidebar_label: VM
44
title: "VM"
55
---

0 commit comments

Comments
 (0)