Skip to content

Commit 607b36d

Browse files
authored
Debulleting the Porch Troubleshooting pages (kptdev#1024)
* Debulleting Troubleshooting Signed-off-by: Dominika Schweier <dominika.schweier@nokia.com> * Adding corrections suggested by copilot/ Signed-off-by: Dominika Schweier <dominika.schweier@nokia.com> Assisted-by: Copilot Code Review:not-known * Correcting review comment Signed-off-by: Dominika Schweier <dominika.schweier@nokia.com> --------- Signed-off-by: Dominika Schweier <dominika.schweier@nokia.com>
1 parent afebe5b commit 607b36d

2 files changed

Lines changed: 31 additions & 81 deletions

File tree

  • docs/content/en/docs/9_troubleshooting_and_faq

docs/content/en/docs/9_troubleshooting_and_faq/disaster-recovery/_index.md

Lines changed: 25 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -26,17 +26,9 @@ in several locations:
2626
Porch's data storage operations are covered in significantly greater depth in the [Architecture and Components section]({{% relref "/docs/5_architecture_and_components/_index.md" %}}).
2727

2828
Each data store serves as the source of truth for different elements of Porch's data structure:
29-
- custom resource objects on the Kubernetes control plane:
30-
- Porch repositories (Repository objects)
31-
- package variants (and by extension package variant sets)
32-
- **if the CR cache is configured:** "work-in-progress" package revisions whose lifecycle stage is "Draft", "Proposed",
33-
or "DeletionProposed"
34-
- Git repositories:
35-
- package revisions (Kpt package file contents and directory structures)
36-
- Package revision cache:
37-
- Kubernetes-related metadata for package revisions (e.g. labels and annotations)
38-
- **if the DB cache is configured:** "work-in-progress" package revisions whose lifecycle stage is "Draft", "Proposed",
39-
or "DeletionProposed"
29+
- **custom resource objects on the Kubernetes control plane**, such as Porch repositories (Repository objects), package variants (and by extension package variant sets) and **if** the CR cache is configured, "work-in-progress" package revisions whose lifecycle stage is "Draft", "Proposed", or "DeletionProposed"
30+
- **Git repositories**, such as package revisions (Kpt package file contents and directory structures)
31+
- **Package revision cache**, such as Kubernetes-related metadata for package revisions (for example, labels and annotations) and **if** the DB cache is configured, "work-in-progress" package revisions whose lifecycle stage is "Draft", "Proposed", or "DeletionProposed"
4032

4133
## Backup strategy
4234

@@ -144,17 +136,13 @@ By default, before running the disaster scenarios, the suite creates, installs,
144136
- a local Kind cluster (the "data cluster") containing:
145137
- a Git server (Gitea)
146138
- containing several copies of public repositories, each containing a large quantity of sample, test, and catalogued
147-
Kpt packages
148-
- to provide a representative workload for Porch when restoring and reconciling package revision data
149-
- a PostgreSQL instance, set up to allow Porch to connect to and use it for the DB cache
150-
- to make it possible to wipe and restore the database independently of Porch
151-
- another local Kind cluster
152-
- with Porch installed
153-
- with DB cache connected to the PostgreSQL instance installed on the data cluster
154-
- with the `porch-server` microservice's memory limits increased to `4GiB`
139+
Kpt packages. This is needed to provide a representative workload for Porch when restoring and reconciling package revision data.
140+
- a PostgreSQL instance, set up to allow Porch to connect to and use it for the DB cache. This is needed to make it possible to wipe and restore the database independently of Porch.
141+
- another local Kind cluster with Porch installed
142+
- with DB cache connected to the PostgreSQL instance installed on the data cluster
143+
- with the `porch-server` microservice's memory limits increased to `4GiB`
155144
- ~115 Repository objects created in Porch, each targeted to a different combination of Git repository and directory within
156-
the repository
157-
- this is calculated to maximize number of package revisions and provide representative workload
145+
the repository. This is calculated to maximize number of package revisions and provide representative workload.
158146
- a small number of new package revisions in various lifecycle states, set up using Porch's API to allow testing that they
159147
also will be properly backed up and restored
160148

@@ -196,12 +184,12 @@ data stores.
196184

197185
Kubernetes cluster is lost with all nodes; Git repositories are lost; DB cache database is lost.
198186

199-
#### Data backed up:
187+
**Data backed up:**
200188
- Porch custom resources
201189
- Git repository contents
202190
- DB cache database contents
203191

204-
#### Data stores lost:
192+
**Data stores lost:**
205193
- Kubernetes control plane: entire Kubernetes cluster deleted
206194
- Git repositories: Git server deleted and recreated empty of data
207195
- DB cache database:
@@ -215,29 +203,23 @@ Kubernetes cluster is lost with all nodes; Git repositories are lost; DB cache d
215203

216204
To ensure data compatibility, backup must be restored into the DB cache of the same version of Porch.
217205

218-
**In step 2, ensure Porch is reinstalled with the same version as before the cluster was lost!**
206+
In step 2, ensure Porch is reinstalled with the same version as before the cluster was lost!
219207
{{% /alert %}}
220208
3. Restore backed-up repository contents to Git server
221209
4. Restore backed-up database contents to PostgreSQL server
222210
5. Perform GitOps reconciliation, gradually (in batches of 20) re-creating all backed-up Porch Repository objects
223211
1. For each batch, wait until all Repository objects have condition with type "Ready" and status set "True"
224212

225-
#### Expected data loss
226-
227-
None - complete recovery of state at time data was backed up.
228-
229-
With backups of all data stores, Porch recovers all data.
213+
**Expected data loss:** None. Complete recovery of state at time data was backed up. With backups of all data stores, Porch recovers all data.
230214

231215

232216
### 2. Kubernetes cluster loss
233217

234218
Kubernetes cluster is lost with all nodes; Git repositories and DB cache database remain safe.
235219

236-
#### Data backed up:
237-
- Porch custom resources
220+
**Data backed up:** Porch custom resources
238221

239-
#### Data stores lost:
240-
- Kubernetes control plane: entire Kubernetes cluster deleted
222+
**Data stores lost:** Kubernetes control plane. The entire Kubernetes cluster is deleted.
241223

242224
#### Restoration steps:
243225
1. Recreate Kubernetes cluster
@@ -251,9 +233,7 @@ Kubernetes cluster is lost with all nodes; Git repositories and DB cache databas
251233
3. Perform GitOps reconciliation, gradually (in batches of 20) re-creating all backed-up Porch Repository objects
252234
1. For each batch, wait until all Repository objects have condition with type "Ready" and status set "True"
253235

254-
#### Expected data loss
255-
256-
None - complete recovery of state at time of cluster loss.
236+
**Expected data loss:** None. Complete recovery of state at time of cluster loss.
257237

258238
Through using Git as the source of truth, we might expect Porch to automatically delete any state that only exists in the
259239
cache - e.g., package revisions in "Draft" lifecycle stage. However, the connection between Porch and Git is represented by the Repository
@@ -266,27 +246,20 @@ cached state.
266246
All Porch pods (by default, all in the "porch-system" namespace) are ungracefully restarted (e.g. by forcible pod deletion
267247
with grace-period 0).
268248

269-
#### Data backed up:
249+
**Data backed up:**
270250
- Porch custom resources
271251
- Git repository contents
272252
- DB cache database contents
273253

274-
#### Data stores lost:
275-
- None
276-
- Porch will immediately begin to re-sync all repositories, resulting in a **decrease in quality of service** until all
277-
repositories are deemed Ready
278-
- **Porch API will be unavailable** to perform operations on package revisions
279-
- `get` or `list` operations can be used to monitor Porch for API availability and repository status
254+
**Data stores lost:** None. Porch will immediately begin to re-sync all repositories, resulting in a **decrease in quality of service** until all repositories are deemed Ready. **Porch API will be unavailable** to perform operations on package revisions. You can use `get` or `list` operations to monitor Porch for API availability and repository status.
280255

281256
#### Restoration steps:
282257
1. Wait until all Porch pods return to Ready state
283258
2. Wait until all Repository objects have condition with type "Ready" and status set "True"
284259
1. GitOps reconciliation is unnecessary in this case since the Repository objects are unchanged
285260
3. List package revisions periodically, monitoring results until state stabilises
286261

287-
#### Expected data loss
288-
289-
None - no data stores were impacted, but only Porch's ability to manage them, allowing for full recovery.
262+
**Expected data loss:** None. No data stores were impacted, but only Porch's ability to manage them, allowing for full recovery.
290263

291264
In a representative testing environment, recovery takes **less than 5 minutes** for 115 Repository objects with a `4GiB`
292265
memory limit applied to the `porch-server` microservice
@@ -315,24 +288,16 @@ Kubernetes cluster and Git repositories remain safe; DB cache database is lost.
315288
Applicable only to Porch with DB cache configured.
316289
{{% /alert %}}
317290

318-
#### Data backed up:
319-
- DB cache database contents
291+
**Data backed up:** DB cache database contents.
320292

321-
#### Data stores lost:
322-
- DB cache database:
323-
- SQL script used to drop all Porch tables
324-
- PostgreSQL server deleted and recreated empty of data
293+
**Data stores lost:** DB cache database (SQL script used to drop all Porch tables, PostgreSQL server deleted and recreated empty of data).
325294

326295
#### Restoration steps:
327296
1. Restore backed-up database contents to PostgreSQL server
328297
2. Perform GitOps reconciliation, gradually (in batches of 20) re-creating all backed-up Porch Repository objects
329298
1. For each batch, wait until all Repository objects have condition with type "Ready" and status set "True"
330299

331-
#### Expected data loss
332-
333-
None - complete recovery of state at time DB cache database was backed up.
334-
335-
With a backup of the cache database, Porch recovers all data.
300+
**Expected data loss:** None. Complete recovery of state at time DB cache database was backed up. With a backup of the cache database, Porch recovers all data.
336301

337302
### 5. DB cache loss without backup
338303

@@ -342,13 +307,9 @@ Kubernetes cluster and Git repositories remain safe; DB cache database is lost w
342307
Applicable only to Porch with DB cache configured.
343308
{{% /alert %}}
344309

345-
#### Data backed up:
346-
- None
310+
**Data backed up:** None.
347311

348-
#### Data stores lost:
349-
- DB cache database:
350-
- SQL script used to drop all Porch tables
351-
- PostgreSQL server deleted and recreated empty of data
312+
**Data stores lost:** DB cache database (SQL script used to drop all Porch tables, PostgreSQL server deleted and recreated empty of data).
352313

353314
#### Restoration steps:
354315
1. Perform GitOps reconciliation, gradually (in batches of 20) re-creating all backed-up Porch Repository objects
@@ -362,7 +323,7 @@ Applicable only to Porch with DB cache configured.
362323
3. Wait until all Repository objects have condition with type "Ready" and status set "True"
363324
4. List package revisions periodically, monitoring results until state stabilises
364325

365-
#### Expected data loss
326+
**Expected data loss:**
366327

367328
All "work in progress" on package revisions lost:
368329
- package revisions in "Draft" or "Proposed" lifecycle stages - complete loss

docs/content/en/docs/9_troubleshooting_and_faq/repository-sync/_index.md

Lines changed: 6 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ description: Common repository sync issues and their solutions
99

1010
### Repository Not Syncing
1111

12-
**Problem**: Repository shows as Ready but packages aren't updating
12+
**Problem**: Repository shows as Ready but packages are not updating
1313

1414
**Solutions**:
1515
```bash
@@ -29,10 +29,7 @@ kubectl logs -n porch-system deployment/porch-controllers | grep "sync.*<repo-na
2929
kubectl logs -n porch-system deployment/porch-controllers | grep "<repo-name>.*error"
3030
```
3131

32-
**Common causes**:
33-
- Invalid cron expression falls back to default frequency
34-
- Repository authentication issues
35-
- Network connectivity problems
32+
**Common causes:** invalid cron expression falls back to default frequency, repository authentication issues or network connectivity problems.
3633

3734
### Authentication Failures
3835

@@ -103,14 +100,11 @@ kubectl logs -n porch-system deployment/porch-server | grep "repositorySync.*<re
103100
git ls-remote <repo-url> # For Git repos
104101
```
105102

106-
**Common causes**:
107-
- Large repository taking time to clone/sync
108-
- Network timeouts
109-
- Repository structure issues
103+
**Common causes**: large repository taking time to clone/sync, network timeouts or repository structure issues.
110104

111105
### One-time Sync Not Triggering
112106

113-
**Problem**: `porchctl repo sync` command succeeds but sync doesn't happen
107+
**Problem**: `porchctl repo sync` command succeeds but sync does not happen
114108

115109
**Diagnostic steps**:
116110
```bash
@@ -124,9 +118,7 @@ date -u # Compare with runOnceAt value
124118
kubectl logs -n porch-system deployment/porch-controllers | grep "runOnceAt"
125119
```
126120

127-
**Solutions**:
128-
- Ensure timestamp is at least 1 minute in future
129-
- Verify namespace is correct
121+
**Solutions**: Ensure the timestamp is at least 1 minute in the future and verify that the namespace is correct.
130122

131123
## Error Messages & Diagnostic Steps
132124

@@ -213,10 +205,7 @@ See [Repository Controller Configuration]({{% relref "/docs/6_configuration_and_
213205
**A**: Yes, periodic scheduling and one-time sync work independently. One-time synchronization executes regardless of the periodic schedule.
214206

215207
### Q: Why is my cron expression not working?
216-
**A**: Porch uses standard 5-field cron format. Common mistakes:
217-
- Using 6 fields (seconds not supported)
218-
- Missing fields
219-
- Invalid ranges or values
208+
**A**: Porch uses standard 5-field cron format. Common mistakes include using 6 fields (seconds not supported), missing fields, or invalid ranges or values.
220209

221210
### Q: How do I stop repository syncing?
222211
**A**: Repository synchronization cannot be completely stopped. Porch continuously monitors repositories for changes. You can only modify the sync frequency by updating the sync schedule configuration or remove custom schedules to use the default frequency.

0 commit comments

Comments
 (0)