You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/en/docs/9_troubleshooting_and_faq/disaster-recovery/_index.md
+25-64Lines changed: 25 additions & 64 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,17 +26,9 @@ in several locations:
26
26
Porch's data storage operations are covered in significantly greater depth in the [Architecture and Components section]({{% relref "/docs/5_architecture_and_components/_index.md" %}}).
27
27
28
28
Each data store serves as the source of truth for different elements of Porch's data structure:
29
-
- custom resource objects on the Kubernetes control plane:
30
-
- Porch repositories (Repository objects)
31
-
- package variants (and by extension package variant sets)
32
-
-**if the CR cache is configured:** "work-in-progress" package revisions whose lifecycle stage is "Draft", "Proposed",
33
-
or "DeletionProposed"
34
-
- Git repositories:
35
-
- package revisions (Kpt package file contents and directory structures)
36
-
- Package revision cache:
37
-
- Kubernetes-related metadata for package revisions (e.g. labels and annotations)
38
-
-**if the DB cache is configured:** "work-in-progress" package revisions whose lifecycle stage is "Draft", "Proposed",
39
-
or "DeletionProposed"
29
+
-**custom resource objects on the Kubernetes control plane**, such as Porch repositories (Repository objects), package variants (and by extension package variant sets) and **if** the CR cache is configured, "work-in-progress" package revisions whose lifecycle stage is "Draft", "Proposed", or "DeletionProposed"
30
+
-**Git repositories**, such as package revisions (Kpt package file contents and directory structures)
31
+
-**Package revision cache**, such as Kubernetes-related metadata for package revisions (for example, labels and annotations) and **if** the DB cache is configured, "work-in-progress" package revisions whose lifecycle stage is "Draft", "Proposed", or "DeletionProposed"
40
32
41
33
## Backup strategy
42
34
@@ -144,17 +136,13 @@ By default, before running the disaster scenarios, the suite creates, installs,
144
136
- a local Kind cluster (the "data cluster") containing:
145
137
- a Git server (Gitea)
146
138
- containing several copies of public repositories, each containing a large quantity of sample, test, and catalogued
147
-
Kpt packages
148
-
- to provide a representative workload for Porch when restoring and reconciling package revision data
149
-
- a PostgreSQL instance, set up to allow Porch to connect to and use it for the DB cache
150
-
- to make it possible to wipe and restore the database independently of Porch
151
-
- another local Kind cluster
152
-
- with Porch installed
153
-
- with DB cache connected to the PostgreSQL instance installed on the data cluster
154
-
- with the `porch-server` microservice's memory limits increased to `4GiB`
139
+
Kpt packages. This is needed to provide a representative workload for Porch when restoring and reconciling package revision data.
140
+
- a PostgreSQL instance, set up to allow Porch to connect to and use it for the DB cache. This is needed to make it possible to wipe and restore the database independently of Porch.
141
+
- another local Kind cluster with Porch installed
142
+
- with DB cache connected to the PostgreSQL instance installed on the data cluster
143
+
- with the `porch-server` microservice's memory limits increased to `4GiB`
155
144
-~115 Repository objects created in Porch, each targeted to a different combination of Git repository and directory within
156
-
the repository
157
-
- this is calculated to maximize number of package revisions and provide representative workload
145
+
the repository. This is calculated to maximize number of package revisions and provide representative workload.
158
146
- a small number of new package revisions in various lifecycle states, set up using Porch's API to allow testing that they
159
147
also will be properly backed up and restored
160
148
@@ -196,12 +184,12 @@ data stores.
196
184
197
185
Kubernetes cluster is lost with all nodes; Git repositories are lost; DB cache database is lost.
198
186
199
-
#### Data backed up:
187
+
**Data backed up:**
200
188
- Porch custom resources
201
189
- Git repository contents
202
190
- DB cache database contents
203
191
204
-
#### Data stores lost:
192
+
**Data stores lost:**
205
193
- Kubernetes control plane: entire Kubernetes cluster deleted
206
194
- Git repositories: Git server deleted and recreated empty of data
207
195
- DB cache database:
@@ -215,29 +203,23 @@ Kubernetes cluster is lost with all nodes; Git repositories are lost; DB cache d
215
203
216
204
To ensure data compatibility, backup must be restored into the DB cache of the same version of Porch.
217
205
218
-
**In step 2, ensure Porch is reinstalled with the same version as before the cluster was lost!**
206
+
In step 2, ensure Porch is reinstalled with the same version as before the cluster was lost!
219
207
{{% /alert %}}
220
208
3. Restore backed-up repository contents to Git server
221
209
4. Restore backed-up database contents to PostgreSQL server
222
210
5. Perform GitOps reconciliation, gradually (in batches of 20) re-creating all backed-up Porch Repository objects
223
211
1. For each batch, wait until all Repository objects have condition with type "Ready" and status set "True"
224
212
225
-
#### Expected data loss
226
-
227
-
None - complete recovery of state at time data was backed up.
228
-
229
-
With backups of all data stores, Porch recovers all data.
213
+
**Expected data loss:** None. Complete recovery of state at time data was backed up. With backups of all data stores, Porch recovers all data.
230
214
231
215
232
216
### 2. Kubernetes cluster loss
233
217
234
218
Kubernetes cluster is lost with all nodes; Git repositories and DB cache database remain safe.
235
219
236
-
#### Data backed up:
237
-
- Porch custom resources
220
+
**Data backed up:** Porch custom resources
238
221
239
-
#### Data stores lost:
240
-
- Kubernetes control plane: entire Kubernetes cluster deleted
222
+
**Data stores lost:** Kubernetes control plane. The entire Kubernetes cluster is deleted.
241
223
242
224
#### Restoration steps:
243
225
1. Recreate Kubernetes cluster
@@ -251,9 +233,7 @@ Kubernetes cluster is lost with all nodes; Git repositories and DB cache databas
251
233
3. Perform GitOps reconciliation, gradually (in batches of 20) re-creating all backed-up Porch Repository objects
252
234
1. For each batch, wait until all Repository objects have condition with type "Ready" and status set "True"
253
235
254
-
#### Expected data loss
255
-
256
-
None - complete recovery of state at time of cluster loss.
236
+
**Expected data loss:** None. Complete recovery of state at time of cluster loss.
257
237
258
238
Through using Git as the source of truth, we might expect Porch to automatically delete any state that only exists in the
259
239
cache - e.g., package revisions in "Draft" lifecycle stage. However, the connection between Porch and Git is represented by the Repository
@@ -266,27 +246,20 @@ cached state.
266
246
All Porch pods (by default, all in the "porch-system" namespace) are ungracefully restarted (e.g. by forcible pod deletion
267
247
with grace-period 0).
268
248
269
-
#### Data backed up:
249
+
**Data backed up:**
270
250
- Porch custom resources
271
251
- Git repository contents
272
252
- DB cache database contents
273
253
274
-
#### Data stores lost:
275
-
- None
276
-
- Porch will immediately begin to re-sync all repositories, resulting in a **decrease in quality of service** until all
277
-
repositories are deemed Ready
278
-
-**Porch API will be unavailable** to perform operations on package revisions
279
-
- `get` or `list` operations can be used to monitor Porch for API availability and repository status
254
+
**Data stores lost:** None. Porch will immediately begin to re-sync all repositories, resulting in a **decrease in quality of service** until all repositories are deemed Ready. **Porch API will be unavailable** to perform operations on package revisions. You can use `get` or `list` operations to monitor Porch for API availability and repository status.
280
255
281
256
#### Restoration steps:
282
257
1. Wait until all Porch pods return to Ready state
283
258
2. Wait until all Repository objects have condition with type "Ready" and status set "True"
284
259
1. GitOps reconciliation is unnecessary in this case since the Repository objects are unchanged
285
260
3. List package revisions periodically, monitoring results until state stabilises
286
261
287
-
#### Expected data loss
288
-
289
-
None - no data stores were impacted, but only Porch's ability to manage them, allowing for full recovery.
262
+
**Expected data loss:** None. No data stores were impacted, but only Porch's ability to manage them, allowing for full recovery.
290
263
291
264
In a representative testing environment, recovery takes **less than 5 minutes** for 115 Repository objects with a `4GiB`
292
265
memory limit applied to the `porch-server` microservice
@@ -315,24 +288,16 @@ Kubernetes cluster and Git repositories remain safe; DB cache database is lost.
315
288
Applicable only to Porch with DB cache configured.
316
289
{{% /alert %}}
317
290
318
-
#### Data backed up:
319
-
- DB cache database contents
291
+
**Data backed up:** DB cache database contents.
320
292
321
-
#### Data stores lost:
322
-
- DB cache database:
323
-
- SQL script used to drop all Porch tables
324
-
- PostgreSQL server deleted and recreated empty of data
293
+
**Data stores lost:** DB cache database (SQL script used to drop all Porch tables, PostgreSQL server deleted and recreated empty of data).
325
294
326
295
#### Restoration steps:
327
296
1. Restore backed-up database contents to PostgreSQL server
328
297
2. Perform GitOps reconciliation, gradually (in batches of 20) re-creating all backed-up Porch Repository objects
329
298
1. For each batch, wait until all Repository objects have condition with type "Ready" and status set "True"
330
299
331
-
#### Expected data loss
332
-
333
-
None - complete recovery of state at time DB cache database was backed up.
334
-
335
-
With a backup of the cache database, Porch recovers all data.
300
+
**Expected data loss:** None. Complete recovery of state at time DB cache database was backed up. With a backup of the cache database, Porch recovers all data.
336
301
337
302
### 5. DB cache loss without backup
338
303
@@ -342,13 +307,9 @@ Kubernetes cluster and Git repositories remain safe; DB cache database is lost w
342
307
Applicable only to Porch with DB cache configured.
343
308
{{% /alert %}}
344
309
345
-
#### Data backed up:
346
-
- None
310
+
**Data backed up:** None.
347
311
348
-
#### Data stores lost:
349
-
- DB cache database:
350
-
- SQL script used to drop all Porch tables
351
-
- PostgreSQL server deleted and recreated empty of data
312
+
**Data stores lost:** DB cache database (SQL script used to drop all Porch tables, PostgreSQL server deleted and recreated empty of data).
352
313
353
314
#### Restoration steps:
354
315
1. Perform GitOps reconciliation, gradually (in batches of 20) re-creating all backed-up Porch Repository objects
@@ -362,7 +323,7 @@ Applicable only to Porch with DB cache configured.
362
323
3. Wait until all Repository objects have condition with type"Ready" and status set"True"
363
324
4. List package revisions periodically, monitoring results until state stabilises
364
325
365
-
#### Expected data loss
326
+
**Expected data loss:**
366
327
367
328
All "work in progress" on package revisions lost:
368
329
- package revisions in"Draft" or "Proposed" lifecycle stages - complete loss
**Solutions**: Ensure the timestamp is at least 1 minute in the future and verify that the namespace is correct.
130
122
131
123
## Error Messages & Diagnostic Steps
132
124
@@ -213,10 +205,7 @@ See [Repository Controller Configuration]({{% relref "/docs/6_configuration_and_
213
205
**A**: Yes, periodic scheduling and one-time sync work independently. One-time synchronization executes regardless of the periodic schedule.
214
206
215
207
### Q: Why is my cron expression not working?
216
-
**A**: Porch uses standard 5-field cron format. Common mistakes:
217
-
- Using 6 fields (seconds not supported)
218
-
- Missing fields
219
-
- Invalid ranges or values
208
+
**A**: Porch uses standard 5-field cron format. Common mistakes include using 6 fields (seconds not supported), missing fields, or invalid ranges or values.
220
209
221
210
### Q: How do I stop repository syncing?
222
211
**A**: Repository synchronization cannot be completely stopped. Porch continuously monitors repositories for changes. You can only modify the sync frequency by updating the sync schedule configuration or remove custom schedules to use the default frequency.
0 commit comments