Skip to content

Commit e27cb74

Browse files
w13915984028w13915984028
authored
Update the global ip pool (#1056)
Signed-off-by: Jian Wang <jian.wang@suse.com>
1 parent 4503084 commit e27cb74

5 files changed

Lines changed: 130 additions & 50 deletions

File tree

‎docs/networking/ippool.md‎

Lines changed: 34 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ To create a new IP pool:
3131
1. Specify the **Name** of the IP pool.
3232
1. Go to the **Range** tab to specify the **IP ranges** for the IP pool. You can add multiple IP ranges.
3333
![](/img/v1.2/networking/multiple-ranges.png)
34-
1. Go to the **Selector** tab to specify the **Scope** and **Priority** of the IP pool.
34+
1. Go to the **Selector** tab to specify the **Selector**, **Scope** and **Priority** of the IP pool.
3535
![](/img/v1.2/networking/ippool-scope.png)
3636
3737
When you operate from the Harvester UI, the `Scope` only includes `Namespace`. Click `Add Scope` to add new items.
@@ -44,20 +44,37 @@ If the Harvester cluster is imported to `Rancher Manager` from `Rancher Manager
4444

4545
The `Scope` includes `Project`, `Namespace` and `Guest Kubernetes Cluster`. For more information, see [Multi-Tenancy Example](../rancher/virtualization-management.md#multi-tenancy-example) and [Projects and Kubernetes Namespaces with Rancher](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/manage-clusters/projects-and-namespaces#about-projects).
4646

47-
When a pool has only one `Scope` and each selects `All`, then this IP Pool is marked as `global` automatically.
47+
When a pool has only one `Scope` and each selects `All`, then this IP Pool is marked as **global** automatically.
4848

4949
## Selection policy
50+
5051
Each IP pool will have a specific range, and you can specify the corresponding requirements in the LB `annotations`. IP pools that meet the specified requirements will automatically assign IP addresses to LBs.
5152

5253
- LBs utilize the following annotations to express requirements (all annotations are optional):
53-
- `loadbalancer.harvesterhci.io/network` specifies the VM network the guest cluster nodes use.
54+
- `loadbalancer.harvesterhci.io/network` specifies the VM network the guest cluster nodes use. If this annotation is omitted, the mutator automatically selects the first Multus network defined in the guest cluster's virtual machine instances.
5455
- `loadbalancer.harvesterhci.io/project` and `loadbalancer.harvesterhci.io/namespace` identify the project and namespace of the VMs that comprise the guest cluster.
5556
- `loadbalancer.harvesterhci.io/cluster` denotes the name of the guest cluster.
5657
- The IP pool has a selector, including network and scope, to match the requirements of the LB.
57-
- Network is a hard condition. The optional IP pool must match the value of the LB annotation `loadbalancer.harvesterhci.io/network`.
58-
- Every IP pool, except the global IP pool, has a unique scope different from others if its priority is `0`. The project, namespace, or cluster name of LBs should be in the scope of the IP pool if they want to get an IP from this pool.
58+
- Network is a **hard condition**. The target IP pool must match the value of the LB annotation `loadbalancer.harvesterhci.io/network`.
59+
- Every IP pool, except the **global** IP pool, has a unique scope different from others if its priority is `0`. The project, namespace, or cluster name of LBs should be in the scope of the IP pool if they want to get an IP from this pool.
5960
- `spec.selector.priority` specifies the priority of the IP Pool. The larger the number, the higher the priority. If the priority is not `0`, the value should differ. The priority helps you to migrate the old IP pool to the new one.
60-
- If the IP Pool has a scope that matches all projects, namespaces, and guest clusters, it's called a global IP pool, and only one global IP pool is allowed. If there is no IP pool matching the requirements of the LB, the IPAM will allocate an IP address from the global IP pool if it exists.
61+
- If the IP Pool has a scope that matches all projects, namespaces, and guest clusters, it's called a **global** IP pool, and only one **global** IP pool is allowed under one **Network**. If there is no IP pool matching the requirements of the LB, the IPAM will allocate an IP address from the **global** IP pool if it exists and matches the **Network**.
62+
63+
:::caution
64+
65+
**Legacy Behavior (Harvester v1.8 and earlier):**
66+
67+
- The entire system was restricted to a maximum of one **global** IP pool.
68+
- Network matching was not enforced, meaning a **global** IP pool could inadvertently allocate IPs to the incorrect network.
69+
70+
**Current Behavior (Harvester v1.9.0 and later):**
71+
72+
- You may define at most one **global** IP pool per specific VM Network.
73+
- Network matching is strictly enforced. An IP pool with an empty `.spec.selector.network` will **not** be selected for allocation, even if it is marked as **global**.
74+
75+
- For guidance on resource allocation, see [IPPool Best Practices](#ippool-for-guest-cluster-type-loadbalancer).
76+
77+
:::
6178

6279
### Examples
6380
- **Example 1:** You wish to set up an IP pool within the range `192.168.100.0/24` for the `default` namespace. In this scenario, all load balancers within the `default` namespace will receive an IP address from this designated IP pool:
@@ -119,12 +136,13 @@ Each IP pool will have a specific range, and you can specify the corresponding r
119136
kind: IPPool
120137
metadata:
121138
name: global-ip-pool
122-
labels:
123-
loadbalancer.harvesterhci.io/global-ip-pool: 'true' # Added by the controller automatically
139+
labels:
140+
loadbalancer.harvesterhci.io/global-ip-pool: 'true' # Added automatically by Harvester
124141
spec:
125142
ranges:
126143
- subnet: 192.168.20.0/24
127144
selector:
145+
network: default/vlan1
128146
scope:
129147
- project: "*"
130148
namespace: "*"
@@ -145,7 +163,13 @@ Each IP pool will have a specific range, and you can specify the corresponding r
145163

146164
### IPPool for Guest Cluster type Loadbalancer
147165

148-
1. It is better to [Create IP Pool from Rancher Managery UI](#create-ip-pool-from-rancher-manager-ui), because it allows you to tune `Project` and `Guest Kubernetes Cluster` for better resource management and isolation.
166+
1. It is better to [Create IP Pool from Rancher Managery UI](#create-ip-pool-from-rancher-manager-ui), because it allows you to tune `Project`, `Namespace` and `Guest Kubernetes Cluster` for better resource management and isolation.
167+
168+
1. Be careful when creating **global** IP pool as one guest cluster might allocate too many IPs and starve other clusters. The pool can't be deleted if any of the IPs is still in use.
149169

150-
1. Be careful when creating global IP pool as one guest cluster might allocate too many IPs and starve other clusters. The pool can't be deleted if any of the IPs is still in use.
170+
1. Starting with v1.9.0, an IP pool with an empty `.spec.selector.network` will not be selected for allocation, even if it is marked as **global**. Always set the correct `.spec.selector.network`.
151171

172+
1. **Use project/namespace-scoped pools for better isolation:** Since each guest cluster is deployed within a specific project and namespace, you should create IP pools scoped to that project or namespace with defined IP ranges. This approach offers several advantages:
173+
* **Granular Control:** A single network can support multiple non-global pools, allowing you to partition IP availability across different projects or namespaces.
174+
* **Resource Protection:** The exhaustion of one pool does not affect others, even though they share the same underlying network.
175+
* **Scalability:** It is easier to increase an IP pool's range than to decrease it. Shrinking a range is difficult when the currently used IPs are dispersed throughout the range (e.g., a pool of [1-10] where IPs 1, 5, and 9 are already in use). Consequently, a best practice is to start with a smaller, conservative range and expand as needed.

‎versioned_docs/version-v1.5/networking/ippool.md‎

Lines changed: 24 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ To create a new IP pool:
3131
1. Specify the **Name** of the IP pool.
3232
1. Go to the **Range** tab to specify the **IP ranges** for the IP pool. You can add multiple IP ranges.
3333
![](/img/v1.2/networking/multiple-ranges.png)
34-
1. Go to the **Selector** tab to specify the **Scope** and **Priority** of the IP pool.
34+
1. Go to the **Selector** tab to specify the **Selector**, **Scope** and **Priority** of the IP pool.
3535
![](/img/v1.2/networking/ippool-scope.png)
3636
3737
When you operate from the Harvester UI, the `Scope` only includes `Namespace`. Click `Add Scope` to add new items.
@@ -44,20 +44,29 @@ If the Harvester cluster is imported to `Rancher Manager` from `Rancher Manager
4444

4545
The `Scope` includes `Project`, `Namespace` and `Guest Kubernetes Cluster`. For more information, see [Multi-Tenancy Example](../rancher/virtualization-management.md#multi-tenancy-example) and [Projects and Kubernetes Namespaces with Rancher](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/manage-clusters/projects-and-namespaces#about-projects).
4646

47-
When a pool has only one `Scope` and each selects `All`, then this IP Pool is marked as `global` automatically.
47+
When a pool has only one `Scope` and each selects `All`, then this IP Pool is marked as **global** automatically.
4848

4949
## Selection policy
50+
5051
Each IP pool will have a specific range, and you can specify the corresponding requirements in the LB `annotations`. IP pools that meet the specified requirements will automatically assign IP addresses to LBs.
5152

5253
- LBs utilize the following annotations to express requirements (all annotations are optional):
53-
- `loadbalancer.harvesterhci.io/network` specifies the VM network the guest cluster nodes use.
54+
- `loadbalancer.harvesterhci.io/network` specifies the VM network the guest cluster nodes use. If this annotation is omitted, the mutator automatically selects the first Multus network defined in the guest cluster's virtual machine instances.
5455
- `loadbalancer.harvesterhci.io/project` and `loadbalancer.harvesterhci.io/namespace` identify the project and namespace of the VMs that comprise the guest cluster.
5556
- `loadbalancer.harvesterhci.io/cluster` denotes the name of the guest cluster.
5657
- The IP pool has a selector, including network and scope, to match the requirements of the LB.
57-
- Network is a hard condition. The optional IP pool must match the value of the LB annotation `loadbalancer.harvesterhci.io/network`.
58-
- Every IP pool, except the global IP pool, has a unique scope different from others if its priority is `0`. The project, namespace, or cluster name of LBs should be in the scope of the IP pool if they want to get an IP from this pool.
58+
- Network is a **hard condition**. The target IP pool must match the value of the LB annotation `loadbalancer.harvesterhci.io/network`.
59+
- Every IP pool, except the **global** IP pool, has a unique scope different from others if its priority is `0`. The project, namespace, or cluster name of LBs should be in the scope of the IP pool if they want to get an IP from this pool.
5960
- `spec.selector.priority` specifies the priority of the IP Pool. The larger the number, the higher the priority. If the priority is not `0`, the value should differ. The priority helps you to migrate the old IP pool to the new one.
60-
- If the IP Pool has a scope that matches all projects, namespaces, and guest clusters, it's called a global IP pool, and only one global IP pool is allowed. If there is no IP pool matching the requirements of the LB, the IPAM will allocate an IP address from the global IP pool if it exists.
61+
- If the IP Pool has a scope that matches all projects, namespaces, and guest clusters, it's called a **global** IP pool, and only one **global** IP pool is allowed. If there is no IP pool matching the requirements of the LB, the IPAM will allocate an IP address from the **global** IP pool if it exists.
62+
63+
:::caution
64+
65+
- Network matching was not enforced for **global** IP pool, meaning a **global** IP pool could inadvertently allocate IPs to the incorrect network.
66+
67+
- For guidance on mitigating these risks and managing resource allocation, see [IPPool Best Practices](#ippool-for-guest-cluster-type-loadbalancer).
68+
69+
:::
6170

6271
### Examples
6372
- **Example 1:** You wish to set up an IP pool within the range `192.168.100.0/24` for the `default` namespace. In this scenario, all load balancers within the `default` namespace will receive an IP address from this designated IP pool:
@@ -119,8 +128,8 @@ Each IP pool will have a specific range, and you can specify the corresponding r
119128
kind: IPPool
120129
metadata:
121130
name: global-ip-pool
122-
labels:
123-
loadbalancer.harvesterhci.io/global-ip-pool: 'true' # Added by the controller automatically
131+
labels:
132+
loadbalancer.harvesterhci.io/global-ip-pool: 'true' # Added automatically by Harvester
124133
spec:
125134
ranges:
126135
- subnet: 192.168.20.0/24
@@ -145,7 +154,12 @@ Each IP pool will have a specific range, and you can specify the corresponding r
145154

146155
### IPPool for Guest Cluster type Loadbalancer
147156

148-
1. It is better to [Create IP Pool from Rancher Managery UI](#create-ip-pool-from-rancher-manager-ui), because it allows you to tune `Project` and `Guest Kubernetes Cluster` for better resource management and isolation.
157+
1. It is better to [Create IP Pool from Rancher Managery UI](#create-ip-pool-from-rancher-manager-ui), because it allows you to tune `Project`, `Namespace` and `Guest Kubernetes Cluster` for better resource management and isolation.
149158

150-
1. Be careful when creating global IP pool as one guest cluster might allocate too many IPs and starve other clusters. The pool can't be deleted if any of the IPs is still in use.
159+
1. Be careful when creating **global** IP pool as one guest cluster might allocate too many IPs and starve other clusters. The pool can't be deleted if any of the IPs is still in use. And be aware that versions prior to v1.9.0 lacked mandatory network matching, which could lead to IP allocation across incorrect networks.
151160

161+
1. **Use project/namespace-scoped pools for better isolation:** Since each guest cluster is deployed within a specific project and namespace, you should create IP pools scoped to that project or namespace with defined IP ranges. This approach offers several advantages:
162+
* **Granular Control:** A single network can support multiple non-global pools, allowing you to partition IP availability across different projects or namespaces.
163+
* **Resource Protection:** The exhaustion of one pool does not affect others, even though they share the same underlying network.
164+
* **Reduced Risk:** Because these pools are not **global**, they remain isolated from the risk of incorrect network selection inherent to **global** IP pools.
165+
* **Scalability:** It is easier to increase an IP pool's range than to decrease it. Shrinking a range is difficult when the currently used IPs are dispersed throughout the range (e.g., a pool of [1-10] where IPs 1, 5, and 9 are already in use). Consequently, a best practice is to start with a smaller, conservative range and expand as needed.

0 commit comments

Comments
 (0)