Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 130 additions & 0 deletions docs/en/solutions/Handling_VPC_Egress_Gateway_BFD_session_issues.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
products:
- Alauda Container Platform
kind:
- Troubleshooting
ProductsVersion:
- 4.3.0,4.3.1,4.3.2
id: TBD
---

# Handle VPC Egress Gateway BFD session issues

This article provides solutions for two VPC Egress Gateway (VEG) BFD session issues. Select the scenario that matches the observed behavior.

## Scenario 1: BFD sessions repeatedly transition between Up and Down

**Applicable versions:** ACP v4.3.0 and v4.3.1

### Symptoms

After BFD is enabled for the VEG, `bfdd-control status` reports one or more sessions that repeatedly transition between `Up` and `Down`. Application outbound traffic can be interrupted when OVN marks the corresponding VEG next hop as unavailable.

### Causes

BFD packets must be processed within the negotiated detection time. The following conditions can cause an established session to time out repeatedly:

- In ACP v4.3.0 and v4.3.1, the default CPU request and limit for the VEG `bfdd` container are both `50m`. CPU throttling or node scheduling delays can prevent `bfdd` from processing BFD packets in time.
- When the VEG uses `minRX: 100` and `minTX: 100`, the session is more sensitive to brief scheduling delays and packet loss.

These conditions can occur independently or together.

### Resolution

Upgrade to ACP v4.3.2 or later. The default CPU request and limit for the VEG `bfdd` container are both `200m` in these versions.

If you cannot upgrade ACP, apply the following changes to the affected VEG.

:::warning
Changing `spec.resources` or `spec.bfd` rolls out the VEG Deployment and recreates its Pods. For a single-replica VEG, schedule a maintenance window because application outbound traffic can be interrupted during the rollout.
:::

Edit the affected VEG:

```bash
kubectl -n <namespace> edit veg <veg-name>
```

Set the CPU request and limit to `200m`:

```yaml
spec:
resources:
requests:
cpu: 200m
limits:
cpu: 200m
```

If the session remains unstable after increasing the CPU, and the VEG explicitly uses a 100 ms detection interval, set `minRX` and `minTX` to `1000`:

```yaml
spec:
bfd:
minRX: 1000
minTX: 1000
```

Increasing the interval reduces sensitivity to brief scheduling delays and packet loss, but increases failure detection time. With a 1000 ms interval and a `multiplier` of `5`, BFD takes approximately 5 seconds to detect a failure, excluding the time OVN needs to update the forwarding path.

## Scenario 2: No BFD sessions, but the health check succeeds

**Applicable versions:** ACP v4.3.0, v4.3.1, and v4.3.2

### Symptoms

After BFD is enabled for the VEG, `bfdd-control status` reports `There are 0 sessions:`, but the VEG Pod remains `Running` and `Ready`. The corresponding VEG next hop can become unavailable and interrupt application outbound traffic.

### Cause

The existing `bfdd` health check runs `bfdd-control status` directly. The command returns exit code `0` when it successfully communicates with the local daemon, even when no BFD sessions exist:

| BFD state | Example output | Exit code |
| --- | --- | --- |
| A session exists | `There are 1 sessions:` | `0` |
| No session exists | `There are 0 sessions:` | `0` |

Kubernetes exec probes evaluate only the command exit code and do not interpret its output. Therefore, the existing liveness probe does not treat an empty session table as a failure, and the kubelet does not restart the `bfdd` container.

### Resolution

Upgrade to ACP v4.3.3 or later. These versions treat an empty BFD session table as a health-check failure.

If the cluster must remain on ACP v4.3.0, v4.3.1, or v4.3.2, use the following temporary workaround.

:::warning
Updating the Deployment recreates VEG Pods. For a single-replica VEG, schedule a maintenance window because application outbound traffic can be interrupted during the rollout.

The VEG Deployment is generated by the Kube-OVN controller. This manual change can be overwritten when the VEG specification or replica count changes, Kube-OVN is upgraded, or the VEG or Deployment is recreated.
:::

Set the VEG namespace and name, and obtain the generated Deployment name:

```bash
NAMESPACE="<veg-namespace>"
VEG_NAME="<veg-name>"

DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \
-l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \
-o jsonpath='{.items[0].metadata.name}')"
```

Edit the generated Deployment:

```bash
kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}"
```

In the `bfdd` container, modify `exec.command` in the existing `livenessProbe` and set `timeoutSeconds` to `10`:

```yaml
livenessProbe:
exec:
command:
- bash
- -ec
- 'output="$(bfdd-control status)"; ! grep -q "^There are 0 sessions:" <<< "${output}"'
timeoutSeconds: 10
```

This change detects an empty session table and attempts recovery by restarting the `bfdd` container. If the session table remains empty after the restart, the change can only expose the failure; it cannot repair the BFD session. Investigate the OVN BFD peer, network path, and BFD configuration.
131 changes: 131 additions & 0 deletions docs/zh/solutions/Handling_VPC_Egress_Gateway_BFD_session_issues.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
---
products:
- Alauda Container Platform
kind:
- Troubleshooting
ProductsVersion:
- 4.3.0,4.3.1,4.3.2
id: TBD
sourceSHA: c48fd57ace1a3d90fe76bc3d76b1fdd85309e959e82d509b119541ca01507338
---

# VPC 出口网关 BFD 会话异常的处理方法

本文提供两类 VPC 出口网关(VEG)BFD 会话异常的处理方法。请根据现场现象选择对应场景。

## 场景 1:BFD 会话频繁在 Up 和 Down 之间切换

**适用版本:** ACP v4.3.0 和 v4.3.1

### 问题现象

VEG 启用 BFD 后,`bfdd-control status` 能够查询到一个或多个会话,但会话频繁在 `Up` 和 `Down` 之间切换。当 OVN 将对应的 VEG 下一跳标记为不可用时,业务出网可能中断。

### 原因

BFD 报文必须在协商出的检测时间内完成处理。以下情况可能导致已建立的会话反复超时:

- ACP v4.3.0 和 v4.3.1 中,VEG `bfdd` 容器的默认 CPU request 和 limit 均为 `50m`。CPU 限流或节点调度延迟可能导致 `bfdd` 无法及时处理 BFD 报文。
- VEG 使用 `minRX: 100` 和 `minTX: 100` 时,会话对短暂的调度延迟和网络丢包更加敏感。

以上情况可能独立出现,也可能同时存在。

### 解决方案

升级到 ACP v4.3.2 或更高版本。该版本已将 VEG `bfdd` 容器的默认 CPU request 和 limit 调整为 `200m`。

如果暂时无法升级 ACP,请对受影响的 VEG 应用以下调整。

:::warning
修改 `spec.resources` 或 `spec.bfd` 会滚动更新 VEG Deployment 并重建 Pod。对于单副本 VEG,请安排维护窗口,因为滚动更新期间业务出网可能中断。
:::

编辑受影响的 VEG:

```bash
kubectl -n <namespace> edit veg <veg-name>
```

将 CPU request 和 limit 调整为 `200m`:

```yaml
spec:
resources:
requests:
cpu: 200m
limits:
cpu: 200m
```

增加 CPU 后,如果会话仍不稳定,并且 VEG 显式使用 100 ms 探测间隔,请将 `minRX` 和 `minTX` 调整为 `1000`:

```yaml
spec:
bfd:
minRX: 1000
minTX: 1000
```

增加探测间隔可以降低会话对短暂调度延迟和网络丢包的敏感度,但也会增加故障检测时间。当探测间隔为 1000 ms、`multiplier` 为 `5` 时,BFD 故障检测时间约为 5 秒,不包含 OVN 更新转发路径所需的时间。

## 场景 2:BFD 会话为零,但健康检查仍然通过

**适用版本:** ACP v4.3.0、v4.3.1 和 v4.3.2

### 问题现象

VEG 启用 BFD 后,`bfdd-control status` 显示 `There are 0 sessions:`,但 VEG Pod 仍处于 `Running` 和 `Ready` 状态。对应的 VEG 下一跳可能变为不可用,并造成业务出网中断。

### 原因

现有 `bfdd` 健康检查直接执行 `bfdd-control status`。该命令只要能够成功连接本地守护进程,就会返回退出码 `0`,即使本地不存在 BFD 会话:

| BFD 状态 | 示例输出 | 退出码 |
| --- | --- | --- |
| 存在 BFD 会话 | `There are 1 sessions:` | `0` |
| 不存在 BFD 会话 | `There are 0 sessions:` | `0` |

Kubernetes exec 探针只判断命令退出码,不解析命令输出。因此,现有 livenessProbe 无法将会话表为空识别为检查失败,kubelet 也不会重启 `bfdd` 容器。

### 解决方案

升级到 ACP v4.3.3 或更高版本。该版本会将 BFD 会话表为空识别为健康检查失败。

如果集群暂时必须继续使用 ACP v4.3.0、v4.3.1 或 v4.3.2,请使用以下临时方案。

:::warning
更新 Deployment 会重建 VEG Pod。对于单副本 VEG,请安排维护窗口,因为滚动更新期间业务出网可能中断。

VEG Deployment 由 Kube-OVN 控制器生成。修改 VEG spec 或副本数、升级 Kube-OVN,或者重建 VEG 或 Deployment 时,手工修改的探针可能被覆盖。
:::

设置 VEG 所在的命名空间和名称,并获取控制器生成的 Deployment 名称:

```bash
NAMESPACE="<veg-namespace>"
VEG_NAME="<veg-name>"

DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \
-l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \
-o jsonpath='{.items[0].metadata.name}')"
```

编辑控制器生成的 Deployment:

```bash
kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}"
```

找到名为 `bfdd` 的容器,修改其现有 `livenessProbe` 中的 `exec.command`,并将 `timeoutSeconds` 调整为 `10`:

```yaml
livenessProbe:
exec:
command:
- bash
- -ec
- 'output="$(bfdd-control status)"; ! grep -q "^There are 0 sessions:" <<< "${output}"'
timeoutSeconds: 10
```

该修改用于识别空会话,并尝试通过重启 `bfdd` 容器恢复。如果重启后会话仍为空,该功能只能暴露异常,不能修复 BFD 会话,需要继续排查 OVN BFD peer、网络路径和 BFD 配置。
Loading