From bd1c2056510dea5652002a004e6e40a5ea8221b7 Mon Sep 17 00:00:00 2001 From: YueGuobin Date: Mon, 28 Sep 2026 22:17:51 +0800 Subject: [PATCH] docs: frozen uBridge spec for kernel impairment (netem extensions, eBPF/cBPF classifiers) --- docs/design/ubridge-kernel-impairment-spec.md | 277 ++++++++++++++++++ docs/features/docker-kernel-datapath.md | 9 +- 2 files changed, 284 insertions(+), 2 deletions(-) create mode 100644 docs/design/ubridge-kernel-impairment-spec.md diff --git a/docs/design/ubridge-kernel-impairment-spec.md b/docs/design/ubridge-kernel-impairment-spec.md new file mode 100644 index 000000000..9999456b1 --- /dev/null +++ b/docs/design/ubridge-kernel-impairment-spec.md @@ -0,0 +1,277 @@ + + +> Frozen requirements spec for the **uBridge** project. The gns3-server side is +> NOT implemented yet — it lands once uBridge delivers this surface. Everything +> marked *shipped* already exists in uBridge 1.2.3+ and is verified against +> gns3-server branch `feat/docker-kernel-filters`. + +# uBridge kernel impairment: tc netem extensions + eBPF classifiers + +## 0. Background & scope + +On the kernel datapath (Docker veth pairs enslaved into per-link Linux +bridges) frames never pass through uBridge's userspace relay, so link +impairment must live in the kernel: a **netem qdisc** on each veth host end +(serving delay / jitter / loss / dup / corrupt — *shipped*) plus **tc +classifiers** on the clsact egress hook for everything netem cannot express. + +This spec freezes: + +| Part | Content | New capability needed | +|---|---|---| +| A | netem keyword extensions: `rate`, `reorder`, loss models, jitter distributions, correlation, `seed`, `limit` | none (netlink only) | +| B | eBPF stateful classifier: `frequency_drop` (nth-packet), byte quota, time window, flow select | **CAP_BPF** | +| C | classic-BPF match-drop: the GNS3 `bpf` filter | none (netlink only) | +| D | lifecycle, composition & ordering contract, `tc reset` semantics fix | — | +| E | error strings, `tc capabilities` probe | — | +| F | test requirements | — | + +Non-goals (explicitly out of scope for now): packet mangling (`pedit`, +`csum`), mirroring (`mirred`), HTB hierarchies, RST/connection-kill +injection, ingress-side (XDP) impairment. + +All commands operate on **egress of the veth host end** (traffic entering +the container), consistent with the existing netem placement. + +--- + +## A. `tc netem set` keyword extensions *(P6a)* + +Extend the shipped grammar — existing keywords and their encoding are +frozen and unchanged: + +``` +tc netem set + [delay ] [jitter ] [distribution uniform|normal|pareto|paretonormal] [seed ] + [loss [correl ]] + [loss gemodel

[ [<1-h>]]] # mutually exclusive with plain loss + [dup [correl ]] [corrupt ] + [reorder [correl ] [gap ]] # requires delay + [rate ] # e.g. 10mbit, 512kbit + [limit ] +``` + +### A.1 Parameter validation + +| Keyword | Range / format | Errors | +|---|---|---| +| `rate` | `` = integer + unit `bit\|kbit\|mbit\|gbit\|bps\|kbps\|mbps` (decimal), max 100gbit | 204 `invalid rate ''` | +| `reorder` | pct 0–100, correl 0–100, gap 1–1000 | 204; 204 `reorder requires delay` if no `delay` present (mirror the kernel's EINVAL instead of passing it through — the server keys on the string) | +| `loss gemodel` | p 0–100, r 0–100, h as `1-h` where h is 0–100 (default r=0, 1-h=0) | 204 | +| `distribution` | one of the four names | 204 `unknown distribution ''` | +| `seed` | u32 decimal | 204 | +| `limit` | 1–1000000 packets (replaces the current fixed 1000 default) | 204 | +| `correl` suffix on loss/dup | 0–100, only directly after its keyword | 204 | + +Max token count for the command table rises accordingly (recommend 32). + +### A.2 Netlink encoding notes + +* Correlation: loss/dup/reorder use their existing nested structs with the + `correlation` field set (`tc_netem_loss`? no — see `struct tc_netem_qopt` + + `TCA_NETEM_CORR` carrying `tc_netem_corr` {delay, loss, dup}). Fill the + loss correlation there; dup likewise. +* gemodel: `TCA_NETEM_LOSS` nested attr, kind `TCA_NETEM_LOSS_GEMODEL` + (`struct tc_netem_gemodel` {p, r, h}), probabilities in the same % × 2³² + encoding as `qopt.loss`. When gemodel is set, `qopt.loss` stays 0. +* reorder: `TCA_NETEM_REORDER` (`struct tc_netem_reorder` + {probability, correlation}). +* rate: prefer `TCA_NETEM_RATE64` (u64 bits/s) and also fill legacy + `TCA_NETEM_RATE` (`tc_netem_rate`) with the truncated value for older + kernels; `slot_overhead = 0`. +* distribution: `TCA_NETEM_JITTER64` + `TCA_NETEM_DIST` must carry the + distribution table (s16 deltas). **Embed the three standard tables** + (normal / pareto / paretonormal — same values iproute2 ships as + `*.dist`, generated by netem's `maketable`) as const arrays in uBridge; + select by name. Without DIST the kernel silently uses uniform jitter. +* seed: `TCA_NETEM_SEED` (u64 allowed) — makes random draws reproducible + for tests. When omitted keep the current behaviour (kernel picks). + +### A.3 Reply contract (unchanged shape) + +``` +100-netem set on +204-invalid value '' +204-reorder requires delay +207-Could not set netem on : +``` + +--- + +## B. eBPF stateful classifier *(frequency_drop + P6b modes)* + +**One** SCHED_CLS eBPF program, loaded once per interface, attached to the +clsact egress hook. All modes are configured through one ARRAY map entry — +no program reload on parameter changes. + +### B.1 Program + +* Source `src/tc_impair.bpf.c` committed for reference; the build commits + the compiled object (`tc_impair.bpf.o`, freestanding: no CO-RE, no + BTF-typed ptr, plain `struct __sk_buff` + direct packet access). + No runtime clang/libbpf dependency. +* Maps: + * `CFG` — `BPF_MAP_TYPE_ARRAY`, 1 entry: + ``` + struct cfg { + __u32 nth; /* 0 = off, else drop every Nth packet */ + __u64 quota_bytes; /* 0 = off */ + __u32 quota_pct; /* random drop % after quota reached */ + __u64 win_start_ns; /* recurring window start (monotonic) */ + __u64 win_len_ns; /* 0 = off */ + __u32 win_pct; /* random drop % inside the window */ + __u32 flow_mask; /* bitmask: 1=src 2=dst 4=sport 8=dport 16=proto */ + __u32 flow_target; /* required hash remainder, 0 = off */ + }; + ``` + * `CNT` — `BPF_MAP_TYPE_ARRAY`, 1 entry: `{__u64 packets; __u64 bytes; __u64 nth_state;}` + Counters are updated with `__sync_fetch_and_add` on the map/global value + so the every-Nth count is exact across CPUs (requires kernel ≥ 5.1 — + acceptable; probe at load, see E). +* Evaluation order is fixed: **nth → quota → window → flow**; the first + rule that decides a drop returns `TC_ACT_SHOT`, otherwise `TC_ACT_OK`. + Random percentages use the same 2³²-probability comparison as netem + (`prandom` seeded via map so tests are reproducible). +* Verifier-friendly: no loops, bounded accesses (check `data + eth+h` before + reading fields; when flow_mask touches ports, verify L4 header length). + +### B.2 Command surface + +``` +tc nth_drop +tc quota_drop | off +tc window_drop | off +tc flow_drop | off +``` + +* First enable on an interface: create clsact (EEXIST tolerated), load the + program, attach one filter at **prio 1** (see D). Subsequent commands only + update `CFG`/reset the relevant `CNT` fields. +* `off` for every mode → the filter is removed (and the prog fd closed). + Each `off` resets that mode's counters. +* `flow_drop` mask is the decimal bitmask from B.1 (server composes it); + hash = Jenkins/equal-fold over the selected header fields modulo + `flow_target` == remainder 0 → drop. Server documents: per-flow drop only + (a filter cannot delay; delay stays netem's job). +* Window semantics: `win_start_ns`/`win_len_ns` define a recurring window on + the monotonic clock — inside the window drop with `win_pct`; outside pass. + The server computes the first start; the program advances by `len` when + `now >= start + len` (single writer — the CFG update — so this is safe). + +### B.3 Capability requirement + +`BPF_PROG_LOAD(SCHED_CLS)` needs **CAP_BPF** (or CAP_SYS_ADMIN) on kernels +≥ 5.8. Installation must therefore set: + +``` +setcap cap_bpf,cap_net_admin,cap_net_raw=ep +``` + +On EPERM: reply `210-uBridge lacks CAP_BPF (setcap cap_bpf,cap_net_admin,cap_net_raw=ep) and the kernel requires it for stateful filters` +— the gns3-server side falls back to the relay datapath on this string (E). + +--- + +## C. classic-BPF match drop (the GNS3 `bpf` filter) + +No eBPF, no CAP_BPF: `cls_bpf` accepts **classic bytecode** via +`TCA_BPF_BYTECODE` (`sock_fprog`), and the kernel internally migrates it to +eBPF. uBridge already links libpcap. + +``` +tc bpf_drop add "" # prio: 10..99, server-assigned +tc bpf_drop flush +``` + +* `add`: `pcap_compile(DLT_EN10MB, snaplen 65535)` the expression; on + failure reply + `209-Cannot compile filter '': ` — gns3-server keys on + this exact prefix (same shape as the relay's error so one regex serves + both datapaths). Netlink: RTM_NEWTFILTER on clsact egress, protocol + ETH_P_ALL, `TCA_KIND "bpf"`, `TCA_BPF_BYTECODE`, action nested + `TCA_BPF_ACT` → `gact` with `action = TC_ACT_SHOT` (pcap match ≠ 0 → + filter matches → drop; non-match falls through, no action → pass). +* Multiple expressions (GNS3 sends one line per filter): server adds one + filter per line at successive prios — any match drops (OR). +* `flush`: delete every bpf_drop filter uBridge added on this interface + (track prios in-process), tolerate ENOENT per filter. Does NOT touch the + eBPF prio-1 filter or netem. +* Update-by-replace: server always `flush` + re-`add` (mirrors `netem set`). + +--- + +## D. Lifecycle, composition & ordering (frozen contract) + +* **clsact coexists with the netem root qdisc** — never replace the root. +* Execution order on egress: classifier filters by ascending prio — + eBPF impair filter at **prio 1**, `bpf_drop` at **prio 10..99** — then the + root netem qdisc. Dropped packets therefore never reach netem **nor the + AF_PACKET tap points** (capture/markers will not observe cls_bpf-dropped + frames; documented server-side). +* **`tc reset ` becomes the full restore** (behaviour change + fix): + 1. remove all bpf_drop filters and the eBPF filter (clsact egress side we + own), 2. delete clsact, 3. delete the root qdisc. On an untouched + interface it must now reply **100-OK** (idempotent — currently returns + 207-ENOENT; gns3-server tolerates both, but the new contract is OK). +* All state is kernel state tied to the interface: deleting the veth + (container stop) clears everything; no ubridge-side bookkeeping survives + a restart (a re-attached link rebuilds from the NIO, as today). + +--- + +## E. Errors & capability probe + +Error strings are contract (gns3-server matches them): + +| Reply | Meaning | +|---|---| +| `100-netem set on ` / `100-qdisc reset on ` | success | +| `203-Bad number of parameters (%d with min/max=%d/%d)` | argc mismatch | +| `204-invalid value ''` | value validation | +| `204-reorder requires delay` | keyword dependency | +| `207-Could not (set netem\|reset qdisc) on : ` | netlink/kernel failure | +| `209-Cannot compile filter '': ` | bpf_drop pcap_compile failure | +| `210-uBridge lacks CAP_BPF ...` | stateful filters unavailable | + +New introspection command (also serves the gns3-server degradation TODO — +old uBridge without these commands simply stays on the relay datapath): + +``` +tc capabilities +100-netem=delay,jitter,loss,dup,corrupt,rate,reorder,gemodel,dist,seed,limit;ebpf=1;cbpf=1 +``` + +`ebpf=0` when the load probe fails at startup (kernel < 5.1 or no CAP_BPF); +`cbpf=0` when RTM_NEWTFILTER/cls_bpf is unavailable. gns3-server hides the +corresponding filter types per capability. + +--- + +## F. Test requirements + +* **Unit**: per keyword → expected netlink attrs (assert message bytes); + every 204 table row; `bpf_drop` compile-fail path; capabilities string. +* **Functional** (root, veth pair + netns + nsenter ping): + delay+reorder observable (mdev), gemodel loss within ±5% of target, + rate within ±10% (measured byte throughput), nth_drop **exact** pattern + (ICMP seq survives 1,2,…,N-1, Nth dropped), quota/window/flow with + crafted senders, `tc reset` idempotent on a clean interface, P5 surface + regression (delay/jitter/loss/dup/corrupt, REPLACE semantics). +* Determinism: netem `seed` + the CFG-seeded prandom make runs repeatable; + CI asserts identical drop patterns for identical seeds. + +--- + +## gns3-server alignment (informational — not uBridge scope) + +* P6a types map 1:1 onto new GNS3 filter entries (rate, reorder, gemodel…), + translated in `DockerVM._ubridge_apply_netem`. +* `frequency_drop` / `bpf` move out of `KERNEL_UNSUPPORTED_FILTERS` once + `tc capabilities` reports `ebpf=1` / `cbpf=1` (per-compute probe, cached); + without the capability the link stays relay-wired exactly as today. +* Every veth end owns one qdisc + its filters: per-direction impairment is + an architectural freebie to expose later (API `direction` field), aligned + with marker `dir` semantics. diff --git a/docs/features/docker-kernel-datapath.md b/docs/features/docker-kernel-datapath.md index 21c95f352..a8a2e6da6 100644 --- a/docs/features/docker-kernel-datapath.md +++ b/docs/features/docker-kernel-datapath.md @@ -168,8 +168,8 @@ ends (uBridge `tc netem set`, raw netlink — no `tc` binary needed): | `delay [ms, jitter]` | `delay X jitter Y` | jitter 0 omitted | | `packet_loss [%]` | `loss P` | per direction (see below) | | `corrupt [%]` | `corrupt P` | | -| `frequency_drop` | — no netem equivalent | 409 on kernel links; relay fallback | -| `bpf` | — needs a tc classifier | 409 on kernel links; relay fallback | +| `frequency_drop` | — no netem equivalent | 409 on kernel links; relay fallback (eBPF classifier spec'd: [ubridge-kernel-impairment-spec](../design/ubridge-kernel-impairment-spec.md)) | +| `bpf` | — needs a tc classifier | 409 on kernel links; relay fallback (cBPF match-drop spec'd, same doc) | Semantics: @@ -192,6 +192,11 @@ Semantics: * A link carrying `frequency_drop` or `bpf` is wired on the **relay** (they only exist in uBridge's userspace filters); `available_filters` hides them on kernel links and setting them returns 409 with a clear message. + Kernel-side implementations are frozen in + [docs/design/ubridge-kernel-impairment-spec.md](../design/ubridge-kernel-impairment-spec.md) + (eBPF stateful classifier for `frequency_drop` + quota/window/flow modes, + classic-BPF match-drop for `bpf`, plus netem keyword extensions: rate, + reorder, gemodel loss, jitter distributions). ## uBridge command surface