mirror of
https://github.com/GNS3/gns3-server.git
synced 2026-10-21 18:32:23 +03:00
278 lines
12 KiB
Markdown
278 lines
12 KiB
Markdown
<!--
|
||
SPDX-License-Identifier: CC-BY-SA-4.0
|
||
See LICENSE file for licensing information.
|
||
-->
|
||
|
||
> Frozen requirements spec for the **uBridge** project. The gns3-server side is
|
||
> NOT implemented yet — it lands once uBridge delivers this surface. Everything
|
||
> marked *shipped* already exists in uBridge 1.2.3+ and is verified against
|
||
> gns3-server branch `feat/docker-kernel-filters`.
|
||
|
||
# uBridge kernel impairment: tc netem extensions + eBPF classifiers
|
||
|
||
## 0. Background & scope
|
||
|
||
On the kernel datapath (Docker veth pairs enslaved into per-link Linux
|
||
bridges) frames never pass through uBridge's userspace relay, so link
|
||
impairment must live in the kernel: a **netem qdisc** on each veth host end
|
||
(serving delay / jitter / loss / dup / corrupt — *shipped*) plus **tc
|
||
classifiers** on the clsact egress hook for everything netem cannot express.
|
||
|
||
This spec freezes:
|
||
|
||
| Part | Content | New capability needed |
|
||
|---|---|---|
|
||
| A | netem keyword extensions: `rate`, `reorder`, loss models, jitter distributions, correlation, `seed`, `limit` | none (netlink only) |
|
||
| B | eBPF stateful classifier: `frequency_drop` (nth-packet), byte quota, time window, flow select | **CAP_BPF** |
|
||
| C | classic-BPF match-drop: the GNS3 `bpf` filter | none (netlink only) |
|
||
| D | lifecycle, composition & ordering contract, `tc reset` semantics fix | — |
|
||
| E | error strings, `tc capabilities` probe | — |
|
||
| F | test requirements | — |
|
||
|
||
Non-goals (explicitly out of scope for now): packet mangling (`pedit`,
|
||
`csum`), mirroring (`mirred`), HTB hierarchies, RST/connection-kill
|
||
injection, ingress-side (XDP) impairment.
|
||
|
||
All commands operate on **egress of the veth host end** (traffic entering
|
||
the container), consistent with the existing netem placement.
|
||
|
||
---
|
||
|
||
## A. `tc netem set` keyword extensions *(P6a)*
|
||
|
||
Extend the shipped grammar — existing keywords and their encoding are
|
||
frozen and unchanged:
|
||
|
||
```
|
||
tc netem set <if>
|
||
[delay <ms>] [jitter <ms>] [distribution uniform|normal|pareto|paretonormal] [seed <u32>]
|
||
[loss <pct> [correl <pct>]]
|
||
[loss gemodel <p> [<r> [<1-h>]]] # mutually exclusive with plain loss
|
||
[dup <pct> [correl <pct>]] [corrupt <pct>]
|
||
[reorder <pct> [correl <pct>] [gap <n>]] # requires delay
|
||
[rate <bw>] # e.g. 10mbit, 512kbit
|
||
[limit <pkts>]
|
||
```
|
||
|
||
### A.1 Parameter validation
|
||
|
||
| Keyword | Range / format | Errors |
|
||
|---|---|---|
|
||
| `rate` | `<bw>` = integer + unit `bit\|kbit\|mbit\|gbit\|bps\|kbps\|mbps` (decimal), max 100gbit | 204 `invalid rate '<v>'` |
|
||
| `reorder` | pct 0–100, correl 0–100, gap 1–1000 | 204; 204 `reorder requires delay` if no `delay` present (mirror the kernel's EINVAL instead of passing it through — the server keys on the string) |
|
||
| `loss gemodel` | p 0–100, r 0–100, h as `1-h` where h is 0–100 (default r=0, 1-h=0) | 204 |
|
||
| `distribution` | one of the four names | 204 `unknown distribution '<v>'` |
|
||
| `seed` | u32 decimal | 204 |
|
||
| `limit` | 1–1000000 packets (replaces the current fixed 1000 default) | 204 |
|
||
| `correl` suffix on loss/dup | 0–100, only directly after its keyword | 204 |
|
||
|
||
Max token count for the command table rises accordingly (recommend 32).
|
||
|
||
### A.2 Netlink encoding notes
|
||
|
||
* Correlation: loss/dup/reorder use their existing nested structs with the
|
||
`correlation` field set (`tc_netem_loss`? no — see `struct tc_netem_qopt`
|
||
+ `TCA_NETEM_CORR` carrying `tc_netem_corr` {delay, loss, dup}). Fill the
|
||
loss correlation there; dup likewise.
|
||
* gemodel: `TCA_NETEM_LOSS` nested attr, kind `TCA_NETEM_LOSS_GEMODEL`
|
||
(`struct tc_netem_gemodel` {p, r, h}), probabilities in the same % × 2³²
|
||
encoding as `qopt.loss`. When gemodel is set, `qopt.loss` stays 0.
|
||
* reorder: `TCA_NETEM_REORDER` (`struct tc_netem_reorder`
|
||
{probability, correlation}).
|
||
* rate: prefer `TCA_NETEM_RATE64` (u64 bits/s) and also fill legacy
|
||
`TCA_NETEM_RATE` (`tc_netem_rate`) with the truncated value for older
|
||
kernels; `slot_overhead = 0`.
|
||
* distribution: `TCA_NETEM_JITTER64` + `TCA_NETEM_DIST` must carry the
|
||
distribution table (s16 deltas). **Embed the three standard tables**
|
||
(normal / pareto / paretonormal — same values iproute2 ships as
|
||
`*.dist`, generated by netem's `maketable`) as const arrays in uBridge;
|
||
select by name. Without DIST the kernel silently uses uniform jitter.
|
||
* seed: `TCA_NETEM_SEED` (u64 allowed) — makes random draws reproducible
|
||
for tests. When omitted keep the current behaviour (kernel picks).
|
||
|
||
### A.3 Reply contract (unchanged shape)
|
||
|
||
```
|
||
100-netem set on <if>
|
||
204-invalid <kw> value '<v>'
|
||
204-reorder requires delay
|
||
207-Could not set netem on <if>: <strerror>
|
||
```
|
||
|
||
---
|
||
|
||
## B. eBPF stateful classifier *(frequency_drop + P6b modes)*
|
||
|
||
**One** SCHED_CLS eBPF program, loaded once per interface, attached to the
|
||
clsact egress hook. All modes are configured through one ARRAY map entry —
|
||
no program reload on parameter changes.
|
||
|
||
### B.1 Program
|
||
|
||
* Source `src/tc_impair.bpf.c` committed for reference; the build commits
|
||
the compiled object (`tc_impair.bpf.o`, freestanding: no CO-RE, no
|
||
BTF-typed ptr, plain `struct __sk_buff` + direct packet access).
|
||
No runtime clang/libbpf dependency.
|
||
* Maps:
|
||
* `CFG` — `BPF_MAP_TYPE_ARRAY`, 1 entry:
|
||
```
|
||
struct cfg {
|
||
__u32 nth; /* 0 = off, else drop every Nth packet */
|
||
__u64 quota_bytes; /* 0 = off */
|
||
__u32 quota_pct; /* random drop % after quota reached */
|
||
__u64 win_start_ns; /* recurring window start (monotonic) */
|
||
__u64 win_len_ns; /* 0 = off */
|
||
__u32 win_pct; /* random drop % inside the window */
|
||
__u32 flow_mask; /* bitmask: 1=src 2=dst 4=sport 8=dport 16=proto */
|
||
__u32 flow_target; /* required hash remainder, 0 = off */
|
||
};
|
||
```
|
||
* `CNT` — `BPF_MAP_TYPE_ARRAY`, 1 entry: `{__u64 packets; __u64 bytes; __u64 nth_state;}`
|
||
Counters are updated with `__sync_fetch_and_add` on the map/global value
|
||
so the every-Nth count is exact across CPUs (requires kernel ≥ 5.1 —
|
||
acceptable; probe at load, see E).
|
||
* Evaluation order is fixed: **nth → quota → window → flow**; the first
|
||
rule that decides a drop returns `TC_ACT_SHOT`, otherwise `TC_ACT_OK`.
|
||
Random percentages use the same 2³²-probability comparison as netem
|
||
(`prandom` seeded via map so tests are reproducible).
|
||
* Verifier-friendly: no loops, bounded accesses (check `data + eth+h` before
|
||
reading fields; when flow_mask touches ports, verify L4 header length).
|
||
|
||
### B.2 Command surface
|
||
|
||
```
|
||
tc nth_drop <if> <n | off>
|
||
tc quota_drop <if> <bytes> <pct> | off
|
||
tc window_drop <if> <start_ms> <len_ms> <pct> | off
|
||
tc flow_drop <if> <mask> <target> | off
|
||
```
|
||
|
||
* First enable on an interface: create clsact (EEXIST tolerated), load the
|
||
program, attach one filter at **prio 1** (see D). Subsequent commands only
|
||
update `CFG`/reset the relevant `CNT` fields.
|
||
* `off` for every mode → the filter is removed (and the prog fd closed).
|
||
Each `off` resets that mode's counters.
|
||
* `flow_drop` mask is the decimal bitmask from B.1 (server composes it);
|
||
hash = Jenkins/equal-fold over the selected header fields modulo
|
||
`flow_target` == remainder 0 → drop. Server documents: per-flow drop only
|
||
(a filter cannot delay; delay stays netem's job).
|
||
* Window semantics: `win_start_ns`/`win_len_ns` define a recurring window on
|
||
the monotonic clock — inside the window drop with `win_pct`; outside pass.
|
||
The server computes the first start; the program advances by `len` when
|
||
`now >= start + len` (single writer — the CFG update — so this is safe).
|
||
|
||
### B.3 Capability requirement
|
||
|
||
`BPF_PROG_LOAD(SCHED_CLS)` needs **CAP_BPF** (or CAP_SYS_ADMIN) on kernels
|
||
≥ 5.8. Installation must therefore set:
|
||
|
||
```
|
||
setcap cap_bpf,cap_net_admin,cap_net_raw=ep <ubridge>
|
||
```
|
||
|
||
On EPERM: reply `210-uBridge lacks CAP_BPF (setcap cap_bpf,cap_net_admin,cap_net_raw=ep) and the kernel requires it for stateful filters`
|
||
— the gns3-server side falls back to the relay datapath on this string (E).
|
||
|
||
---
|
||
|
||
## C. classic-BPF match drop (the GNS3 `bpf` filter)
|
||
|
||
No eBPF, no CAP_BPF: `cls_bpf` accepts **classic bytecode** via
|
||
`TCA_BPF_BYTECODE` (`sock_fprog`), and the kernel internally migrates it to
|
||
eBPF. uBridge already links libpcap.
|
||
|
||
```
|
||
tc bpf_drop add <if> <prio> "<expression>" # prio: 10..99, server-assigned
|
||
tc bpf_drop flush <if>
|
||
```
|
||
|
||
* `add`: `pcap_compile(DLT_EN10MB, snaplen 65535)` the expression; on
|
||
failure reply
|
||
`209-Cannot compile filter '<expr>': <pcap_geterr>` — gns3-server keys on
|
||
this exact prefix (same shape as the relay's error so one regex serves
|
||
both datapaths). Netlink: RTM_NEWTFILTER on clsact egress, protocol
|
||
ETH_P_ALL, `TCA_KIND "bpf"`, `TCA_BPF_BYTECODE`, action nested
|
||
`TCA_BPF_ACT` → `gact` with `action = TC_ACT_SHOT` (pcap match ≠ 0 →
|
||
filter matches → drop; non-match falls through, no action → pass).
|
||
* Multiple expressions (GNS3 sends one line per filter): server adds one
|
||
filter per line at successive prios — any match drops (OR).
|
||
* `flush`: delete every bpf_drop filter uBridge added on this interface
|
||
(track prios in-process), tolerate ENOENT per filter. Does NOT touch the
|
||
eBPF prio-1 filter or netem.
|
||
* Update-by-replace: server always `flush` + re-`add` (mirrors `netem set`).
|
||
|
||
---
|
||
|
||
## D. Lifecycle, composition & ordering (frozen contract)
|
||
|
||
* **clsact coexists with the netem root qdisc** — never replace the root.
|
||
* Execution order on egress: classifier filters by ascending prio —
|
||
eBPF impair filter at **prio 1**, `bpf_drop` at **prio 10..99** — then the
|
||
root netem qdisc. Dropped packets therefore never reach netem **nor the
|
||
AF_PACKET tap points** (capture/markers will not observe cls_bpf-dropped
|
||
frames; documented server-side).
|
||
* **`tc reset <if>` becomes the full restore** (behaviour change + fix):
|
||
1. remove all bpf_drop filters and the eBPF filter (clsact egress side we
|
||
own), 2. delete clsact, 3. delete the root qdisc. On an untouched
|
||
interface it must now reply **100-OK** (idempotent — currently returns
|
||
207-ENOENT; gns3-server tolerates both, but the new contract is OK).
|
||
* All state is kernel state tied to the interface: deleting the veth
|
||
(container stop) clears everything; no ubridge-side bookkeeping survives
|
||
a restart (a re-attached link rebuilds from the NIO, as today).
|
||
|
||
---
|
||
|
||
## E. Errors & capability probe
|
||
|
||
Error strings are contract (gns3-server matches them):
|
||
|
||
| Reply | Meaning |
|
||
|---|---|
|
||
| `100-netem set on <if>` / `100-qdisc reset on <if>` | success |
|
||
| `203-Bad number of parameters (%d with min/max=%d/%d)` | argc mismatch |
|
||
| `204-invalid <kw> value '<v>'` | value validation |
|
||
| `204-reorder requires delay` | keyword dependency |
|
||
| `207-Could not (set netem\|reset qdisc) on <if>: <strerror>` | netlink/kernel failure |
|
||
| `209-Cannot compile filter '<expr>': <err>` | bpf_drop pcap_compile failure |
|
||
| `210-uBridge lacks CAP_BPF ...` | stateful filters unavailable |
|
||
|
||
New introspection command (also serves the gns3-server degradation TODO —
|
||
old uBridge without these commands simply stays on the relay datapath):
|
||
|
||
```
|
||
tc capabilities
|
||
100-netem=delay,jitter,loss,dup,corrupt,rate,reorder,gemodel,dist,seed,limit;ebpf=1;cbpf=1
|
||
```
|
||
|
||
`ebpf=0` when the load probe fails at startup (kernel < 5.1 or no CAP_BPF);
|
||
`cbpf=0` when RTM_NEWTFILTER/cls_bpf is unavailable. gns3-server hides the
|
||
corresponding filter types per capability.
|
||
|
||
---
|
||
|
||
## F. Test requirements
|
||
|
||
* **Unit**: per keyword → expected netlink attrs (assert message bytes);
|
||
every 204 table row; `bpf_drop` compile-fail path; capabilities string.
|
||
* **Functional** (root, veth pair + netns + nsenter ping):
|
||
delay+reorder observable (mdev), gemodel loss within ±5% of target,
|
||
rate within ±10% (measured byte throughput), nth_drop **exact** pattern
|
||
(ICMP seq survives 1,2,…,N-1, Nth dropped), quota/window/flow with
|
||
crafted senders, `tc reset` idempotent on a clean interface, P5 surface
|
||
regression (delay/jitter/loss/dup/corrupt, REPLACE semantics).
|
||
* Determinism: netem `seed` + the CFG-seeded prandom make runs repeatable;
|
||
CI asserts identical drop patterns for identical seeds.
|
||
|
||
---
|
||
|
||
## gns3-server alignment (informational — not uBridge scope)
|
||
|
||
* P6a types map 1:1 onto new GNS3 filter entries (rate, reorder, gemodel…),
|
||
translated in `DockerVM._ubridge_apply_netem`.
|
||
* `frequency_drop` / `bpf` move out of `KERNEL_UNSUPPORTED_FILTERS` once
|
||
`tc capabilities` reports `ebpf=1` / `cbpf=1` (per-compute probe, cached);
|
||
without the capability the link stays relay-wired exactly as today.
|
||
* Every veth end owns one qdisc + its filters: per-direction impairment is
|
||
an architectural freebie to expose later (API `direction` field), aligned
|
||
with marker `dir` semantics.
|