[ [<1-h>]]] # mutually exclusive with plain loss [dup [correl ]] [corrupt ] [reorder [correl ] [gap ]] # requires delay [rate ] # e.g. 10mbit, 512kbit [limit ] ``` ### A.1 Parameter validation | Keyword | Range / format | Errors | |---|---|---| | `rate` | `` = integer + unit `bit\|kbit\|mbit\|gbit\|bps\|kbps\|mbps` (decimal), max 100gbit | 204 `invalid rate ''` | | `reorder` | pct 0–100, correl 0–100, gap 1–1000 | 204; 204 `reorder requires delay` if no `delay` present (mirror the kernel's EINVAL instead of passing it through — the server keys on the string) | | `loss gemodel` | p 0–100, r 0–100, h as `1-h` where h is 0–100 (default r=0, 1-h=0) | 204 | | `distribution` | one of the four names | 204 `unknown distribution ''` | | `seed` | u32 decimal | 204 | | `limit` | 1–1000000 packets (replaces the current fixed 1000 default) | 204 | | `correl` suffix on loss/dup | 0–100, only directly after its keyword | 204 | Max token count for the command table rises accordingly (recommend 32). `loss gemodel` semantics (mirrors the kernel, verified against a veth link with per-direction qdisc counters): `p` is the good→bad state transition probability (PGAP), `r` the bad→good transition (PGBP), and `1-h` the loss probability **inside the bad state** (the good state loses nothing). The steady-state mean loss is `p/(p+r) * (1-h)` per direction — e.g. `loss gemodel 10 30 100` drops 25 % of the offered packets in each direction, `loss gemodel 5 45 100` drops 10 %. ### A.2 Netlink encoding notes * Correlation: loss/dup/reorder use their existing nested structs with the `correlation` field set (`tc_netem_loss`? no — see `struct tc_netem_qopt` + `TCA_NETEM_CORR` carrying `tc_netem_corr` {delay, loss, dup}). Fill the loss correlation there; dup likewise. * gemodel: `TCA_NETEM_LOSS` nested attr, kind `TCA_NETEM_LOSS_GEMODEL` (`struct tc_netem_gemodel` {p, r, h}), probabilities in the same % × 2³² encoding as `qopt.loss`. When gemodel is set, `qopt.loss` stays 0. * reorder: `TCA_NETEM_REORDER` (`struct tc_netem_reorder` {probability, correlation}). * rate: prefer `TCA_NETEM_RATE64` (u64 bits/s) and also fill legacy `TCA_NETEM_RATE` (`tc_netem_rate`) with the truncated value for older kernels; `slot_overhead = 0`. * distribution: `TCA_NETEM_JITTER64` + `TCA_NETEM_DIST` must carry the distribution table (s16 deltas). **Embed the three standard tables** (normal / pareto / paretonormal — same values iproute2 ships as `*.dist`, generated by netem's `maketable`) as const arrays in uBridge; select by name. Without DIST the kernel silently uses uniform jitter. * seed: `TCA_NETEM_SEED` (u64 allowed) — makes random draws reproducible for tests. When omitted keep the current behaviour (kernel picks). ### A.3 Reply contract (unchanged shape) ``` 100-netem set on 204-invalid value '' 204-reorder requires delay 207-Could not set netem on : ``` --- ## B. eBPF stateful classifier *(frequency_drop + P6b modes)* **One** SCHED_CLS eBPF program, loaded once per interface, attached to the clsact egress hook. All modes are configured through one ARRAY map entry — no program reload on parameter changes. ### B.1 Program * Source `src/tc_impair.bpf.c` committed for reference; the build commits the compiled object (`tc_impair.bpf.o`, freestanding: no CO-RE, no BTF-typed ptr, plain `struct __sk_buff` + direct packet access). No runtime clang/libbpf dependency. * Maps: * `CFG` — `BPF_MAP_TYPE_ARRAY`, 1 entry: ``` struct cfg { __u32 nth; /* 0 = off, else drop every Nth packet */ __u64 quota_bytes; /* 0 = off */ __u32 quota_pct; /* random drop % after quota reached */ __u64 win_start_ns; /* recurring window start (monotonic) */ __u64 win_len_ns; /* 0 = off */ __u32 win_pct; /* random drop % inside the window */ __u32 flow_mask; /* bitmask: 1=src 2=dst 4=sport 8=dport 16=proto */ __u32 flow_target; /* required hash remainder, 0 = off */ }; ``` * `CNT` — `BPF_MAP_TYPE_ARRAY`, 1 entry: `{__u64 packets; __u64 bytes; __u64 nth_state;}` Counters are updated with `__sync_fetch_and_add` on the map/global value so the every-Nth count is exact across CPUs (requires kernel ≥ 5.1 — acceptable; probe at load, see E). * Evaluation order is fixed: **nth → quota → window → flow**; the first rule that decides a drop returns `TC_ACT_SHOT`, otherwise `TC_ACT_OK`. Random percentages use the same 2³²-probability comparison as netem (`prandom` seeded via map so tests are reproducible). * Verifier-friendly: no loops, bounded accesses (check `data + eth+h` before reading fields; when flow_mask touches ports, verify L4 header length). ### B.2 Command surface ``` tc nth_drop tc quota_drop | off tc window_drop [ []] | off tc flow_drop | off ``` * First enable on an interface: create clsact (EEXIST tolerated), load the program, attach one filter at **prio 1** (see D). Subsequent commands only update `CFG`/reset the relevant `CNT` fields. * `off` for every mode → the filter is removed (and the prog fd closed). Each `off` resets that mode's counters. * `flow_drop` mask is the decimal bitmask from B.1 (server composes it); hash = Jenkins/equal-fold over the selected header fields modulo `flow_target` == remainder 0 → drop. Server documents: per-flow drop only (a filter cannot delay; delay stays netem's job). * Window semantics: `win_start_ns`/`win_len_ns` define a window on the monotonic clock — inside the window drop with `win_pct`; outside pass. The program computes the first start (command time + `start_ms`) and then advances whole cycles, catching up in a bounded loop when it fell behind (single writer — the CFG update — so this is safe). Omitting `period` makes it a **single** window: after `start + outage` nothing drops. With `period ≥ outage` the outage recurs; `jitter` re-draws each cycle's outage and period uniformly in nominal ± jitter (0 = fixed schedule), on the integer-ms grid so the multiply-shift draw cannot overflow. ### B.3 Capability requirement `BPF_PROG_LOAD(SCHED_CLS)` needs **CAP_BPF** (or CAP_SYS_ADMIN) on kernels ≥ 5.8. Installation must therefore set: ``` setcap cap_bpf,cap_net_admin,cap_net_raw=ep ``` On EPERM: reply `210-uBridge lacks CAP_BPF (setcap cap_bpf,cap_net_admin,cap_net_raw=ep) and the kernel requires it for stateful filters` — the gns3-server side falls back to the relay datapath on this string (E). --- ## C. classic-BPF match drop (the GNS3 `bpf` filter) No eBPF, no CAP_BPF: `cls_bpf` accepts **classic bytecode** via `TCA_BPF_BYTECODE` (`sock_fprog`), and the kernel internally migrates it to eBPF. uBridge already links libpcap. ``` tc bpf_drop add "" # prio: 10..99, server-assigned tc bpf_drop flush ``` * `add`: `pcap_compile(DLT_EN10MB, snaplen 65535)` the expression; on failure reply `209-Cannot compile filter '': ` — gns3-server keys on this exact prefix (same shape as the relay's error so one regex serves both datapaths). Netlink: RTM_NEWTFILTER on clsact egress, protocol ETH_P_ALL, `TCA_KIND "bpf"`, `TCA_BPF_BYTECODE`, action nested `TCA_BPF_ACT` → `gact` with `action = TC_ACT_SHOT` (pcap match ≠ 0 → filter matches → drop; non-match falls through, no action → pass). * Multiple expressions (GNS3 sends one line per filter): server adds one filter per line at successive prios — any match drops (OR). * `flush`: delete every bpf_drop filter uBridge added on this interface (track prios in-process), tolerate ENOENT per filter. Does NOT touch the eBPF prio-1 filter or netem. * Update-by-replace: server always `flush` + re-`add` (mirrors `netem set`). --- ## D. Lifecycle, composition & ordering (frozen contract) * **clsact coexists with the netem root qdisc** — never replace the root. * Execution order on egress: classifier filters by ascending prio — eBPF impair filter at **prio 1**, `bpf_drop` at **prio 10..99** — then the root netem qdisc. Dropped packets therefore never reach netem **nor the AF_PACKET tap points** (capture/markers will not observe cls_bpf-dropped frames; documented server-side). * **`tc reset ` becomes the full restore** (behaviour change + fix): 1. remove all bpf_drop filters and the eBPF filter (clsact egress side we own), 2. delete clsact, 3. delete the root qdisc. On an untouched interface it must now reply **100-OK** (idempotent — currently returns 207-ENOENT; gns3-server tolerates both, but the new contract is OK). * All state is kernel state tied to the interface: deleting the veth (container stop) clears everything; no ubridge-side bookkeeping survives a restart (a re-attached link rebuilds from the NIO, as today). --- ## E. Errors & capability probe Error strings are contract (gns3-server matches them): | Reply | Meaning | |---|---| | `100-netem set on ` / `100-qdisc reset on ` | success | | `203-Bad number of parameters (%d with min/max=%d/%d)` | argc mismatch | | `204-invalid value ''` | value validation | | `204-reorder requires delay` | keyword dependency | | `207-Could not (set netem\|reset qdisc) on : ` | netlink/kernel failure | | `209-Cannot compile filter '': ` | bpf_drop pcap_compile failure | | `210-uBridge lacks CAP_BPF ...` | stateful filters unavailable | New introspection command (also serves the gns3-server degradation TODO — old uBridge without these commands simply stays on the relay datapath): ``` tc capabilities 100-netem=delay,jitter,loss,dup,corrupt,rate,reorder,gemodel,dist,seed,limit;ebpf=1;cbpf=1;ebpf_modes=nth,quota,window,flow ``` `ebpf=0` when the load probe fails at startup (kernel < 5.1 or no CAP_BPF); `cbpf=0` when RTM_NEWTFILTER/cls_bpf is unavailable. gns3-server hides the corresponding filter types per capability. `ebpf_modes` (added on uBridge `feature/tc-window`, commit 40c36ae) is a build fact, orthogonal to the runtime probes: the same binary flips `ebpf` between users and kernels while the mode set changes only with the binary. A mode is usable iff `ebpf=1` **and** its token is listed — gating `window_drop` on `ebpf=1` alone silently accepts the 4-arg form on a pre-correction build (`feature/tc-precision`: back-to-back windows, no gap). Builds that do not emit the field at all predate it and get legacy treatment; an incompatible mode revision renames its token (`window` → `window2`) rather than versioning in place. --- ## F. Test requirements * **Unit**: per keyword → expected netlink attrs (assert message bytes); every 204 table row; `bpf_drop` compile-fail path; capabilities string. * **Functional** (root, veth pair + netns + nsenter ping): delay+reorder observable (mdev), gemodel loss within ±5% of `p/(p+r) * (1-h)` measured **per direction** — a round-trip ping figure reads `1-(1-L)²` instead, and with `1-h=100` the ARP cache must be pre-warmed or the neighbour-resolution blackhole during a long burst inflates the observed loss, rate within ±10% (measured byte throughput), nth_drop **exact** pattern (ICMP seq survives 1,2,…,N-1, Nth dropped), quota/window/flow with crafted senders, `tc reset` idempotent on a clean interface, P5 surface regression (delay/jitter/loss/dup/corrupt, REPLACE semantics). * Determinism: netem `seed` + the CFG-seeded prandom make runs repeatable; CI asserts identical drop patterns for identical seeds. --- ## gns3-server alignment (informational — not uBridge scope) * **Done** (`feat/docker-kernel-netem-ext`): the P6a types map 1:1 onto new GNS3 filter entries (rate, reorder, gemodel, duplicate, seed, limit, plus the delay `distribution` and loss/dup `correlation` parameters), translated in `DockerVM._ubridge_apply_netem`; extension keywords are gated on the `tc capabilities` netem token list (old uBridge builds are never probed for the original surface). * **Done** (`feat/docker-kernel-bpf-drop`): `bpf` runs as cBPF match-drop. * **Done** (`feat/docker-kernel-ebpf-drops`): the eBPF classifier (part B) is consumed — `frequency_drop` → `tc nth_drop` (-1 → every 1st), the new kernel-only `quota` type → `tc quota_drop`, both keyed on `ebpf=1` from the per-process `tc capabilities` probe. `KERNEL_UNSUPPORTED_FILTERS` is gone entirely: kernel/relay eligibility is purely topological now. * **Done** (`feat/docker-kernel-window-drop`): `window_drop` is exposed with the corrected semantics (uBridge `feature/tc-window`) as the kernel-only `window_drop` filter type `[start, outage, pct, period?, jitter?]` → `tc window_drop`; the server mirrors the grammar's `period ≥ outage` rule and treats `start = 0` as active (an immediate outage). Not yet exposed: `flow_drop` (parameter-shape UX decision). * **Done** (capability reporting): the `ebpf_modes` contract is consumed end to end — the compute applies it per mode (window requires its token; pre-field builds keep their shipped modes), the compute `GET /capabilities` payload reports `ubridge_tc` through a standalone spawn-as-own-user probe cached by binary identity, and the controller's `available_filters` hides kernel-datapath filter types an endpoint compute reports it cannot run (computes without a report keep the full list; apply-time 409 stays the guard). * Every veth end owns one qdisc + its filters: per-direction impairment is an architectural freebie to expose later (API `direction` field), aligned with marker `dir` semantics.