gns3-server/docs/design/ubridge-kernel-impairment-spec.md

Frozen requirements spec for the uBridge project. Delivery status: all parts delivered and integrated. A (netem keyword extensions, uBridge feature/tc-netem-ext), C + D + E (cBPF match-drop, full-restore idempotent tc reset, tc capabilities — uBridge feature/tc-bpf-drop) and B (eBPF stateful classifier — uBridge feature/tc-precision 59e2b38, including the non-root setcap load fix) are consumed by gns3-server on the branch stack ending at feat/docker-kernel-window-drop (B: frequency_drop → tc nth_drop, the kernel-only quota type → tc quota_drop, and the kernel-only window_drop type → tc window_drop against uBridge feature/tc-window). The originally shipped netem surface (delay/jitter/loss/dup/corrupt) exists in uBridge 1.2.3+ and is verified against gns3-server branch feat/docker-kernel-filters.

Deviation found during integration (part A/D): the delivered tc netem set uses NLM_F_REPLACE, but the kernel's netem change merges optional attributes — rate, correlation, reorder, corrupt, gemodel, distribution and seed keep their previous value when the new message omits them, so "netem set atomically replaces" (D's reconcile assumption) does not hold for parameter removals. gns3-server works around it by sending tc reset before every netem set (the bpf_drop filters are re-added right after in the same flow, so clsact churn is invisible). If uBridge later emits explicit zero/absent clears for every optional attribute, the server-side reset-before-set can be dropped; the two behaviours are compatible either way.

Deviation found during B integration, resolved uBridge-side (2026-09-29): window_drop as first delivered advanced the window start by one length on expiry with no gap, making the windows back-to-back — after the first start_ms the "outside the window packets pass" behaviour described in B.2 was unreachable (inside ≡ always), so gns3-server did not expose it. uBridge now takes tc window_drop <if> <start_ms> <outage_ms> <pct> [<period_ms> [<jitter_ms>]]: 4-arg = single window (pass before and after), 5-arg = recurring outage with period ≥ outage, 6-arg adds a uniform ±jitter re-drawn per cycle (0 = exactly the fixed schedule) for randomized flap. The program advances whole cycles in a handful of straight-line steps — no loop at all, because the verifier's non-root path (the one production ubridge runs under: CAP_BPF, never uid 0) cannot keep a loop counter's bound and rejects the whole program with E2BIG past its jump-sequence budget, which took all four modes down on the gns3-server host while the same binary verified fine as root; uBridge's doc/tc.md has the trace. The steps are resumable by each further packet, so the B.2 semantics are now reachable; no caller ever depended on the old back-to-back form (it was never exposed). flow_drop is delivered and functional but not yet exposed as a GNS3 filter type (its mask/target parameter shape needs a UX decision). Additionally, the delivered B.1 program had to fold the flow-hash L4 port reads to constant offsets (IHL==20 only) because the verifier prohibits variable packet-pointer arithmetic for non-root — even with CAP_BPF (Spectre-mitigation gating by uid); documented in uBridge's doc/tc.md.

uBridge kernel impairment: tc netem extensions + eBPF classifiers

0. Background & scope

On the kernel datapath (Docker veth pairs enslaved into per-link Linux bridges) frames never pass through uBridge's userspace relay, so link impairment must live in the kernel: a netem qdisc on each veth host end (serving delay / jitter / loss / dup / corrupt — shipped) plus tc classifiers on the clsact egress hook for everything netem cannot express.

This spec freezes:

Part	Content	New capability needed
A	netem keyword extensions: rate, reorder, loss models, jitter distributions, correlation, seed, limit	none (netlink only)
B	eBPF stateful classifier: frequency_drop (nth-packet), byte quota, time window, flow select	CAP_BPF
C	classic-BPF match-drop: the GNS3 bpf filter	none (netlink only)
D	lifecycle, composition & ordering contract, tc reset semantics fix	—
E	error strings, tc capabilities probe	—
F	test requirements	—

Non-goals (explicitly out of scope for now): packet mangling (pedit, csum), mirroring (mirred), HTB hierarchies, RST/connection-kill injection, ingress-side (XDP) impairment.

All commands operate on egress of the veth host end (traffic entering the container), consistent with the existing netem placement.

---

A. tc netem set keyword extensions (P6a)

Extend the shipped grammar — existing keywords and their encoding are frozen and unchanged:

tc netem set <if>
   [delay <ms>] [jitter <ms>] [distribution uniform|normal|pareto|paretonormal] [seed <u32>]
   [loss <pct> [correl <pct>]]
   [loss gemodel <p> [<r> [<1-h>]]]              # mutually exclusive with plain loss
   [dup <pct> [correl <pct>]] [corrupt <pct>]
   [reorder <pct> [correl <pct>] [gap <n>]]      # requires delay
   [rate <bw>]                                    # e.g. 10mbit, 512kbit
   [limit <pkts>]

A.1 Parameter validation

Keyword	Range / format	Errors
rate	<bw> = integer + unit bit|kbit|mbit|gbit|bps|kbps|mbps (decimal), max 100gbit	204 invalid rate '<v>'
reorder	pct 0–100, correl 0–100, gap 1–1000	204; 204 reorder requires delay if no delay present (mirror the kernel's EINVAL instead of passing it through — the server keys on the string)
loss gemodel	p 0–100, r 0–100, h as 1-h where h is 0–100 (default r=0, 1-h=0)	204
distribution	one of the four names	204 unknown distribution '<v>'
seed	u32 decimal	204
limit	1–1000000 packets (replaces the current fixed 1000 default)	204
correl suffix on loss/dup	0–100, only directly after its keyword	204

Max token count for the command table rises accordingly (recommend 32).

loss gemodel semantics (mirrors the kernel, verified against a veth link with per-direction qdisc counters): p is the good→bad state transition probability (PGAP), r the bad→good transition (PGBP), and 1-h the loss probability inside the bad state (the good state loses nothing). The steady-state mean loss is p/(p+r) * (1-h) per direction — e.g. loss gemodel 10 30 100 drops 25 % of the offered packets in each direction, loss gemodel 5 45 100 drops 10 %.

A.2 Netlink encoding notes

• Correlation: loss/dup/reorder use their existing nested structs with the correlation field set (tc_netem_loss? no — see struct tc_netem_qopt
  • TCA_NETEM_CORR carrying tc_netem_corr {delay, loss, dup}). Fill the loss correlation there; dup likewise.
• gemodel: TCA_NETEM_LOSS nested attr, kind TCA_NETEM_LOSS_GEMODEL (struct tc_netem_gemodel {p, r, h}), probabilities in the same % × 2³² encoding as qopt.loss. When gemodel is set, qopt.loss stays 0.
• reorder: TCA_NETEM_REORDER (struct tc_netem_reorder {probability, correlation}).
• rate: prefer TCA_NETEM_RATE64 (u64 bits/s) and also fill legacy TCA_NETEM_RATE (tc_netem_rate) with the truncated value for older kernels; slot_overhead = 0.
• distribution: TCA_NETEM_JITTER64 + TCA_NETEM_DIST must carry the distribution table (s16 deltas). Embed the three standard tables (normal / pareto / paretonormal — same values iproute2 ships as *.dist, generated by netem's maketable) as const arrays in uBridge; select by name. Without DIST the kernel silently uses uniform jitter.
• seed: TCA_NETEM_SEED (u64 allowed) — makes random draws reproducible for tests. When omitted keep the current behaviour (kernel picks).

A.3 Reply contract (unchanged shape)

100-netem set on <if>
204-invalid <kw> value '<v>'
204-reorder requires delay
207-Could not set netem on <if>: <strerror>

---

B. eBPF stateful classifier (frequency_drop + P6b modes)

One SCHED_CLS eBPF program, loaded once per interface, attached to the clsact egress hook. All modes are configured through one ARRAY map entry — no program reload on parameter changes.

B.1 Program

• Source src/tc_impair.bpf.c committed for reference; the build commits the compiled object (tc_impair.bpf.o, freestanding: no CO-RE, no BTF-typed ptr, plain struct __sk_buff + direct packet access). No runtime clang/libbpf dependency.
• Maps:
  • CFG — BPF_MAP_TYPE_ARRAY, 1 entry:

    struct cfg {
        __u32 nth;          /* 0 = off, else drop every Nth packet      */
        __u64 quota_bytes;  /* 0 = off                                  */
        __u32 quota_pct;    /* random drop % after quota reached        */
        __u64 win_start_ns; /* recurring window start (monotonic)       */
        __u64 win_len_ns;   /* 0 = off                                  */
        __u32 win_pct;      /* random drop % inside the window          */
        __u32 flow_mask;    /* bitmask: 1=src 2=dst 4=sport 8=dport 16=proto */
        __u32 flow_target;  /* required hash remainder, 0 = off         */
    };
    

  • CNT — BPF_MAP_TYPE_ARRAY, 1 entry: {__u64 packets; __u64 bytes; __u64 nth_state;} Counters are updated with __sync_fetch_and_add on the map/global value so the every-Nth count is exact across CPUs (requires kernel ≥ 5.1 — acceptable; probe at load, see E).
• Evaluation order is fixed: nth → quota → window → flow; the first rule that decides a drop returns TC_ACT_SHOT, otherwise TC_ACT_OK. Random percentages use the same 2³²-probability comparison as netem (prandom seeded via map so tests are reproducible).
• Verifier-friendly: no loops, bounded accesses (check data + eth+h before reading fields; when flow_mask touches ports, verify L4 header length).

B.2 Command surface

tc nth_drop   <if> <n | off>
tc quota_drop <if> <bytes> <pct> | off
tc window_drop <if> <start_ms> <outage_ms> <pct> [<period_ms> [<jitter_ms>]] | off
tc flow_drop  <if> <mask> <target> | off

• First enable on an interface: create clsact (EEXIST tolerated), load the program, attach one filter at prio 1 (see D). Subsequent commands only update CFG/reset the relevant CNT fields.
• off for every mode → the filter is removed (and the prog fd closed). Each off resets that mode's counters.
• flow_drop mask is the decimal bitmask from B.1 (server composes it); hash = Jenkins/equal-fold over the selected header fields modulo flow_target == remainder 0 → drop. Server documents: per-flow drop only (a filter cannot delay; delay stays netem's job).
• Window semantics: win_start_ns/win_len_ns define a window on the monotonic clock — inside the window drop with win_pct; outside pass. The program computes the first start (command time + start_ms) and then advances whole cycles, catching up in a bounded loop when it fell behind (single writer — the CFG update — so this is safe). Omitting period makes it a single window: after start + outage nothing drops. With period ≥ outage the outage recurs; jitter re-draws each cycle's outage and period uniformly in nominal ± jitter (0 = fixed schedule), on the integer-ms grid so the multiply-shift draw cannot overflow.

B.3 Capability requirement

BPF_PROG_LOAD(SCHED_CLS) needs CAP_BPF (or CAP_SYS_ADMIN) on kernels ≥ 5.8. Installation must therefore set:

setcap cap_bpf,cap_net_admin,cap_net_raw=ep <ubridge>

On EPERM: reply 210-uBridge lacks CAP_BPF (setcap cap_bpf,cap_net_admin,cap_net_raw=ep) and the kernel requires it for stateful filters — the gns3-server side falls back to the relay datapath on this string (E).

---

C. classic-BPF match drop (the GNS3 bpf filter)

No eBPF, no CAP_BPF: cls_bpf accepts classic bytecode via TCA_BPF_BYTECODE (sock_fprog), and the kernel internally migrates it to eBPF. uBridge already links libpcap.

tc bpf_drop add <if> <prio> "<expression>"    # prio: 10..99, server-assigned
tc bpf_drop flush <if>

• add: pcap_compile(DLT_EN10MB, snaplen 65535) the expression; on failure reply 209-Cannot compile filter '<expr>': <pcap_geterr> — gns3-server keys on this exact prefix (same shape as the relay's error so one regex serves both datapaths). Netlink: RTM_NEWTFILTER on clsact egress, protocol ETH_P_ALL, TCA_KIND "bpf", TCA_BPF_BYTECODE, action nested TCA_BPF_ACT → gact with action = TC_ACT_SHOT (pcap match ≠ 0 → filter matches → drop; non-match falls through, no action → pass).
• Multiple expressions (GNS3 sends one line per filter): server adds one filter per line at successive prios — any match drops (OR).
• flush: delete every bpf_drop filter uBridge added on this interface (track prios in-process), tolerate ENOENT per filter. Does NOT touch the eBPF prio-1 filter or netem.
• Update-by-replace: server always flush + re-add (mirrors netem set).

---

D. Lifecycle, composition & ordering (frozen contract)

• clsact coexists with the netem root qdisc — never replace the root.
• Execution order on egress: classifier filters by ascending prio — eBPF impair filter at prio 1, bpf_drop at prio 10..99 — then the root netem qdisc. Dropped packets therefore never reach netem nor the AF_PACKET tap points (capture/markers will not observe cls_bpf-dropped frames; documented server-side).
• tc reset <if> becomes the full restore (behaviour change + fix):
  1. remove all bpf_drop filters and the eBPF filter (clsact egress side we own), 2. delete clsact, 3. delete the root qdisc. On an untouched interface it must now reply 100-OK (idempotent — currently returns 207-ENOENT; gns3-server tolerates both, but the new contract is OK).
• All state is kernel state tied to the interface: deleting the veth (container stop) clears everything; no ubridge-side bookkeeping survives a restart (a re-attached link rebuilds from the NIO, as today).

---

E. Errors & capability probe

Error strings are contract (gns3-server matches them):

Reply	Meaning
100-netem set on <if> / 100-qdisc reset on <if>	success
203-Bad number of parameters (%d with min/max=%d/%d)	argc mismatch
204-invalid <kw> value '<v>'	value validation
204-reorder requires delay	keyword dependency
207-Could not (set netem|reset qdisc) on <if>: <strerror>	netlink/kernel failure
209-Cannot compile filter '<expr>': <err>	bpf_drop pcap_compile failure
210-uBridge lacks CAP_BPF ...	stateful filters unavailable

New introspection command (also serves the gns3-server degradation TODO — old uBridge without these commands simply stays on the relay datapath):

tc capabilities
100-netem=delay,jitter,loss,dup,corrupt,rate,reorder,gemodel,dist,seed,limit;ebpf=1;cbpf=1;ebpf_modes=nth,quota,window,flow

ebpf=0 when the load probe fails at startup (kernel < 5.1 or no CAP_BPF); cbpf=0 when RTM_NEWTFILTER/cls_bpf is unavailable. gns3-server hides the corresponding filter types per capability.

ebpf_modes (added on uBridge feature/tc-window, commit 40c36ae) is a build fact, orthogonal to the runtime probes: the same binary flips ebpf between users and kernels while the mode set changes only with the binary. A mode is usable iff ebpf=1 and its token is listed — gating window_drop on ebpf=1 alone silently accepts the 4-arg form on a pre-correction build (feature/tc-precision: back-to-back windows, no gap). Builds that do not emit the field at all predate it and get legacy treatment; an incompatible mode revision renames its token (window → window2) rather than versioning in place.

---

F. Test requirements

• Unit: per keyword → expected netlink attrs (assert message bytes); every 204 table row; bpf_drop compile-fail path; capabilities string.
• Functional (root, veth pair + netns + nsenter ping): delay+reorder observable (mdev), gemodel loss within ±5% of p/(p+r) * (1-h) measured per direction — a round-trip ping figure reads 1-(1-L)² instead, and with 1-h=100 the ARP cache must be pre-warmed or the neighbour-resolution blackhole during a long burst inflates the observed loss, rate within ±10% (measured byte throughput), nth_drop exact pattern (ICMP seq survives 1,2,…,N-1, Nth dropped), quota/window/flow with crafted senders, tc reset idempotent on a clean interface, P5 surface regression (delay/jitter/loss/dup/corrupt, REPLACE semantics).
• Determinism: netem seed + the CFG-seeded prandom make runs repeatable; CI asserts identical drop patterns for identical seeds.

---

gns3-server alignment (informational — not uBridge scope)

• Done (feat/docker-kernel-netem-ext): the P6a types map 1:1 onto new GNS3 filter entries (rate, reorder, gemodel, duplicate, seed, limit, plus the delay distribution and loss/dup correlation parameters), translated in DockerVM._ubridge_apply_netem; extension keywords are gated on the tc capabilities netem token list (old uBridge builds are never probed for the original surface).
• Done (feat/docker-kernel-bpf-drop): bpf runs as cBPF match-drop.
• Done (feat/docker-kernel-ebpf-drops): the eBPF classifier (part B) is consumed — frequency_drop → tc nth_drop (-1 → every 1st), the new kernel-only quota type → tc quota_drop, both keyed on ebpf=1 from the per-process tc capabilities probe. KERNEL_UNSUPPORTED_FILTERS is gone entirely: kernel/relay eligibility is purely topological now.
• Done (feat/docker-kernel-window-drop): window_drop is exposed with the corrected semantics (uBridge feature/tc-window) as the kernel-only window_drop filter type [start, outage, pct, period?, jitter?] → tc window_drop; the server mirrors the grammar's period ≥ outage rule and treats start = 0 as active (an immediate outage). Not yet exposed: flow_drop (parameter-shape UX decision).
• Done (capability reporting): the ebpf_modes contract is consumed end to end — the compute applies it per mode (window requires its token; pre-field builds keep their shipped modes), the compute GET /capabilities payload reports ubridge_tc through a standalone spawn-as-own-user probe cached by binary identity, and the controller's available_filters hides kernel-datapath filter types an endpoint compute reports it cannot run (computes without a report keep the full list; apply-time 409 stays the guard).
• Every veth end owns one qdisc + its filters: per-direction impairment is an architectural freebie to expose later (API direction field), aligned with marker dir semantics.