mirror of
https://github.com/GNS3/gns3-server.git
synced 2026-10-13 05:55:35 +03:00
feat(docker): expose the eBPF time-window filter on kernel links
uBridge's window mode gained the semantics the B.2 spec asked for — `tc window_drop <if> <start> <outage> <pct> [<period> [<jitter>]]`: a single outage that traffic passes before *and* after, recurring flaps with period >= outage, and a per-cycle jitter — so the kernel-only `window_drop` filter type is now exposed: - validation mirrors the tc grammar (3-5 params, ranges, period >= outage, jitter cap) and adds window_drop to KERNEL_ONLY_FILTERS; filter_inactive_filters keeps start=0 — unlike every other filter a zero first parameter means "the outage starts now", not "disabled". - `_ubridge_apply_ebpf_drops` sends `tc window_drop` (positional pass-through) and turns the mode off explicitly when absent, under the same ebpf=1 gate. - FILTERS entry (Start/Outage/Chance/Period/Jitter) + available_filters. The schedule is relative to the apply: reconcile re-sets the classifier on every apply, so any filter change, node restart or link reset restarts the window. Verified on a two-container kernel link, 15/15: a single outage [1000, 1500, 100] arrives per-seq as [0..4, 13..19] (passing before the window, dropped inside, passing after); recurring 1000/2000 ms = 52 % round trip (duty cycle); period == outage = permanent outage; period < outage = 409 before uBridge; jittered flap keeps the ~50 % duty cycle; 30 % chance on a permanent window = 53 % round trip against the per-direction theory 1-0.7^2 = 51 %; coexist with netem; node restart restores; available_filters kernel vs relay (409 + hidden); teardown clean. P6b re-run as a regression on the same uBridge build: 13/13. Worth recording for the next rebuild: the first delivered window program loaded as root but reported ebpf=0 through gns3-server — the verifier's non-root path loses the loop counter's bound, treats the catch-up loop as unbounded and rejects the *whole* program (E2BIG, 8193 pending states), which also took frequency_drop and quota down, with ebpf=0 as the only symptom. Smaller trip caps did not help; the loop had to go (uBridge 572ca0c), on top of a Makefile header-dependency fix (53e34bb) that had been shipping a stale program object.
This commit is contained in:
YueGuobin
parent
0c4ec8c38c
commit
316e7002c1
@ -9,11 +9,12 @@ See LICENSE file for licensing information.
|
||||
> idempotent `tc reset`, `tc capabilities` — uBridge `feature/tc-bpf-drop`)
|
||||
> and **B (eBPF stateful classifier — uBridge `feature/tc-precision`
|
||||
> 59e2b38, including the non-root setcap load fix)** are consumed by
|
||||
> gns3-server on the branch stack ending at `feat/docker-kernel-ebpf-drops`
|
||||
> (B: `frequency_drop` → `tc nth_drop`, the new kernel-only `quota` type →
|
||||
> `tc quota_drop`). The originally shipped netem surface (delay/jitter/
|
||||
> loss/dup/corrupt) exists in uBridge 1.2.3+ and is verified against
|
||||
> gns3-server branch `feat/docker-kernel-filters`.
|
||||
> gns3-server on the branch stack ending at `feat/docker-kernel-window-drop`
|
||||
> (B: `frequency_drop` → `tc nth_drop`, the kernel-only `quota` type →
|
||||
> `tc quota_drop`, and the kernel-only `window_drop` type → `tc window_drop`
|
||||
> against uBridge `feature/tc-window`). The originally shipped netem surface
|
||||
> (delay/jitter/loss/dup/corrupt) exists in uBridge 1.2.3+ and is verified
|
||||
> against gns3-server branch `feat/docker-kernel-filters`.
|
||||
>
|
||||
> **Deviation found during integration (part A/D):** the delivered
|
||||
> `tc netem set` uses NLM_F_REPLACE, but the *kernel's* netem change merges
|
||||
@ -27,19 +28,32 @@ See LICENSE file for licensing information.
|
||||
> optional attribute, the server-side reset-before-set can be dropped; the
|
||||
> two behaviours are compatible either way.
|
||||
>
|
||||
> **Deviation found during B integration:** `window_drop`'s implementation
|
||||
> advances the window start by one length on expiry with no gap, making the
|
||||
> windows back-to-back — after the first `start_ms` the "outside the window
|
||||
> packets pass" behaviour described in B.2 is unreachable (inside ≡ always).
|
||||
> gns3-server therefore does **not** expose window_drop yet; it needs either
|
||||
> a second length field (outage vs period) or a next-start semantics fix on
|
||||
> the uBridge side. `flow_drop` is delivered and functional but not yet
|
||||
> exposed as a GNS3 filter type (its mask/target parameter shape needs a
|
||||
> UX decision). Additionally, the delivered B.1 program had to fold the
|
||||
> flow-hash L4 port reads to constant offsets (IHL==20 only) because the
|
||||
> verifier prohibits variable packet-pointer arithmetic for non-root —
|
||||
> even with CAP_BPF (Spectre-mitigation gating by uid); documented in
|
||||
> uBridge's doc/tc.md.
|
||||
> **Deviation found during B integration, resolved uBridge-side
|
||||
> (2026-09-29):** `window_drop` as first delivered advanced the window
|
||||
> start by one length on expiry with no gap, making the windows
|
||||
> back-to-back — after the first `start_ms` the "outside the window
|
||||
> packets pass" behaviour described in B.2 was unreachable (inside ≡
|
||||
> always), so gns3-server did not expose it. uBridge now takes
|
||||
> `tc window_drop <if> <start_ms> <outage_ms> <pct> [<period_ms>
|
||||
> [<jitter_ms>]]`: 4-arg = single window (pass before *and after*),
|
||||
> 5-arg = recurring outage with `period ≥ outage`, 6-arg adds a uniform
|
||||
> ±jitter re-drawn per cycle (0 = exactly the fixed schedule) for
|
||||
> randomized flap. The program advances whole cycles in a handful of
|
||||
> straight-line steps — **no loop at all**, because the verifier's
|
||||
> non-root path (the one production ubridge runs under: `CAP_BPF`, never
|
||||
> uid 0) cannot keep a loop counter's bound and rejects the *whole*
|
||||
> program with `E2BIG` past its jump-sequence budget, which took all four
|
||||
> modes down on the gns3-server host while the same binary verified fine
|
||||
> as root; uBridge's `doc/tc.md` has the trace. The steps are resumable by
|
||||
> each further packet, so the
|
||||
> B.2 semantics are now reachable; no caller ever depended on the old
|
||||
> back-to-back form (it was never exposed). `flow_drop` is delivered and
|
||||
> functional but not yet exposed as a GNS3 filter type (its mask/target
|
||||
> parameter shape needs a UX decision). Additionally, the delivered B.1
|
||||
> program had to fold the flow-hash L4 port reads to constant offsets
|
||||
> (IHL==20 only) because the verifier prohibits variable packet-pointer
|
||||
> arithmetic for non-root — even with CAP_BPF (Spectre-mitigation gating
|
||||
> by uid); documented in uBridge's doc/tc.md.
|
||||
|
||||
# uBridge kernel impairment: tc netem extensions + eBPF classifiers
|
||||
|
||||
@ -176,7 +190,7 @@ no program reload on parameter changes.
|
||||
```
|
||||
tc nth_drop <if> <n | off>
|
||||
tc quota_drop <if> <bytes> <pct> | off
|
||||
tc window_drop <if> <start_ms> <len_ms> <pct> | off
|
||||
tc window_drop <if> <start_ms> <outage_ms> <pct> [<period_ms> [<jitter_ms>]] | off
|
||||
tc flow_drop <if> <mask> <target> | off
|
||||
```
|
||||
|
||||
@ -189,10 +203,15 @@ tc flow_drop <if> <mask> <target> | off
|
||||
hash = Jenkins/equal-fold over the selected header fields modulo
|
||||
`flow_target` == remainder 0 → drop. Server documents: per-flow drop only
|
||||
(a filter cannot delay; delay stays netem's job).
|
||||
* Window semantics: `win_start_ns`/`win_len_ns` define a recurring window on
|
||||
the monotonic clock — inside the window drop with `win_pct`; outside pass.
|
||||
The server computes the first start; the program advances by `len` when
|
||||
`now >= start + len` (single writer — the CFG update — so this is safe).
|
||||
* Window semantics: `win_start_ns`/`win_len_ns` define a window on the
|
||||
monotonic clock — inside the window drop with `win_pct`; outside pass.
|
||||
The program computes the first start (command time + `start_ms`) and then
|
||||
advances whole cycles, catching up in a bounded loop when it fell behind
|
||||
(single writer — the CFG update — so this is safe). Omitting `period`
|
||||
makes it a **single** window: after `start + outage` nothing drops. With
|
||||
`period ≥ outage` the outage recurs; `jitter` re-draws each cycle's
|
||||
outage and period uniformly in nominal ± jitter (0 = fixed schedule), on
|
||||
the integer-ms grid so the multiply-shift draw cannot overflow.
|
||||
|
||||
### B.3 Capability requirement
|
||||
|
||||
@ -312,8 +331,13 @@ corresponding filter types per capability.
|
||||
kernel-only `quota` type → `tc quota_drop`, both keyed on `ebpf=1` from
|
||||
the per-process `tc capabilities` probe. `KERNEL_UNSUPPORTED_FILTERS` is
|
||||
gone entirely: kernel/relay eligibility is purely topological now.
|
||||
Not yet exposed: `window_drop` (implementation deviation above),
|
||||
`flow_drop` (parameter-shape UX decision).
|
||||
* **Done** (`feat/docker-kernel-window-drop`): `window_drop` is exposed
|
||||
with the corrected semantics (uBridge `feature/tc-window`) as the
|
||||
kernel-only `window_drop` filter type
|
||||
`[start, outage, pct, period?, jitter?]` → `tc window_drop`; the server
|
||||
mirrors the grammar's `period ≥ outage` rule and treats `start = 0` as
|
||||
active (an immediate outage).
|
||||
Not yet exposed: `flow_drop` (parameter-shape UX decision).
|
||||
* Every veth end owns one qdisc + its filters: per-direction impairment is
|
||||
an architectural freebie to expose later (API `direction` field), aligned
|
||||
with marker `dir` semantics.
|
||||
|
||||
@ -24,14 +24,14 @@ Frames never leave the kernel.
|
||||
| Link | uBridge bridge, TAP fd ↔ UDP, userspace copy | per-link kernel bridge `gns3{link_id[:11]}`, zero-copy |
|
||||
| Relay latency | ~0.3 ms RTT | ~0.05 ms RTT (measured) |
|
||||
| Link attach at runtime | yes (relay attaches the pre-existing TAP) | yes (brctl addif / add_nio_ethernet — interface untouched) |
|
||||
| Packet filters | uBridge userspace filters | tc netem on the veth host end (delay / packet_loss / corrupt / duplicate + the netem extensions rate, reorder, gemodel, seed, limit, jitter distributions, loss correlation); bpf as cls_bpf match-drop classifiers; frequency_drop and quota as the eBPF stateful classifier — every filter type has a kernel equivalent |
|
||||
| Packet filters | uBridge userspace filters | tc netem on the veth host end (delay / packet_loss / corrupt / duplicate + the netem extensions rate, reorder, gemodel, seed, limit, jitter distributions, loss correlation); bpf as cls_bpf match-drop classifiers; frequency_drop, quota and window_drop as the eBPF stateful classifier — every filter type has a kernel equivalent |
|
||||
| Capture / markers | relay bridge | uBridge AF_PACKET modules on the veth host end |
|
||||
|
||||
The work landed in eight stages on branch stack `feat/docker-kernel-datapath` →
|
||||
The work landed in nine stages on branch stack `feat/docker-kernel-datapath` →
|
||||
`feat/docker-kernel-capture` → `feat/docker-kernel-markers` →
|
||||
`feat/docker-veth-everywhere` → `feat/docker-kernel-filters` →
|
||||
`feat/docker-kernel-bpf-drop` → `feat/docker-kernel-netem-ext` →
|
||||
`feat/docker-kernel-ebpf-drops`:
|
||||
`feat/docker-kernel-ebpf-drops` → `feat/docker-kernel-window-drop`:
|
||||
|
||||
1. **Kernel datapath** — NIOBridge NIO type, per-link bridges, carrier-based
|
||||
suspend, crash-safe reconciliation.
|
||||
@ -66,6 +66,14 @@ The work landed in eight stages on branch stack `feat/docker-kernel-datapath`
|
||||
non-root server process, the production credential shape). With this,
|
||||
no filter type forces the relay anymore: kernel/relay is a purely
|
||||
topological choice.
|
||||
9. **window_drop flaps** — the eBPF classifier's time-window mode
|
||||
(`tc window_drop`, uBridge `feature/tc-window`): a single outage that
|
||||
passes traffic before and after, a recurring flap with `period ≥ outage`,
|
||||
and an optional per-cycle jitter that randomizes each beat's outage and
|
||||
period. The originally delivered back-to-back window implementation
|
||||
(which made "outside the window packets pass" unreachable) was fixed
|
||||
uBridge-side; this stage exposes the corrected semantics as the
|
||||
kernel-only `window_drop` filter type.
|
||||
|
||||
## Adapter interface types
|
||||
|
||||
@ -201,6 +209,7 @@ ends (uBridge `tc netem set`, raw netlink — no `tc` binary needed):
|
||||
| `bpf` (one line per expression, OR) | `tc bpf_drop add <if> <prio> "<expr>"` — cls_bpf on clsact egress + gact drop | needs uBridge `cbpf=1` (`tc capabilities`, probed once per uBridge process); a line that fails to compile on the compute is skipped with a warning, mirroring the relay |
|
||||
| `frequency_drop [N]` | `tc nth_drop <if> N` — the eBPF stateful classifier's exact every-Nth mode (clsact egress prio 1) | needs uBridge `ebpf=1`; -1 = drop everything → every 1st; per-direction counters (see below) |
|
||||
| `quota [bytes, %]` | `tc quota_drop <if> <bytes> <pct>` — after the byte quota is consumed, each further packet drops with the given chance (100 = hard cutoff) | kernel-only type; needs `ebpf=1`; per-direction byte counters |
|
||||
| `window_drop [start, outage, %, period?, jitter?]` | `tc window_drop <if> <start> <outage> <pct> [<period> [<jitter>]]` — packets drop with the given chance inside `[start, start+outage)` | kernel-only type; needs `ebpf=1`; see the window semantics below |
|
||||
|
||||
Semantics:
|
||||
|
||||
@ -241,7 +250,25 @@ Semantics:
|
||||
* **Classifier order** on clsact egress: the eBPF stateful filter runs at
|
||||
prio 1, `bpf_drop` expressions at 10–99 — a packet dropped by the
|
||||
stateful modes never reaches the expression drops, and dropped packets
|
||||
never reach netem.
|
||||
never reach netem. Within the stateful program the modes evaluate in the
|
||||
fixed order nth → quota → window.
|
||||
* **window_drop semantics.** `start` is relative to the moment the filter
|
||||
is applied — and because reconcile re-sets the classifier on every apply
|
||||
(any filter change, node restart, link reset reloads the program after
|
||||
the netem reset), every such event **restarts the schedule**. 3
|
||||
parameters = one single outage (`[start, start+outage)`), traffic passes
|
||||
before and after — the classic "link dies at T, comes back at T+outage".
|
||||
4 parameters = recurring flap: `period ≥ outage` (validated), so the
|
||||
outage occupies `outage/period` of each cycle. 5 parameters = randomized
|
||||
flap: each cycle's outage and period are re-drawn uniformly in nominal
|
||||
± `jitter` (integer-ms grid; jitter 0 draws nothing and equals the fixed
|
||||
schedule). Inside a window each packet drops with the given chance (100
|
||||
= full outage, lower = degraded service during the window). Both
|
||||
endpoints run their own schedule, started ~simultaneously by the
|
||||
controller — a round trip survives only when **both** directions are
|
||||
outside their windows. Note `start = 0` (outage begins immediately) is a
|
||||
valid, active configuration — unlike other filters, a zero first
|
||||
parameter does not mean "disabled".
|
||||
* **Validation** mirrors the tc grammar: `reorder` requires `delay`,
|
||||
`gemodel` and `packet_loss` are mutually exclusive (both map to the netem
|
||||
loss keyword), `distribution` requires jitter > 0, rate must be integer +
|
||||
@ -278,6 +305,7 @@ tc netem set <if> [delay <ms>] [jitter <ms>] [loss <%> [correl <%>] | loss gemod
|
||||
tc bpf_drop add <if> <prio 10-99> "<expr>" / tc bpf_drop flush <if>
|
||||
tc nth_drop <if> <n | off> # eBPF exact every-Nth (frequency_drop)
|
||||
tc quota_drop <if> <bytes> <pct> | off # eBPF byte cap (quota)
|
||||
tc window_drop <if> <start_ms> <outage_ms> <pct> [<period_ms> [<jitter_ms>]] | off # eBPF time window (window_drop)
|
||||
tc reset <if> # full restore: eBPF filter -> bpf_drops -> clsact -> root qdisc, idempotent
|
||||
tc capabilities # "netem=<kw,...>;ebpf=0|1;cbpf=0|1", probed once per process
|
||||
bridge add_nio_ethernet / add_nio_udp / start / stop / start_capture / stop_capture
|
||||
@ -358,3 +386,38 @@ node restart with a relay link skipped `bridge create` and failed. Unit tests:
|
||||
`tests/utils/test_packet_filter_validation.py` (P6a class),
|
||||
`tests/compute/docker/test_docker_kernel_datapath.py`,
|
||||
`tests/controller/test_kernel_datapath_link.py`.
|
||||
|
||||
window_drop (same setup, 15/15 checks, against uBridge `feature/tc-window`):
|
||||
a single outage `[1000, 1500, 100]` delivers **per-seq evidence** of the B.2
|
||||
semantics — one ping per 200 ms arrives as `[0,1,2,3,4, 13…19]`: passing
|
||||
before the window, dropped inside it, passing again after; a recurring flap
|
||||
`[300, 1000, 100, 2000]` measures 52 % round-trip loss (the 1000/2000 duty
|
||||
cycle); `period == outage` is a permanent outage (100 %); `period < outage`
|
||||
is a 409 before uBridge sees it; a jittered flap
|
||||
`[0, 1000, 100, 2000, 400]` keeps the ~50 % duty cycle (62 % sampled over
|
||||
4 cycles — the per-cycle re-draw widens the variance by design); a 30 %
|
||||
chance on a permanent window measures 53 % round trip against the
|
||||
per-direction theory 1 − 0.7² = 51 % (the same arithmetic as
|
||||
`packet_loss 30`); `start = 0` is stored as an *active* filter; the window
|
||||
coexists with netem (`delay 50` + a permanent window = 100 % inside the
|
||||
outage); clearing removes the classifier; a node restart restores the
|
||||
schedule on the fresh veth; `available_filters` shows window_drop on kernel
|
||||
links and a relay link both hides it and 409s it; teardown leaves no filter
|
||||
or qdisc behind. The P6b e2e was re-run as a regression on the same uBridge
|
||||
build (13/13 — nth/quota unaffected by the de-looping below).
|
||||
|
||||
The uBridge-side window work took two fixes that are worth recording, both
|
||||
found by *this* integration rather than by unit tests: the delivered
|
||||
catch-up loop tripped the verifier's 8192 **jump-sequence** budget — on the
|
||||
non-root path the verifier drops the loop counter's bound
|
||||
(`R4=scalar(smax=umax32=0xfffff086)`), treats the loop as unbounded and
|
||||
unrolls it until the pending-state limit, so the program was rejected
|
||||
wholesale (`E2BIG`, `The sequence of 8193 jumps is too complex`) and
|
||||
`tc capabilities` reported `ebpf=0` — taking `frequency_drop` and `quota`
|
||||
down with it, since the classifier loads as one program; lowering the trip
|
||||
cap did not help (128 trips failed identically), removing the loop did
|
||||
(572ca0c: 16-step unroll + outer guard, zero back edges). A latent Makefile
|
||||
header-dependency gap (53e34bb) had been shipping the *old* truncated
|
||||
program object — the kernel answered `jump out of range from insn 9 to
|
||||
421` with `processed 0 insns`. Both matter to anyone rebuilding uBridge:
|
||||
`ebpf=1` from a direct probe is the only trustworthy signal.
|
||||
|
||||
@ -1950,7 +1950,8 @@ class DockerVM(BaseNode):
|
||||
# Impairment filters become one tc netem qdisc on the veth host
|
||||
# end (restored here on node restart, like the capture above);
|
||||
# bpf expressions become cls_bpf match-drop classifiers and
|
||||
# frequency_drop/quota become the eBPF stateful classifier.
|
||||
# frequency_drop/quota/window_drop become the eBPF stateful
|
||||
# classifier.
|
||||
await self._ubridge_apply_netem(host_ifc, nio.filters)
|
||||
await self._ubridge_apply_bpf_drops(host_ifc, nio.filters)
|
||||
await self._ubridge_apply_ebpf_drops(host_ifc, nio.filters)
|
||||
@ -2379,20 +2380,28 @@ class DockerVM(BaseNode):
|
||||
Translate the NIO's stateful filters into uBridge's eBPF impairment
|
||||
classifier on the veth host end's clsact egress (prio 1, below
|
||||
bpf_drop's 10-99): ``frequency_drop`` becomes the exact every-Nth
|
||||
mode (``tc nth_drop``; -1 = drop everything maps to every 1st), and
|
||||
the kernel-only ``quota`` type the byte-cap mode (``tc quota_drop``).
|
||||
mode (``tc nth_drop``; -1 = drop everything maps to every 1st), the
|
||||
kernel-only ``quota`` type the byte-cap mode (``tc quota_drop``),
|
||||
and the kernel-only ``window_drop`` type the time-window mode
|
||||
(``tc window_drop``: single outage, or recurring flaps with an
|
||||
optional per-cycle jitter). The program evaluates the modes in the
|
||||
fixed order nth → quota → window.
|
||||
|
||||
Reconcile is a plain re-set on every apply: the preceding netem
|
||||
apply resets the interface (which detaches the program and drops
|
||||
uBridge's per-interface registry entry), so a mode set re-loads the
|
||||
program from scratch, and a mode left absent is turned off
|
||||
explicitly (idempotent) so a removed filter stops dropping.
|
||||
explicitly (idempotent) so a removed filter stops dropping. For
|
||||
window_drop this also means the schedule restarts on every apply:
|
||||
the first window opens Start-ms from the moment of this call.
|
||||
|
||||
Semantic difference vs the relay, documented server-side: the relay
|
||||
counts packets of BOTH directions through its single filtered
|
||||
bridge, while each veth end's classifier counts only the traffic
|
||||
entering that container — with both endpoints applying theirs, each
|
||||
direction drops every Nth independently.
|
||||
direction drops every Nth independently (their window schedules
|
||||
start ~simultaneously, so a round trip survives only when both
|
||||
directions are outside their windows).
|
||||
|
||||
:param host_ifc: veth host-end interface name
|
||||
:param filters: NIO filters dictionary
|
||||
@ -2406,24 +2415,28 @@ class DockerVM(BaseNode):
|
||||
|
||||
frequency = values_of("frequency_drop")
|
||||
quota = values_of("quota")
|
||||
if not frequency and not quota and self._ubridge_tc_caps is None:
|
||||
window = values_of("window_drop")
|
||||
if not frequency and not quota and not window and self._ubridge_tc_caps is None:
|
||||
# Nothing to set and nothing was ever installed (no probe yet) —
|
||||
# skip touching an old uBridge entirely.
|
||||
return
|
||||
caps = await self._ubridge_tc_capabilities()
|
||||
if not frequency and not quota:
|
||||
if not frequency and not quota and not window:
|
||||
if caps.get("ebpf") == "1":
|
||||
# Only when the eBPF module answers can anything be
|
||||
# installed; "off" is idempotent in uBridge.
|
||||
await self._ubridge_send(f'tc nth_drop "{host_ifc}" off')
|
||||
await self._ubridge_send(f'tc quota_drop "{host_ifc}" off')
|
||||
await self._ubridge_send(f'tc window_drop "{host_ifc}" off')
|
||||
return
|
||||
if caps.get("ebpf") != "1":
|
||||
raise DockerError(
|
||||
"Packet filter(s) {} on a kernel-datapath link need a uBridge with eBPF support "
|
||||
"(tc capabilities reports no ebpf; setcap cap_bpf,cap_net_admin,cap_net_raw=ep on "
|
||||
"the uBridge binary); upgrade uBridge on this compute or keep the link on the "
|
||||
"relay datapath".format(", ".join(sorted(f for f in ("frequency_drop", "quota") if f in filters)))
|
||||
"relay datapath".format(
|
||||
", ".join(sorted(f for f in ("frequency_drop", "quota", "window_drop") if f in filters))
|
||||
)
|
||||
)
|
||||
if frequency:
|
||||
# relay semantics: -1 = drop everything, N = every Nth packet
|
||||
@ -2437,6 +2450,13 @@ class DockerVM(BaseNode):
|
||||
)
|
||||
else:
|
||||
await self._ubridge_send(f'tc quota_drop "{host_ifc}" off')
|
||||
if window:
|
||||
# [start, outage, chance, period?, jitter?] — start is relative
|
||||
# to this apply (the schedule restarts on every reconcile).
|
||||
params = " ".join(str(int(v)) for v in window)
|
||||
await self._ubridge_send(f'tc window_drop "{host_ifc}" {params}')
|
||||
else:
|
||||
await self._ubridge_send(f'tc window_drop "{host_ifc}" off')
|
||||
|
||||
async def _remove_kernel_markers(self, host_ifc):
|
||||
"""
|
||||
|
||||
@ -147,6 +147,20 @@ FILTERS = [
|
||||
{"name": "Chance", "minimum": 0, "maximum": 100, "type": "int", "unit": "%"},
|
||||
],
|
||||
},
|
||||
{
|
||||
"type": "window_drop",
|
||||
"name": "Time window drop",
|
||||
"description": "Drop packets with the given chance inside a time window starting Start ms from the "
|
||||
"moment the filter is applied — a single outage (traffic passes before and after), or with a "
|
||||
"period recurring flaps whose per-cycle timing a jitter randomizes (kernel-datapath links only)",
|
||||
"parameters": [
|
||||
{"name": "Start", "minimum": 0, "maximum": 1000000000000, "type": "int", "unit": "ms"},
|
||||
{"name": "Outage", "minimum": 1, "maximum": 1000000000000, "type": "int", "unit": "ms"},
|
||||
{"name": "Chance", "minimum": 0, "maximum": 100, "type": "int", "unit": "%"},
|
||||
{"name": "Period", "minimum": 1, "maximum": 1000000000000, "type": "int", "unit": "ms"},
|
||||
{"name": "Jitter", "minimum": 0, "maximum": 1000000000, "type": "int", "unit": "ms"},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@ -308,7 +322,8 @@ class Link:
|
||||
raise ControllerError(f"Invalid packet filter parameters: {e!s}")
|
||||
|
||||
if self._created and not self.kernel_datapath:
|
||||
# Kernel-only filters (the netem extensions, the eBPF quota mode)
|
||||
# Kernel-only filters (the netem extensions, the eBPF quota and
|
||||
# window modes)
|
||||
# have no relay equivalent. Only enforced on created links — while
|
||||
# loading a project the datapath is not decided yet (a link that
|
||||
# will be wired on the kernel datapath must accept them), and the
|
||||
@ -723,7 +738,8 @@ class Link:
|
||||
if self.kernel_datapath:
|
||||
# Kernel-datapath links serve every filter type: netem for
|
||||
# delay/loss/corrupt and the netem extensions, cls_bpf for
|
||||
# bpf, the eBPF classifier for frequency_drop and quota.
|
||||
# bpf, the eBPF classifier for frequency_drop, quota and
|
||||
# window_drop.
|
||||
return FILTERS
|
||||
# Relay links: hide the kernel-only types the relay cannot run
|
||||
# (they would be rejected with 409 on update).
|
||||
|
||||
@ -24,10 +24,10 @@ class FilterValidationError(Exception):
|
||||
# kernel/relay choice is purely topological (same compute, docker/docker).
|
||||
#
|
||||
# Packet filters with no uBridge *relay* equivalent: the netem extensions
|
||||
# plus the eBPF quota mode run on the kernel datapath only. The relay's
|
||||
# packet-filter registry only knows frequency_drop / packet_loss / delay /
|
||||
# corrupt / bpf / mark.
|
||||
KERNEL_ONLY_FILTERS = frozenset({"rate", "reorder", "gemodel", "duplicate", "seed", "limit", "quota"})
|
||||
# plus the eBPF quota/window modes run on the kernel datapath only. The
|
||||
# relay's packet-filter registry only knows frequency_drop / packet_loss /
|
||||
# delay / corrupt / bpf / mark.
|
||||
KERNEL_ONLY_FILTERS = frozenset({"rate", "reorder", "gemodel", "duplicate", "seed", "limit", "quota", "window_drop"})
|
||||
|
||||
# Jitter distributions embedded in the netem-extension uBridge (tc_netem_dist).
|
||||
NETEM_DISTRIBUTIONS = ("uniform", "normal", "pareto", "paretonormal")
|
||||
@ -229,6 +229,19 @@ def validate_filter_parameters(filter_type: str, values: List[Any]) -> None:
|
||||
"names": ["Quota", "Chance"],
|
||||
"units": ["bytes", "%"],
|
||||
},
|
||||
"window_drop": {
|
||||
# eBPF stateful classifier (uBridge tc window_drop): packets drop
|
||||
# with the given chance inside [start, start+outage) measured from
|
||||
# the moment the filter is applied — a single outage, traffic
|
||||
# passes before AND after. A period makes the outage recur every
|
||||
# cycle (period >= outage); a jitter re-draws each cycle's outage
|
||||
# and period uniformly in nominal ± jitter (0 = the fixed
|
||||
# schedule). Kernel-datapath only.
|
||||
"params_count": (3, 5),
|
||||
"ranges": [(0, 10**12), (1, 10**12), (0, 100), (1, 10**12), (0, 10**9)],
|
||||
"names": ["Start", "Outage", "Chance", "Period", "Jitter"],
|
||||
"units": ["ms", "ms", "%", "ms", "ms"],
|
||||
},
|
||||
"bpf": {"params_count": (1, 1), "is_text": True, "names": ["Filters"]},
|
||||
}
|
||||
|
||||
@ -335,6 +348,12 @@ def filter_inactive_filters(filters: Dict[str, List[Any]]) -> Dict[str, List[Any
|
||||
else:
|
||||
# latency>0, normal configuration
|
||||
active_filters[filter_type] = values
|
||||
# window_drop's first parameter is a start offset: 0 means "the
|
||||
# outage starts now", not "disabled" — the filter is active whenever
|
||||
# present (an outage of 0 is rejected by validation, so there is no
|
||||
# natural zero-value encoding for "off").
|
||||
elif filter_type == "window_drop":
|
||||
active_filters[filter_type] = values
|
||||
# For other filters, skip if first value is 0 or empty string (means "disabled")
|
||||
elif values[0] != 0 and values[0] != "":
|
||||
active_filters[filter_type] = values
|
||||
@ -398,7 +417,7 @@ def validate_all_filters(filters: Dict[str, List[Any]]) -> None:
|
||||
Validate all packet filters, including cross-filter dependencies that
|
||||
mirror the tc netem grammar (reorder requires delay; gemodel and
|
||||
packet_loss both translate to the netem loss keyword and are mutually
|
||||
exclusive).
|
||||
exclusive) and the eBPF window grammar (period >= outage).
|
||||
|
||||
Args:
|
||||
filters: Dictionary mapping filter types to their values
|
||||
@ -420,3 +439,6 @@ def validate_all_filters(filters: Dict[str, List[Any]]) -> None:
|
||||
raise FilterValidationError("reorder requires delay")
|
||||
if "gemodel" in filters and "packet_loss" in filters:
|
||||
raise FilterValidationError("gemodel and packet_loss are mutually exclusive (both map to the netem loss keyword)")
|
||||
window = filters.get("window_drop")
|
||||
if window and len(window) >= 4 and int(window[3]) < int(window[1]):
|
||||
raise FilterValidationError("window_drop period must be greater than or equal to the outage length")
|
||||
|
||||
@ -497,6 +497,7 @@ async def test_update_nio_kernel_turns_ebpf_modes_off(vm):
|
||||
|
||||
vm._ubridge_send.assert_any_call(f'tc nth_drop "{host_ifc}" off')
|
||||
vm._ubridge_send.assert_any_call(f'tc quota_drop "{host_ifc}" off')
|
||||
vm._ubridge_send.assert_any_call(f'tc window_drop "{host_ifc}" off')
|
||||
for c in vm._ubridge_send.call_args_list:
|
||||
assert "nth_drop" not in c.args[0] or "off" in c.args[0]
|
||||
|
||||
@ -520,6 +521,54 @@ async def test_apply_ebpf_drops_requires_ebpf_capability(vm):
|
||||
vm._ubridge_send.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_nio_kernel_applies_window_drop(vm):
|
||||
"""
|
||||
window_drop becomes the eBPF time-window mode: the positional
|
||||
[start, outage, chance, period, jitter] maps 1:1 onto the tc command
|
||||
(trailing optionals omitted), and an absent window is turned off.
|
||||
"""
|
||||
|
||||
vm._ubridge_hypervisor = _caps_hypervisor(ebpf="1")
|
||||
vm._ubridge_send = AsyncioMagicMock()
|
||||
vm.status = "started"
|
||||
host_ifc, _ = vm._veth_names(0, 0)
|
||||
vm._kernel_veths[(0, 0)] = host_ifc
|
||||
nio = vm.manager.create_nio({"type": "nio_bridge", "bridge": BRIDGE})
|
||||
vm._ethernet_adapters[0].add_nio(0, nio)
|
||||
|
||||
nio.filters = {"window_drop": [0, 2000, 100]}
|
||||
await vm.adapter_update_nio_binding(0, nio)
|
||||
vm._ubridge_send.assert_any_call(f'tc window_drop "{host_ifc}" 0 2000 100')
|
||||
|
||||
vm._ubridge_send.reset_mock()
|
||||
nio.filters = {"window_drop": [500, 800, 100, 2400, 300]}
|
||||
await vm.adapter_update_nio_binding(0, nio)
|
||||
vm._ubridge_send.assert_any_call(f'tc window_drop "{host_ifc}" 500 800 100 2400 300')
|
||||
|
||||
# only the window mode present: nth/quota are explicitly off
|
||||
vm._ubridge_send.assert_any_call(f'tc nth_drop "{host_ifc}" off')
|
||||
vm._ubridge_send.assert_any_call(f'tc quota_drop "{host_ifc}" off')
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_apply_window_drop_requires_ebpf_capability(vm):
|
||||
"""
|
||||
The window mode shares the eBPF capability gate (error names setcap).
|
||||
"""
|
||||
|
||||
hyp = MagicMock()
|
||||
hyp.is_running.return_value = True
|
||||
hyp.send = AsyncioMagicMock(return_value=["netem=delay;ebpf=0;cbpf=1"])
|
||||
vm._ubridge_hypervisor = hyp
|
||||
vm._ubridge_send = AsyncioMagicMock()
|
||||
host_ifc, _ = vm._veth_names(0, 0)
|
||||
|
||||
with pytest.raises(DockerError, match="setcap cap_bpf"):
|
||||
await vm._ubridge_apply_ebpf_drops(host_ifc, {"window_drop": [0, 2000, 100]})
|
||||
vm._ubridge_send.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_apply_ebpf_drops_skips_old_ubridge_without_filters(vm):
|
||||
"""
|
||||
|
||||
@ -428,7 +428,7 @@ async def test_update_suspend_kernel_link_sends_no_synthetic_filter(project):
|
||||
async def test_kernel_datapath_eligible_with_netem_extension_filters(project):
|
||||
"""
|
||||
All netem-extension types have a tc netem equivalent — they keep the
|
||||
link kernel-eligible (only frequency_drop still forces the relay).
|
||||
link kernel-eligible (kernel/relay is a purely topological choice).
|
||||
"""
|
||||
|
||||
link, node1, node2 = await _kernel_link(project)
|
||||
@ -441,6 +441,8 @@ async def test_kernel_datapath_eligible_with_netem_extension_filters(project):
|
||||
"gemodel": [100, 0, 30],
|
||||
"seed": [42],
|
||||
"limit": [5000],
|
||||
"quota": [1000000, 100],
|
||||
"window_drop": [0, 800, 100, 2400, 200],
|
||||
}
|
||||
# gemodel/packet_loss are mutually exclusive — drop packet_loss for the
|
||||
# eligibility check (validation rejects the mix elsewhere)
|
||||
@ -469,9 +471,9 @@ async def test_update_netem_extension_filters_accepted_on_kernel_link(project):
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_kernel_only_filters_rejected_on_relay_link(project):
|
||||
"""
|
||||
Netem-extension / quota filters have no relay equivalent: setting them
|
||||
on a created relay link is a 409. (The link is relay-wired because the
|
||||
endpoints are mixed node types.)
|
||||
Netem-extension / quota / window_drop filters have no relay equivalent:
|
||||
setting them on a created relay link is a 409. (The link is relay-wired
|
||||
because the endpoints are mixed node types.)
|
||||
"""
|
||||
|
||||
link, _n1, _n2 = await _relay_link(project)
|
||||
@ -481,6 +483,11 @@ async def test_update_kernel_only_filters_rejected_on_relay_link(project):
|
||||
|
||||
with pytest.raises(ControllerError, match="kernel-datapath"):
|
||||
await link.update_filters({"rate": ["512kbit"]})
|
||||
with pytest.raises(ControllerError, match="kernel-datapath"):
|
||||
await link.update_filters({"window_drop": [0, 800, 100, 2400]})
|
||||
with pytest.raises(ControllerError, match="period must be greater than or equal"):
|
||||
# cross-parameter rule mirrors the tc window grammar
|
||||
await link.update_filters({"window_drop": [0, 2000, 100, 1000]})
|
||||
# the stored filters are untouched
|
||||
assert link.filters == {}
|
||||
|
||||
@ -489,6 +496,24 @@ async def test_update_kernel_only_filters_rejected_on_relay_link(project):
|
||||
assert link.filters == {"frequency_drop": [7]}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_window_drop_accepted_on_kernel_link(project):
|
||||
"""
|
||||
window_drop rides the kernel NIO to both endpoints — including the
|
||||
start=0 form (an immediate outage is NOT an "inactive" filter).
|
||||
"""
|
||||
|
||||
link, node1, node2 = await _kernel_link(project)
|
||||
await link._prepare()
|
||||
|
||||
filters = {"window_drop": [0, 800, 100, 2400, 200]}
|
||||
await link.update_filters(filters)
|
||||
assert link.filters == filters
|
||||
by_node = {entry[0].id: entry[3] for entry in (await link._prepare())}
|
||||
assert by_node[node1.id]["filters"] == filters
|
||||
assert by_node[node2.id]["filters"] == filters
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_kernel_only_filters_accepted_before_creation(project):
|
||||
"""
|
||||
|
||||
@ -383,3 +383,36 @@ class TestNetemExtensionFilters:
|
||||
# frequency_drop runs on both datapaths (relay userspace filter /
|
||||
# eBPF every-Nth) and stays
|
||||
assert clean == {"frequency_drop": [7]}
|
||||
|
||||
def test_window_drop_valid_and_invalid(self):
|
||||
# the three delivered forms: single outage, recurring, jittered
|
||||
validate_filter_parameters("window_drop", [0, 2000, 100])
|
||||
validate_filter_parameters("window_drop", [500, 800, 100, 2400])
|
||||
validate_filter_parameters("window_drop", [500, 800, 100, 2400, 300])
|
||||
with pytest.raises(FilterValidationError, match="expects 3 to 5"):
|
||||
validate_filter_parameters("window_drop", [500, 800])
|
||||
with pytest.raises(FilterValidationError, match="expects 3 to 5"):
|
||||
validate_filter_parameters("window_drop", [500, 800, 100, 2400, 300, 1])
|
||||
with pytest.raises(FilterValidationError, match="Outage"):
|
||||
validate_filter_parameters("window_drop", [0, 0, 100])
|
||||
with pytest.raises(FilterValidationError, match="Chance"):
|
||||
validate_filter_parameters("window_drop", [0, 2000, 101])
|
||||
with pytest.raises(FilterValidationError, match="Jitter"):
|
||||
validate_filter_parameters("window_drop", [0, 2000, 100, 2400, 10**10])
|
||||
|
||||
def test_window_drop_is_kernel_only(self):
|
||||
clean, dropped = split_kernel_only_features({"window_drop": [0, 800, 100, 2400], "delay": [10, 0]})
|
||||
assert dropped == {"window_drop"}
|
||||
assert clean == {"delay": [10, 0]}
|
||||
|
||||
def test_window_drop_period_must_cover_outage(self):
|
||||
with pytest.raises(FilterValidationError, match="period must be greater than or equal"):
|
||||
validate_all_filters({"window_drop": [0, 2000, 100, 1000]})
|
||||
# period == outage is accepted (uBridge allows it); shorter is not
|
||||
validate_all_filters({"window_drop": [0, 2000, 100, 2000]})
|
||||
|
||||
def test_window_drop_start_zero_is_active(self):
|
||||
# start=0 means "the outage starts now" — unlike other filters a
|
||||
# zero first parameter must NOT be treated as "disabled"
|
||||
active = filter_inactive_filters({"window_drop": [0, 2000, 100]})
|
||||
assert active == {"window_drop": [0, 2000, 100]}
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user