A correctness read-through of the five delivered datapath documents, plus
the two code comments that carried the same stale vocabulary.
Datapath selection: the VendorDockerVM selector is `console_type=docker_exec`
alone (`GNS3_SKIP_INIT` and friends are behavior knobs of the chosen class);
the cross-compute and anchor-less fallbacks are named for what they are; the
Ethernet-switch endpoint is the documented exception to the anchoring rule,
and the IOL runner container is called out as the Docker node that does not
anchor on a veth.
Filter vocabulary: the stateful eBPF program evaluates four modes
(nth → quota → window → flow), GNS3 exposes three of them, and window_drop
joins the kernel-only list; the correl suffixes gate on the `rate` token as
the extension-build marker.
group_fwd_mask: the 0xfffd port mask is applied inside uBridge's `brctl addif`
itself, not as a separate command — the link-creation sequences and the
operation tables no longer show an arrow the server never sends. Same class
of fix for `link set up` on the IOL anchor (addif already brought it up; the
command only pushes a suspended anchor back down) and for the QEMU carrier
replication (QMP `set_link`, gated by `replicate_network_connection_state`).
Lifecycles and stop orders re-worded around the invariant rather than the
node type: the fd holder must release before the devices are deleted, which
is QEMU's process-first order, IOU's uBridge-first order and Dynamips'
hypervisor-NIO-first order. Deferred wiring for a stopped endpoint is
documented as the common path it is, and the Dynamips "stop is not a close"
note now says what recovery from a dead hypervisor actually requires.
Dynamips specifics: a c7200 needs an adapter in slot 1 (slot 0 is the fixed
I/O slot), the tap-fd direction is stated correctly (writes inject into the
kernel, reads return what the kernel transmits), and the anchor registry is
name-keyed against the controller's port matrix by convention, not derivation.
IOL port bridge naming: `bridge{bay}` on unit 0, `bridge{bay}_{unit}`
elsewhere; the leg swap happens on link delete/re-create or a datapath
switch, while suspend is an in-place NIO update and a project reopen is a
fresh build.
Verified and Roadmap sections: the in-repo e2e suite is distinguished from
the isolated-instance development harness (IOU), assertions are stated with
their bounds and capability gates, and the roadmap entries collapse into the
one remaining project — VXLAN/GENEVE for cross-compute links.
Code comments: `flow` added to the documented mode order in the eBPF apply
path, and the `nio_bridge` module docstring corrected on which filter types
run on the kernel datapath (bpf via cls_bpf, frequency_drop/quota/window_drop
via the eBPF classifier — neither is relay-only any more).
The dynamips doc gets the full suite: anatomy (anchors reached by the
hypervisors' TAP fds, uBridge's AF_PACKET-tap / tc-qdisc sidebands), a
one-way copy-numbered relay comparison (six kernel/user crossings per
way against the kernel datapath's two TAP-fd legs), a staged walkthrough
(reserved-range diamond, egress clsact chain, capture/markers observation
sideband, copies 1-2) and a control sequence of real commands in real
order (hypervisor fd first, capture before markers, carrier last; router
handlers, not computes, as the per-node participants).
The four sibling frame paths are unified on the same form: netem qdisc
node labels, one capture/markers node outside the stages with AF_PACKET
tap edges anchored at ingress and post-chain egress, copy numbering
wherever the trace crosses the kernel/user boundary (qemu's two TAP-fd
legs, iol-docker's six socket/TAP legs), stage subgraphs with role-noted
titles everywhere (iol-docker restructured from its flat chain, iou's
chA/tapA variants folded into the canonical shapes).
The documentation skill records the rules as v3.1.0: four diagram kinds
(anatomy / comparison / control sequence / walkthrough), one step per
node, dashed sidebands per role with meaningful direction, numbered
crossings when copies are the point, glossary before the first diagram.
Replace the ASCII art with a mermaid suite (steady-state topology with
namespace boxes and sideband edge semantics, one-way frame path with
per-object boxes, link-creation sequence, full relay path with the four
kernel/user copies), add a term glossary ahead of the diagrams, reflow the
prose to soft-wrap, and refresh the stale parts: the configuration section
now describes enable_kernel_datapath as the global switch it already is,
with every anchorable node type landed and linked to its feature doc.
A Linux bridge standing in for a cable does not forward the IEEE 802.1D
reserved range (01:80:c2:00:00:00-0f) by default, so LACP/LLDP/802.1X never
crossed a kernel-datapath link while the UDP relay carried them fine. uBridge
now opens every enslave'd port's IFLA_BRPORT_GROUP_FWD_MASK to 0xfffd at
brctl addif time (branch feature/bridge-tap-l2only, commits 0284500/03a18f1/
cf4e494), which the six kernel-datapath feature docs now document as a
section: STP, LACP, 802.1X and LLDP/DCBX cross; 802.3x PAUSE / PFC stays
dropped — br_handle_frame()'s case 0x01 discards it unconditionally, no mask
can enable it, and PFC labs must use a relay link. The full contract, kernel
references and requirements live in the new design spec, marked landed.
The relay engines' observable paths had no live coverage: every
capture/start call in the suite sat on a kernel anchor (AF_PACKET
capture start_kernel) and markers were only asserted on marker
add_kernel. The two relay engines' own command shapes were unit-tested
only:
* bridge engine - bridge start_capture / bridge add_packet_filter ... mark
* iol_bridge engine - iol_bridge start_capture / iol_bridge add_packet_filter ... mark
Docker relay control (bridge engine, 1 passed 33s): the capture start
REST call writes the ICMP exchange to a pcap off the relay bridge; a
mark filter attached with capture_node_id n1 signals exactly 10
marker.match events over the dedicated marker WS for 5 pings (5
requests + 5 replies, both directions), carrying node_id/link_id/tag and
the busybox frame length, writes exactly those 10 frames to the marker
pcap, and deleting the marker takes the pcap with it.
IOU relay control (iol_bridge engine, 1 passed 173s): the same two
checks on iol_bridge start_capture and the port's mark filter, driven
by the real IOS ping through the fabric socket relay.
Not duplicated in the ESW/VPCS/cloud relay controls: they issue the same
uBridge bridge commands through their own call sites, which stay covered
by unit tests; the IOL container relay goes through the same iol_bridge
engine already exercised here.
Extend the docker and IOU relay negative controls with the filters the
relay itself supports (delay, frequency_drop, bpf), asserted through real
ICMP — the first live coverage of the relay filter path:
* bridge engine (docker): delay 100 grows the RTT by ~200 ms and comes
back on clear; frequency_drop 2 kills the round trips (the filter counts
every packet crossing the bridge, both directions sharing one counter,
so on an alternating ping stream one whole direction dies; verified
against uBridge directly), frequency_drop -1 blackholes, and bpf drops
by libpcap expression ('icmp', then 'greater 150' discriminating by
frame size).
* iol_bridge engine (IOU): the same delay and frequency_drop effects
through the port's iol_bridge filter list, plus the available_filters
spot check.
Docker 33 s, IOU 168 s, both green.
The feature doc's Verification section now points at
tests/e2e/test_docker_kernel_datapath.py and what the suite pins down
(anchors/bridges, the filter matrix through real traffic, suspend,
capture, the stop/start and delete/re-create beats, the relay control and
the relay->kernel reopen upgrade), including the digest-pinned image the
scenarios share.
The feature doc covers the architecture (one irreducible userspace hop
on the unix-socket leg; the link segment kernelized through the port
bridge's swappable TAP leg), the capability gate, the anchor lifecycle,
the swap flow, the inherited capture/markers/filters, the relay
fallback and the live verification — including the ensure-then-add
lesson (tap create is strictly create-only).
The other datapath docs' roadmaps catch up: the IOL runner container
has landed (and the ethernet-switch doc's stale 'cascade deferred'
exclusion from d2f8aba58 is corrected while there); the docker doc's
adapter table no longer rejects IOLDockerVM.
* docs/features/iou-kernel-datapath.md: architecture (one irreducible
userspace hop on the fabric leg, kernel link segment), datapath
selection, the anchor lifecycle with the reverse-of-QEMU stop order,
capture/markers/filters, relay fallback, and the verified numbers (an
isolated server with two IOU nodes behind a fake ELF image, the test
playing the IOL fabric: 64/64 kernel, 62/62 relay).
* docs/design/ubridge-iol-tap-anchor-spec.md: delivered — implemented on
the ubridge fork's feature/iol-tap-anchor branch (both commands, the
listener hardening, plus releasing every port NIO on iol_bridge delete
even when the bridge was not running), with the implementation notes
that came back (208 vs the older 214 for a missing bridge; one perror
line per frame dropped on a down anchor).
* The Docker/QEMU feature docs point at it from their roadmaps.
Kernel eligibility becomes "both endpoints on one compute, each able to
anchor": Docker always (adapters are born as veth pairs), QEMU when that
compute's uBridge can create the persistent TAPs its adapters anchor on. The
question is asked per compute, because the link needs both ends anchored --
so a mixed or half-upgraded deployment degrades to the relay instead of
failing an attach.
probe_tap_support() creates and deletes one throwaway persistent TAP through a
scratch uBridge, cached by binary identity like the tc report; /capabilities
exposes `ubridge_tap` (absent = unknown, which keeps links on the relay).
The compute route for a QEMU adapter NIO took UDPNIO only, so a kernel link
was rejected with 422: its body and response now accept
Union[UDPNIO, BridgeNIO] exactly like Docker's.
Stale "Docker-to-Docker" wording in the config, link and MCP descriptions now
talks about anchors, and docs/features/qemu-kernel-datapath.md documents the
datapath -- verified end-to-end on a real instance: 32/32 kernel-datapath
checks and 29/29 on the relay variant.
Every host-side device uBridge creates for the data plane (persistent TAP
anchors, veth host ends, per-link kernel bridges) must be pure L2: no IPv4
address, no IPv6 address and no IPv6 stack activity on that device.
Measured today: an idle TAP emits ~6 frames / 2 s of the host's own MLD/DAD
from its own link-local, the production Docker veth host end carries
fe80::5c30:e8ff:fe00:5d8d/64 and the per-link bridge fe80::f0d9:94ff:feac:e42b/64.
On a kernel link the bridge floods those frames into the emulated segment, on a
relay link uBridge relays them to the peer, and the host answers neighbor
solicitations for its own address (a phantom IPv6 neighbour for the emulated
nodes). IPv4 needs no equivalent knob: nothing self-provisions it, and the
anchors hold no IPv4 address.
The spec adds one command (link l2only <iface> [on|off], netlink addr-gen-mode
none -- never a /proc sysctl write, which fails the DAC check for a non-root
setcap'd uBridge), applied by the four creators themselves, with an
idle-silence acceptance test and a known-noise note in the Docker kernel
datapath document. No change to existing command grammar.
Three fixes from live-calibrating the kernel impairment filters on an
isolated instance (Docker-to-Docker veth link, per-sequence evidence):
- update_filters skipped an unchanged filters dict as a no-op, so a
one-shot window_drop outage that had already passed could never be
re-armed: re-applying the same values produced no datapath command at
all. A PUT still carrying window_drop now reconciles, restarting the
schedule from the apply as documented. Measured: identical re-apply
moves the outage window from seq 50-70 to 70-91 (arm at +8s of the
new apply).
- gemodel parameter semantics were documented backwards in the
validation rules, available_filters, the MCP/copilot descriptions and
the feature docs. Measured against the kernel (per-direction counters,
offered = sent + dropped): p is the good-to-bad transition chance, r
the bad-to-good transition, 1-h the loss chance in the bad state (the
good state loses nothing) - mean one-way loss p/(p+r)*(1-h), e.g.
[10,30,100] -> 24.8%/25.7% measured vs 25% predicted, [5,45,100] ->
10.8% both ways vs 10%.
- kernel_datapath was silently stripped from every REST/MCP link payload
by the response model (LinkBase carried no such field), so clients
could not tell a kernel link from a relay one.
Also verified on the same setup: the 3-argument window_drop form works
exactly as specified ([3000,5000,100] -> a single seq 31-55 outage,
[20000,6000,100] -> seq 115-144), and both generations of the command
builder emit the parameters unpadded.
Adds an API test for the field and controller tests for the no-op skip
and the window_drop re-arm exemption.
Consume uBridge's ebpf_modes contract (feature/tc-window 40c36ae): a
stateful mode is usable iff ebpf=1 and its token is listed; builds
predating the field keep their shipped modes (nth/quota/flow), so the
pre-correction window semantics can never be installed silently.
- gns3server/utils/tc_capabilities.py: shared reply parser, mode
policy and filter-type mappings (compute and controller gate on the
same rules)
- gns3server/compute/ubridge/tc_probe.py: standalone probe spawning a
throwaway uBridge as the server's own user, cached by binary
identity (path/mtime/size - the version string cannot distinguish
rebuilds); failures cache as unknown
- GET /v3/compute/capabilities gains ubridge_tc {netem, ebpf,
ebpf_modes, cbpf}; the controller forwards it (Compute.capabilities)
and available_filters hides kernel-datapath filter types an endpoint
compute cannot run (both ends apply their own filters, so one
incapable compute hides the type); computes without a report keep
the full list and apply-time 409 stays the guard
- docker: _ubridge_apply_ebpf_drops gates per mode token; the cleanup
path only sends "off" for modes the build declares
- split DockerVM's kernel-datapath machinery (veth lifecycle, carrier,
marker primitives, tc filter translation, kernel NIO teardown) into
compute/docker/docker_kernel_datapath.py (DockerKernelDatapathMixin)
The bridge teardown only hung off the link-removal path, but deleting a
node or closing a project never issues a per-link delete — the veths
disappeared with their containers and left the bridge behind as an empty
orphan (reproduced: create a kernel link, DELETE the project, the
gns3{link_id} bridge survives). `_remove_kernel_veths` now deletes the
node's kernel-NIO bridges alongside the veth host ends: both endpoints do
it, the peer's still-enslaved port answers EBUSY (suppressed, last one
wins) exactly like link deletion, a relay NIO's uBridge bridge is left
alone (it dies with the process), and a node restart rebuilds the bridge
from the NIO in _connect_nio.
Found while cleaning up the window_drop e2e: seven such bridges had
accumulated on this host. Re-verified live: the bridge is gone after the
project delete, and the window_drop e2e (which includes a node restart
with the kernel link attached) still passes 15/15.
uBridge's window mode gained the semantics the B.2 spec asked for —
`tc window_drop <if> <start> <outage> <pct> [<period> [<jitter>]]`: a
single outage that traffic passes before *and* after, recurring flaps
with period >= outage, and a per-cycle jitter — so the kernel-only
`window_drop` filter type is now exposed:
- validation mirrors the tc grammar (3-5 params, ranges, period >=
outage, jitter cap) and adds window_drop to KERNEL_ONLY_FILTERS;
filter_inactive_filters keeps start=0 — unlike every other filter a
zero first parameter means "the outage starts now", not "disabled".
- `_ubridge_apply_ebpf_drops` sends `tc window_drop` (positional
pass-through) and turns the mode off explicitly when absent, under the
same ebpf=1 gate.
- FILTERS entry (Start/Outage/Chance/Period/Jitter) + available_filters.
The schedule is relative to the apply: reconcile re-sets the classifier
on every apply, so any filter change, node restart or link reset restarts
the window.
Verified on a two-container kernel link, 15/15: a single outage
[1000, 1500, 100] arrives per-seq as [0..4, 13..19] (passing before the
window, dropped inside, passing after); recurring 1000/2000 ms = 52 %
round trip (duty cycle); period == outage = permanent outage; period <
outage = 409 before uBridge; jittered flap keeps the ~50 % duty cycle;
30 % chance on a permanent window = 53 % round trip against the
per-direction theory 1-0.7^2 = 51 %; coexist with netem; node restart
restores; available_filters kernel vs relay (409 + hidden); teardown
clean. P6b re-run as a regression on the same uBridge build: 13/13.
Worth recording for the next rebuild: the first delivered window program
loaded as root but reported ebpf=0 through gns3-server — the verifier's
non-root path loses the loop counter's bound, treats the catch-up loop as
unbounded and rejects the *whole* program (E2BIG, 8193 pending states),
which also took frequency_drop and quota down, with ebpf=0 as the only
symptom. Smaller trip caps did not help; the loop had to go (uBridge
572ca0c), on top of a Makefile header-dependency fix (53e34bb) that had
been shipping a stale program object.
Consume the eBPF impairment classifier delivered in uBridge
feature/tc-precision: frequency_drop becomes the exact every-Nth mode
(tc nth_drop; -1 = drop everything maps to every 1st) and the new
kernel-only quota type is the byte-cap mode (tc quota_drop, 100 = hard
cutoff). KERNEL_UNSUPPORTED_FILTERS is gone entirely — kernel/relay
eligibility is now purely topological. Both modes gate on ebpf=1 from
the per-process tc capabilities probe (error names the setcap fix); the
cleanup path (modes off) only talks to the eBPF commands when the probe
reported them. Reconcile is a plain re-set: the netem reset-before-set
already detaches the program and clears uBridge's per-interface
registry, so a mode set after it is a fresh load+attach.
Documented semantic difference: the relay counts both directions
through one bridge counter, the kernel counts per direction — N=3
round-trip loss is 1-(2/3)^2 = 55.6% on kernel vs ~1/N on relay
(measured 57%). window_drop is NOT exposed: the delivered
implementation advances the window with no gap, making 'outside the
window packets pass' unreachable (needs outage/period fields —
recorded in the spec); flow_drop awaits a parameter-shape decision.
Also fixes a pre-existing restart bug the e2e exposed: _bridges (relay
bridge names) was never cleared on stop, so restarting a node with a
relay link skipped 'bridge create' and failed on add_nio_udp.
e2e (isolated instance): 13/13 — pref-1 tc_impair classifier on both
veths, 57% round-trip for N=3, -1 = exact 100%, quota hard cutoff,
delay coexistence, clean teardown, suspend/resume, node restart,
relay-leg frequency_drop regression through an Ethernet switch. Full
suite: 1925 passed.
Surface the netem extensions delivered in uBridge feature/tc-netem-ext
as GNS3 filter types: rate (tc-style bandwidth string), reorder (needs
delay), gemodel (Gilbert-Elliot bursty loss, exclusive with
packet_loss), duplicate, seed and limit, plus the delay jitter
distribution and the loss/dup correlation parameters. All are
kernel-datapath only — the relay's packet-filter registry has no
equivalent, so available_filters hides them on relay links, setting one
on a created relay link returns 409, and the project-load path drops
them with a warning (update_filters runs before the datapath is
decided, so it cannot reject there). Validation mirrors the tc grammar:
reorder requires delay, gemodel and packet_loss are mutually exclusive,
a distribution requires jitter > 0, rate is integer+unit capped at
100gbit.
The translation in DockerVM._netem_command_parts emits the frozen
grammar order and gates extension keywords on the 'tc capabilities'
netem token list (probed once per uBridge process; a plain
delay/loss/corrupt/dup filter never probes, keeping old uBridge builds
untouched; the correl suffix has no token and gates on 'rate' as the
extension-build marker).
Reconcile became reset-before-set: the kernel's netem change merges
optional attributes (rate, correlation, reorder, corrupt, gemodel,
distribution, seed keep their previous value when the new message omits
them), so removing a parameter needs an explicit 'tc reset' — the
bpf_drop filters are re-added right after in the same flow, so the
clsact churn is invisible. The capability check runs before the reset
so a failed update leaves the previous qdisc in place.
e2e (isolated instance, kernel link): 24/24 — rate 512kbit adds exactly
2x the per-packet serialization to the RTT (45.5 ms for 1400-byte
pings), paretonormal jitter + rate-cleared leak check, reorder gap 5,
gemodel 50% bursty loss, duplicate, loss 30% correl 50%, seed 42 +
limit 5000 visible in the qdisc dump, suspend/resume and node restart
keep the filters, relay link rejects rate with 409, teardown leaves no
netem. Full unit suite: 1916 passed.
The GNS3 bpf filter (one expression per line, any line matching drops the
packet) now runs on the kernel datapath: uBridge's tc bpf_drop compiles
each line with libpcap (same compiler as the relay filter) into classic
BPF, attaches it to the veth host end's clsact egress as a cls_bpf
classifier with a gact TC_ACT_SHOT action, and the kernel migrates the
cBPF to eBPF internally -- no CAP_BPF needed.
- reconcile is flush + one add per line at priorities 10, 11, ... (OR
semantics, same as the relay's per-line filters); tc reset (link
delete / clear) is the full restore in uBridge and clears them
- capability-gated: the compute probes "tc capabilities" once per uBridge
process (cached, invalidated on stop); without cbpf=1 the filter apply
fails with an actionable upgrade error, and old uBridge builds without
the tc module are never touched when no bpf filter is present
- a line that compiled for the controller's tcpdump but not for the
compute's uBridge libpcap is skipped with a warning, mirroring the
relay path
- KERNEL_UNSUPPORTED_FILTERS shrinks to frequency_drop (its eBPF stateful
classifier is still pending, spec part B)
e2e (22/22): bpf "icmp" = 100% loss; size-discriminating expression
("greater 150") passes 98-byte and drops 300-byte pings (real byte-level
match); two-line OR; coexistence with netem (delay 50 -> small pings
100.2 ms while big dropped); clear removes clsact; suspend/resume keeps
the drops; link delete + recreate leaves no residual filters.
Impairment filters with a netem equivalent (delay, packet_loss, corrupt)
now run on the kernel datapath as one tc netem qdisc per veth host end,
pushed through uBridge's netlink tc module (tc netem set / tc reset).
This removes the relay delay filter's nanosleep bottleneck (upstream
Both link endpoints carry the filters: each end's qdisc egress covers
the traffic entering that container, so every direction is impaired
exactly once -- the same net effect as the relay, where both directions
cross the single filtered bridge.
- controller: eligibility keys on relay-only types (frequency_drop, bpf)
instead of any filter; kernel links push the real filters (the
synthetic suspend frequency_drop no longer flips a suspended link to
relay on reopen/reset); update_filters 409s only relay-only types;
available_filters hides them on kernel links
- compute: create_nio whitelists netem-capable filter types; the NIO
update path keeps its own guard (PUT mutates the NIO in place);
_ubridge_apply_netem translates the filter dict into one atomic
netem-set (full re-apply IS the reconcile); node restart restores the
qdisc, link deletion detaches it (the veth survives the link)
- e2e (27/27): delay 100 -> 200.2 ms RTT, atomic replace 50 -> 100.2 ms,
clear -> baseline, packet_loss 30 -> 48 % round-trip (expected 51 %),
suspend/resume and node restart keep the qdisc, link delete leaves no
residual impairment
Cover the four-stage stack (kernel datapath, AF_PACKET capture, markers,
veth-everywhere): adapter interface types per node class, datapath
selection, veth lifecycle, link operations, suspend/capture/markers
semantics, uBridge command surface and the tc netem filters roadmap.
Cross-link from the traffic-insight marker doc for the kernel variant.
