SkillsManager refused to start when the local skills directory was not a
git repository (cloning into a non-empty path fails), and reload_skills()
returned early when injection/ was missing, so device and feature skills
never loaded. reload_packet_analysis_protocols() had the opposite bug:
an empty result wiped the registry while still reporting success.
- initialize(): accept a non-empty directory without .git as a local-only
source (skip automatic updates, keep loading files); degrade to local
files when GitPython is unavailable.
- _update_if_needed(): re-create/clone when the local directory vanished
or moved, instead of reusing a stale handle forever.
- reload_skills(): injection, device and feature load independently; a
missing category keeps its previously loaded entries.
- reload_packet_analysis_protocols(): an empty/missing directory keeps
the existing registry and reports failure.
- reload_skills_repository(): drop the console guard cache so
config/console_guard.txt applies on reload.
MCP device_skill_get now exposes the category filter, documents the
single-file (topic-less) layout, and returns a sections list for
index/summary of skills that have no topics/issues. An uninitialized
skills repository fails with a clear error instead of an empty list.
The skills repository carries per-device operational knowledge the
generic tools cannot know (FRR's console lands in vtysh, its ping has
no -c, scripted config needs 'do' prefixes; VPCS syntax; IOS paging
pitfalls), but only the embedded copilot could read it — MCP's sole
consumer was console_guard.txt.
device_skill_get reuses the copilot's framework-free core
(get_skill/list_available_skills) rather than the LangChain tool shell,
so both surfaces share the same access protocol by construction:
list -> device topic index -> one targeted topic body, bodies only on
an explicit topic request. The caller passes the skill key exactly as
the list returned it; there is no name mapping or node-based resolution
in this layer.
The tool description also tells clients that support subagents to
delegate the discovery steps on long tasks, keeping the full skill
bodies out of the main conversation's context.
Tool handlers run in the session task created at initialize and inherit
its context; contextvars set by later requests never reach them. In a
long-lived API-key session (the documented setup — keys never expire)
the minted 24h JWT eventually expired while the transport kept
authenticating, so every tool call failed with 401.
Remember the raw API key at initialize; when the session's JWT has aged
out, take the current one from the per-request API-key cache (every
tools/call POST refreshes it) or re-mint synchronously. JWT logins are
unchanged: the client's own credential expiring is the client's
business.
SSH disconnect SIGHUPs the session; the server absorbs it (reload) while
every emulator child default-terminates without cleanup, leaving stale
anchor TAPs that 409 the first start afterwards. Root-caused with a
second-level journalctl/gns3.log correlation; addressed by deployment
guidance (systemd / setsid nohup / tmux), product-side hardening noted
as deferred. Indexed under Known Issues.
- Drop the unused fastmcp wrapper pin (mcp-requirements.txt) and depend on
the official SDK directly: mcp>=2.3,<3. The code only ever imported
mcp.server.* modules; fastmcp was pulled in solely as a dependency
vehicle and its <4 pin kept mcp locked at 1.x.
- Migrate to the SDK 2.x API: FastMCP -> MCPServer
(mcp.server.mcpserver); transport_security moves from the constructor
into streamable_http_app().
- Replace the deprecated SSE transport (spec-deprecated 2025-06-18) with
Streamable HTTP at POST/GET/DELETE /v3/mcp/transport, mounted at /v3/mcp.
- Run the StreamableHTTP session manager in the FastAPI lifespan
(Starlette does not execute mounted sub-app lifespans), entered before
startup finishes so no request passes the MCP ready gate first.
- Authenticate every request (the SSE wrapper only checked the opening
GET and trusted POSTs by session). Add a 5-minute TTL cache for API-key
resolutions so the stricter per-request check does not cost a bcrypt
round on every call.
- Update docs to the new endpoint, auth flow, and client configuration.
Wire-format verified unchanged against SDK 1.30 (structured output
auto-detection and content conversion are byte-identical for the
list[dict] returns used by all tool handlers).
Aligns with uBridge's tc.md correction: TCA_NETEM_PRNG_SEED (uapi slot 14,
added in Linux 6.6 — not the nonexistent TCA_NETEM_SEED at "5.1") is what
the seed filter rides on; pre-6.6 kernels parse qdisc options
non-strictly and silently drop the attribute, so seed determinism is
unavailable there while every other netem knob applies. Stated in both
filter tables and the seed section of packet-filters.md, the seed rows of
the MCP and copilot tool descriptions, and the kernel-impairment spec.
Marker pcaps are written by uBridge on the compute that hosts the capture
node. The tag replay only looked in the project folder on the controller.
So a source from a remote compute showed count 0 and no frames, with no
error. Its frame detail returned 404 and said nothing ever matched (#3010).
Each source now gets its pcap through the compute of its capture node.
Local files are read as before. Remote files are fetched over HTTP on
every acquire into a small cache (32 idle files, cleared at shutdown).
A file is replaced only when its bytes changed. This keeps the old mtime,
so the sharkd session for that file stays alive.
When a compute cannot be reached, its source now shows an error message
in the timeline instead of just showing nothing. The detail endpoint now
returns an error instead of 404.
The custom 14789 was picked to keep our demux socket off the ports other
VXLAN software listens on, but a private default surprises operators and
blocks interop with gear that pins the standard port. Default to the
IANA-assigned 4789 instead; the gns3_server.conf override is unchanged
for computes where swarm overlay/flannel-style VXLAN already owns 4789.
Mixed-version deployments must pin the port on every compute until they
all run the new default, or the two tunnel ends will disagree.
Per tunnel, not a host setting: a connected UDP socket runs the
kernel's route lookup for the peer, the chosen source address is
matched back to its owning interface, and that interface becomes
the dev= (and with it the derived MTU and the outer source
address). Different peers may ride different lower devices with
independently derived MTUs; an unresolvable peer is a wiring error
that degrades the link to the relay. Re-creation re-resolves too,
so route changes ride the same node-start / link-recreation paths
as MTU changes.
Verified against the running kernel's sources (7.2.6): the 50-byte
headroom is 20 outer IPv4 + 8 UDP + 8 VXLAN + 14 inner Ethernet
header (vxlan_headroom(), include/net/vxlan.h); the vxlan driver has
no NETDEV_CHANGEMTU handler so a live tunnel never follows its lower
device, while the bridge does follow its ports at runtime
(NETDEV_CHANGEMTU -> br_mtu_auto_adjust). Document the operational
consequence: the wiring path re-creates the device on every node
start and link (re)creation, so raising the underlay MTU on both
computes and stop/start-ing the nodes is the fix — no manual
ip link needed.
The scenario needed root for its network namespace and its fabric subnet
turned out host-specific: policy routing (a proxy's table consulted
before main) sends any non-RFC1918 destination out the default route, so
the peer compute was unreachable behind the veth — unfixable by picking
another subnet, and the failure paths leaked namespaces and processes.
Too environment-coupled to keep in the suite.
Unit coverage is unchanged (selection matrix, tunnel spec derivation,
VNI retry/degrade/self-heal, device lifecycle, capability probe); the
feature doc's Verification section now carries the manual two-compute
checklist instead of the e2e reference.
Cross-compute links can ride kernel VXLAN tunnels instead of the uBridge
UDP relay: each endpoint creates one VXLAN device (uBridge's rtnetlink
vxlan module) enslaved into the same per-link Linux bridge the same-host
datapath uses, so filters, markers, capture and suspend keep running on
the anchor unchanged. Opt-in (Server.enable_vxlan_cross_compute, default
off) because the tunnel caps the usable MTU at the underlay minus the
outer headers; the relay stays the default and the fallback.
The controller derives what the pair must agree on: one VNI per link
(upper VNI space, per-link random salt) and deterministic device names
(gvl{link}{node}) that make crashed-run leftovers claimable
(show -> delete -> create). Same-host compute pairs, switch endpoints
and serial/unix-socket containers stay on the relay.
A VNI the kernel refuses self-heals on every path it can surface:
interactive create retries a fresh salt then degrades to the relay;
project open re-dispatches refused batch entries the same way; node
start recognises the refused device, stops the half-started node,
re-wires both ends on a fresh VNI (or degrades the link) and retries
the start. The device MTU is derived from the lower device (dev=) and
the per-link bridge inherits it - no configuration. The capability
probe (ubridge_vxlan in /capabilities) runs one scratch create/delete
on a scratch port that never equals the configured dstport.
The e2e scenario runs two computes on one machine (a network namespace
joined by a veth pair as the fabric) and needs root.
Follow-up enrichments to the Per-filter guide, one per filter, each
answering the question a reader asks right after "what does it do":
- window_drop: explicit recurrence formula ([Start + k*Period, +Outage),
duty cycle) and ASCII timelines of the three forms (one-shot /
recurring / jittered); Chance spelled out as an independent per-packet
draw gated to the window; jitter explained as per-cycle re-draws that
break phase-locking with protocol timers
- quota: transparent until the budget is consumed; Chance equals
packet_loss in draw mathematics but runs in the eBPF classifier stage
(no correlation parameter, outside seed, invisible to captures)
- limit: the phantom-loss symptom (throughput below the configured rate
with rate+delay set = the 1000 default overflowing)
- seed: reproduction also requires the same traffic pattern (draws are
consumed per arriving packet); one pinned realization is not the
distribution — statistical conclusions need several seeds
- gemodel: scattered-vs-bursty timeline at the same 5% mean (fast
retransmit vs RTO), and a parameter recipe (M and L -> r, p)
- reorder: the mechanism is queue-jumping on the netem delay queue —
hence the delay requirement, the limit overflow interaction and the
delay-bounded reorder depth
- rate: per-direction independent shaping (symmetric full duplex, no
shared budget); asymmetric rates need in-guest netem
- duplicate: real-world prototype (802.11 ARQ retransmissions), example
values with the correlation-collapse anchors (10/25 -> ~1.8%, 25/25
-> ~16.7%)
Make the feature doc usable as a standalone reference:
- Parameter reference: all 13 filters' parameters, ranges and units, the
cross-filter rules (reorder+delay, gemodel/packet_loss exclusivity,
period>=outage, distribution needs jitter), the zero-means-disabled
convention, and the UI display names served by available_filters
(the Web UI labels gemodel as "Gilbert-Elliot loss", never by type)
- Per-filter guide: purpose, per-parameter meaning, caveats and a copyable
example for each of the 13 types, including the four jitter
distributions compared (uniform/normal/pareto/paretonormal)
- REST surface: endpoint table with privileges, a filters PUT example,
and common recipes (WAN latency, bursty loss via gemodel, rate cap,
ICMP blackhole, scheduled outage)
- Clarify gemodel's "one-way" mean: per direction, independent instances
per endpoint, round trip compounds to 1-(1-mean)^2; one-directional
loss is not expressible through the link filters dict
- Fix an inaccuracy: available_filters is a dedicated endpoint's response,
not a field of the link payload
uBridge's capture module now serves one AF_PACKET capture per interface,
any number per process, and `capture stop_kernel` takes the interface it
stops — this rides that (breaking, paired-binary) protocol change.
The server-side single-slot tracker (_kernel_capture_ifc with its
reserve/release/owned_by helpers) is gone: same-port duplicates were
already refused by the NIO capturing flag, uBridge answers EALREADY on a
same-interface duplicate, and per-interface stops removed the process-
wide stop the ownership gating guarded against.
The docker e2e drives ten concurrent captures on one node's uBridge and
asserts a per-interface stop freezes exactly the stopped half while the
live half keeps recording.
The ordered startup (uBridge creates the persistent TAP and hands it to
the server's uid before vpcs is launched, vpcs attaches with -e -d), why
the attach needs no privileges while creating a TAP does, that uBridge
stays off the data path, and that the setcap in the vpcs install target
is for standalone use only.
A VPCS node's single adapter anchors on a persistent TAP the server
creates (tap create + set_owner via uBridge) and the vpcs process
attaches to by name: -e -d <gp{id8}e0p0> -n, the same
server-creates/guest-attaches shape as QEMU's netdev. -n (a vpcs fork
feature) keeps the guest's ip command from ifconfig'ing the anchor on
the host; the vpcs build is probed once per binary identity
(vpcs_tap capability, usage scan) and gates kernel links together
with ubridge_tap — anything else keeps the node on the relay (the
legacy local UDP tunnel on a stock binary, an AF_PACKET endpoint on
the anchor with a capable one).
VPCSVM takes KernelDatapathMixin: _connect_nio dispatches kernel
(enslave into the per-link bridge) / relay-on-anchor / legacy,
captures pick capture start_kernel vs the bridge engine, stop runs
under an execute lock and terminates the process before deleting the
anchor TAPs (a held device refuses deletion), and the duplicate-port
guard backstops the controller's. The NIO routes accept nio_bridge
and write suspend back on update.
_delete_anchor_taps verifies each tap is gone from sysfs and retries
a bounded three times: a holder can release its fd a moment after its
process was reaped (vpcs's daemon parent kills its console child and
exits first), so the unconditional delete raced EBUSY and leaked the
anchor intermittently.
The vpcs side needs IFNAMSIZ (16) instead of its own IFNAMESIZ (12)
for the -d name buffer — anchor names are 14 chars and truncated to a
nonexistent device, which TUNSETIFF then tries to create without
CAP_NET_ADMIN (fixed in the vpcs fork).
e2e (tests/e2e/test_vpcs_kernel_datapath.py): kernel and relay
scenarios green against the real binary — real ICMP both ways, §E.1/§
E.2 on the anchors, delay/suspend/capture, link delete/re-create,
node stop/restart self-heal, zero leftovers; the relay control passes
with both the fork binary and a stock one.
The templated drawing_create_text/_rectangle/_ellipse tools cover the
LLM-relevant creation paths, and line/image/curve creation is not
needed from MCP or copilot for now. The management tools stay, so
existing drawings (including images and lines created in the Web UI)
remain fully manageable; drawing_update's raw svg parameter remains
the escape hatch for edits.
The kernel's correlated-random generator (get_crandom, sch_netem.c) truncates
instead of wrapping, starving both edges of the stationary distribution: an
event threshold below the correlation value is almost never reached. Measured
with stock iproute2 on this kernel — uBridge's netlink encoding is identical:
loss 1% + correl 25% drops nothing over 2000 pings (expected ~2%), 10%/25%
measures ~1.8% one way, only the 50% symmetry point survives. duplicate and
reorder correl share the same comparison structure and collapse the same way;
gemodel is immune (Markov chain over plain uniform draws) and is the honest
tool for bursty loss at a chosen mean rate.
Documented in the packet-filters correlation-bias bullet, and surfaced in the
two machine-facing descriptions (MCP link_update packet_loss note, controller
filter schema) so agents and clients stop walking into the trap.
A "Combining filters" section for packet-filters.md: deterministic drops
are an OR-gate (first kill wins — a clsact-dropped frame consumes no rate
serialization, no limit capacity and no netem counter), netem draws are
independent probabilities that multiply in one qdisc pass, and a round
trip squares everything. The couplings that do exist: quota's byte count
sees only frequency_drop survivors (ordered evaluation slows the burn),
delay/reorder/rate fold into one time_to_send, limit x rate x delay is
the tail-drop triangle to watch, and seed pins netem's draws but not the
eBPF program's own PRNG. Closes with a fully-loaded 12-filter walkthrough
of one frame through the gauntlet.
The Packet filters section of docker-kernel-datapath.md was the de-facto
filter documentation every other datapath doc pointed at; move it to
features/packet-filters.md as the node-type-neutral home (veth or TAP
anchor), centered on a mermaid walkthrough of all 13 types through the
anchor's egress chain: clsact first (the eBPF stateful classifier's nth /
quota / window modes at prio 1, then the bpf expression lines at 10-99,
first drop wins), the single netem qdisc behind it (enqueue draws, one
time_to_send drained by the delay queue).
Clarify a naming trap the table verbs invite: uBridge's tc module only
borrows the iproute2 vocabulary — the tc ... tokens are its own hypervisor
command names over hand-built netlink messages (RTM_NEWQDISC /
RTM_NEWTFILTER), raw bpf() syscalls, embedded distribution tables and
instruction array, libpcap as a library only; the tc binary is never
invoked outside the test suite's oracles. Verified against the uBridge
sources (hypervisor_tc.c, tc_impair.bpf.c, tc_ebpf.c), not just the
commands gns3-server sends.
docker-kernel-datapath.md keeps a pointer section; the iol doc's filter
reference and docs/README.md's tree and catalog point at the new doc.
Compute:
- Track the anchor that owns uBridge's single AF_PACKET capture slot: a
second kernel port's capture is refused with a clear error (its flag
rolled back), a failed start no longer wedges the port, and only the
owning port's stop issues the process-wide `capture stop_kernel`
(docker/qemu/iou/dynamips/switch, via the mixin's slot helpers).
- Unwind failed starts instead of leaking uBridge and the anchor TAPs:
QEMU's stop body runs under try/finally, QEMU and IOU start the datapath
inside the launch try, Dynamips undoes a half-attached port (the retry
used to be short-circuited into a silent black hole), and Docker deletes
the veth pair when the namespace move fails.
- Restore the relay filter guard on IOU (kernel-only filters raise
NodeError, an uncompilable BPF line degrades to a warning), validate the
eBPF/quota/window/bpf filter shapes before touching the bridge (short or
non-numeric values surfaced as 500s after partial state), and refuse a
second NIO on an occupied Docker/QEMU port (the guard IOU and Dynamips
already had).
- Probe fixes: `_probe_bridge_tap` survives a failing uBridge start, and
the probe caches key on ctime too, so a chmod/setcap remedy re-probes.
Controller:
- Parse GNS3_UNIX_SOCKET_NIO like the compute does (line-anchored, truthy
value) and mirror the compute's class selection in kernel_anchor_type.
- Read partial capability reports defensively; make BridgeNIO.bridge
required-but-nullable so an omitted key is a 422 rather than a KeyError.
- Strip the packet_loss correlation slot even when it is zero: the relay
filter takes exactly one argument and rejected `packet_loss 10 0`.
Ethernet switch: skip ports whose absorbed anchor does not exist yet (the
join is deferred by design) and roll the ports_mapping back when the VLAN
reconcile fails, so a retry still diffs against the old mapping.
Agent: reject device config entries without a device_name instead of
silently dropping them (a malformed batch used to execute partially while
reporting success).
Also split the netem builder to clear C901 and drop the two per-file ruff
ignores the PR had added, keeping the remove-only policy.
The reproduction section described the harness only in prose; the actual
driver scripts were throwaway lab files. Embed the distilled, runnable
reproducer (drives uBridge's control socket through the repo's own client,
kept tap for the single-variable A/B), the exact measurement semantics
(40 re-enslavements, ~1 kHz / 200 ms membership sampling, a release = the
brif entry vanishing, latency print) and the flip protocol, so a reader
can reproduce the 19-20/40 vs 0/40 numbers on an affected host. Also point
the host-fix layer at the reproducer for re-verification.
A kernel-datapath link could die silently after a delete/recreate: on
NetworkManager hosts, NM's state machine releases an NM-managed TAP from
its bridge 58-87 ms after (re)enslavement (~50% while the tap has not
been assumed yet; the port stays up, no GNS3 process logs anything).
Root-caused with a server-free minimal harness (one uBridge, one tap,
one bridge, an addif/delif loop) and a single-variable NM A/B; the
NetworkManager side is explained by three of its commits (assumption
recheck on external enslavement, master removal during activation,
external changes not touching assumed slaves).
Records: forensic signatures (ifindex unchanged, flags 0x300 cleared,
heal-on-next-update), the reproducer, the verified host fix
(unmanaged-devices conf.d snippet: 19-20/40 -> 0/40, and the e2e
scenario green with test-side workarounds disabled), scope and history
(2016 issue #440 veth incident; the 2026-10-01 ethernet-switch
incident previously misattributed). Cross-linked from the L2-anchor
spec and the switch feature doc.
The esw kernel fast-path scenario deletes and re-creates a link, which
re-enslaves the peer anchor — and on NetworkManager hosts NM's netdev
state machine can release an NM-managed TAP from its bridge some 60-70 ms
after enslavement, ~50% of re-enslavements, while the tap is in NM's
not-yet-assumed state. The freed port stays administratively up and no
GNS3 process logs anything: the membership assert sees the join, then the
ping measures a silent blackhole (this was the line-277 e2e failure).
Root-caused with a server-free harness (one uBridge, one fresh tap, one
bridge, an addif/delif loop): the same tap frees 20/40 while NM-managed,
0/40 after `nmcli device set <tap> managed no`, 20/40 again when
re-managed. Docker's veth host ends are not NM-managed, which is why only
TAP anchors ever showed it — including the "anchor lost its bridge
membership after a ports_mapping update (TAP peers only)" incident from
the switch's development, whose conclusion is corrected in the docs.
Unmanage the anchors once the routers have booted, before the scenario
re-enslaves them; document the failure signature and the host-side remedy
(per-device nmcli call, or NM unmanaged-devices configuration) in the
L2-anchor spec and the switch feature doc.
A correctness read-through of the five delivered datapath documents, plus
the two code comments that carried the same stale vocabulary.
Datapath selection: the VendorDockerVM selector is `console_type=docker_exec`
alone (`GNS3_SKIP_INIT` and friends are behavior knobs of the chosen class);
the cross-compute and anchor-less fallbacks are named for what they are; the
Ethernet-switch endpoint is the documented exception to the anchoring rule,
and the IOL runner container is called out as the Docker node that does not
anchor on a veth.
Filter vocabulary: the stateful eBPF program evaluates four modes
(nth → quota → window → flow), GNS3 exposes three of them, and window_drop
joins the kernel-only list; the correl suffixes gate on the `rate` token as
the extension-build marker.
group_fwd_mask: the 0xfffd port mask is applied inside uBridge's `brctl addif`
itself, not as a separate command — the link-creation sequences and the
operation tables no longer show an arrow the server never sends. Same class
of fix for `link set up` on the IOL anchor (addif already brought it up; the
command only pushes a suspended anchor back down) and for the QEMU carrier
replication (QMP `set_link`, gated by `replicate_network_connection_state`).
Lifecycles and stop orders re-worded around the invariant rather than the
node type: the fd holder must release before the devices are deleted, which
is QEMU's process-first order, IOU's uBridge-first order and Dynamips'
hypervisor-NIO-first order. Deferred wiring for a stopped endpoint is
documented as the common path it is, and the Dynamips "stop is not a close"
note now says what recovery from a dead hypervisor actually requires.
Dynamips specifics: a c7200 needs an adapter in slot 1 (slot 0 is the fixed
I/O slot), the tap-fd direction is stated correctly (writes inject into the
kernel, reads return what the kernel transmits), and the anchor registry is
name-keyed against the controller's port matrix by convention, not derivation.
IOL port bridge naming: `bridge{bay}` on unit 0, `bridge{bay}_{unit}`
elsewhere; the leg swap happens on link delete/re-create or a datapath
switch, while suspend is an in-place NIO update and a project reopen is a
fresh build.
Verified and Roadmap sections: the in-repo e2e suite is distinguished from
the isolated-instance development harness (IOU), assertions are stated with
their bounds and capability gates, and the roadmap entries collapse into the
one remaining project — VXLAN/GENEVE for cross-compute links.
Code comments: `flow` added to the documented mode order in the eBPF apply
path, and the `nio_bridge` module docstring corrected on which filter types
run on the kernel datapath (bpf via cls_bpf, frequency_drop/quota/window_drop
via the eBPF classifier — neither is relay-only any more).
The dynamips doc gets the full suite: anatomy (anchors reached by the
hypervisors' TAP fds, uBridge's AF_PACKET-tap / tc-qdisc sidebands), a
one-way copy-numbered relay comparison (six kernel/user crossings per
way against the kernel datapath's two TAP-fd legs), a staged walkthrough
(reserved-range diamond, egress clsact chain, capture/markers observation
sideband, copies 1-2) and a control sequence of real commands in real
order (hypervisor fd first, capture before markers, carrier last; router
handlers, not computes, as the per-node participants).
The four sibling frame paths are unified on the same form: netem qdisc
node labels, one capture/markers node outside the stages with AF_PACKET
tap edges anchored at ingress and post-chain egress, copy numbering
wherever the trace crosses the kernel/user boundary (qemu's two TAP-fd
legs, iol-docker's six socket/TAP legs), stage subgraphs with role-noted
titles everywhere (iol-docker restructured from its flat chain, iou's
chA/tapA variants folded into the canonical shapes).
The documentation skill records the rules as v3.1.0: four diagram kinds
(anatomy / comparison / control sequence / walkthrough), one step per
node, dashed sidebands per role with meaningful direction, numbered
crossings when copies are the point, glossary before the first diagram.
Sequence diagram: drop the phantom "port fwd_mask 0xfffd" step — uBridge
sets the per-port mask inside brctl addif and gns3-server never sends it.
Frame path: explain the ingress/egress mechanism in prose (the TAP-fd
write lands at the sender's anchor ingress, where nothing is installed;
the bridge's forward out the far anchor is the egress transmission the
tc chain sits on), and group the netio mailboxes into a kernel subgraph
matching the relay diagram. Suspend: a down anchor answers EIO on the
TAP fd in both directions (the section B hardening), not reads-block.
Term table: the anchor shape is per Ethernet bay (4 units per bay), not
the default 2x4. Link operations: carrier up is applied last, matching
_attach_kernel_nio. Link-local: note the verification matrix runs on the
Docker datapath. Reflow paragraphs to the soft-wrap convention.
Term table, QEMU↔QEMU topology with role-suffixed anchors and
frame-copies/qdisc-install dashed edges, relay copy-count diagram,
boxed frame path, link creation sequence; prose reflowed one
paragraph per line.
Replace the ASCII overview with mermaid diagrams per the documentation
standard: a glossary table first, then the two-container architecture
(port bridges inside their uBridge process blocks, the per-link Linux
bridge circled, dashed sidebands split by direction — frame copies flow
to uBridge, qdisc installs to the anchor), a copy-counted relay diagram
(six kernel/user crossings each way, datapath-invariant — swapping
datapaths only changes the middle two), the frame path with the
reserved-range diamond, and the link creation sequence. Details moved
from the old bullets into per-diagram prose; paragraphs reflowed to one
per line.
Replace the ASCII art with a mermaid suite (steady-state topology with
namespace boxes and sideband edge semantics, one-way frame path with
per-object boxes, link-creation sequence, full relay path with the four
kernel/user copies), add a term glossary ahead of the diagrams, reflow the
prose to soft-wrap, and refresh the stale parts: the configuration section
now describes enable_kernel_datapath as the global switch it already is,
with every anchorable node type landed and linked to its feature doc.
A Linux bridge standing in for a cable does not forward the IEEE 802.1D
reserved range (01:80:c2:00:00:00-0f) by default, so LACP/LLDP/802.1X never
crossed a kernel-datapath link while the UDP relay carried them fine. uBridge
now opens every enslave'd port's IFLA_BRPORT_GROUP_FWD_MASK to 0xfffd at
brctl addif time (branch feature/bridge-tap-l2only, commits 0284500/03a18f1/
cf4e494), which the six kernel-datapath feature docs now document as a
section: STP, LACP, 802.1X and LLDP/DCBX cross; 802.3x PAUSE / PFC stays
dropped — br_handle_frame()'s case 0x01 discards it unconditionally, no mask
can enable it, and PFC labs must use a relay link. The full contract, kernel
references and requirements live in the new design spec, marked landed.
The relay engines' observable paths had no live coverage: every
capture/start call in the suite sat on a kernel anchor (AF_PACKET
capture start_kernel) and markers were only asserted on marker
add_kernel. The two relay engines' own command shapes were unit-tested
only:
* bridge engine - bridge start_capture / bridge add_packet_filter ... mark
* iol_bridge engine - iol_bridge start_capture / iol_bridge add_packet_filter ... mark
Docker relay control (bridge engine, 1 passed 33s): the capture start
REST call writes the ICMP exchange to a pcap off the relay bridge; a
mark filter attached with capture_node_id n1 signals exactly 10
marker.match events over the dedicated marker WS for 5 pings (5
requests + 5 replies, both directions), carrying node_id/link_id/tag and
the busybox frame length, writes exactly those 10 frames to the marker
pcap, and deleting the marker takes the pcap with it.
IOU relay control (iol_bridge engine, 1 passed 173s): the same two
checks on iol_bridge start_capture and the port's mark filter, driven
by the real IOS ping through the fabric socket relay.
Not duplicated in the ESW/VPCS/cloud relay controls: they issue the same
uBridge bridge commands through their own call sites, which stay covered
by unit tests; the IOL container relay goes through the same iol_bridge
engine already exercised here.
The L2-anchor spec's behavioral guarantee — an idle link stays silent for
5 s once the enslavement burst settles — only ran in the Dynamips and
Docker suites. It now also runs in QEMU, IOU, the IOL container, the
switch fast path and the switch cascade, which covers every host-device
creation path:
* tap create (Dynamips/QEMU/IOU anchors), docker create_veth (Docker host
ends; the cascade's race-loser end is hardened after create by the
server), bridge add_nio_tap (the IOL container anchor, the fifth
creator), and the brctl switch bridges, whose multicast snooping is off
precisely so the bridge role can reach silence;
* each scenario silences its guests first (IOS interfaces shut through
the console; Docker guests' links down) and re-verifies the ping after;
* the cascade measures 8 devices at once (four absorbed router anchors,
both cascade ends, both switch bridges).
IOL 158 s, cascade 293 s, ESW fast path 378 s, QEMU 428 s, IOU 308 s —
all green.
The filter matrix (cls_bpf, the eBPF classifier) only ran on Docker veth
host ends; the IOU and IOL-container scenarios now spot-check it on their
TAP anchors and prove a filter survives a node restart:
* bpf "icmp" drops everything with clsact on the anchor, both ends, and
clears back to an empty qdisc;
* frequency_drop 3 (the eBPF every-nth mode) measured 56 % round-trip
loss (1 - (2/3)^2 = 55.6 % expected) in both scenarios;
* the IOU test applies delay 100 before a router stop/start: the fresh
anchor gets netem back (tc level) and the restarted router's pings
measure RTT >= 150 ms until the filter is cleared (a re-attach that
dropped the filter would show ~100 ms).
IOU file 492 s, IOL file 182 s, both green.
Extend the docker and IOU relay negative controls with the filters the
relay itself supports (delay, frequency_drop, bpf), asserted through real
ICMP — the first live coverage of the relay filter path:
* bridge engine (docker): delay 100 grows the RTT by ~200 ms and comes
back on clear; frequency_drop 2 kills the round trips (the filter counts
every packet crossing the bridge, both directions sharing one counter,
so on an alternating ping stream one whole direction dies; verified
against uBridge directly), frequency_drop -1 blackholes, and bpf drops
by libpcap expression ('icmp', then 'greater 150' discriminating by
frame size).
* iol_bridge engine (IOU): the same delay and frequency_drop effects
through the port's iol_bridge filter list, plus the available_filters
spot check.
Docker 33 s, IOU 168 s, both green.
Two Alpine containers on a kernel-only link with traffic-insight markers
attached to a veth anchor (marker add_kernel), observed on the dedicated
marker WebSocket via a new harness piece (WebSocketCollector, a sync
facade over websockets' sync client). Real ICMP proves: the marker.match
identity is complete (filter/tag/link_id/capture node/ts/len and the
tx/rx direction set of a ping), the main project channel carries no
matches, the BPF and direction are enforced (a non-matching sibling and
a tx-only marker each see exactly their traffic), the host pcap holds
exactly the matched frames, pausing silences one marker while a sibling
keeps counting (silence is never a dead channel), tag replay serves the
paused tag's timeline with display-filter/link/window narrowing and
refuses with 409 while capturing, and deleting a marker takes its pcap
and signals with it.
Two switches cascaded through the link-owned veth pair, two c7200
routers per switch, real ICMP crossing both bridges: the pair is born
UP with exactly one end per switch bridge, a dot1q trunk cascade
carries two VLANs at once, the symmetric access/trunk mode flip on the
cascade ends isolates and revives a VLAN in place (asymmetric flips are
semantically broken — the access side egresses untagged into the far
trunk's native PVID), a delay filter lands as netem on both ends (one
per direction, RTT ~= 2x delay), suspend admin-downs both ends, and a
link delete/re-create retires the whole pair and mints a new one under
the new link id — zero host residue after the project goes.
The Verified section only described the /tmp driver with a fake IOU
image; the repository suite now carries the same scenario with real
routers on a real L3 image.
The feature doc's Verification section now points at
tests/e2e/test_docker_kernel_datapath.py and what the suite pins down
(anchors/bridges, the filter matrix through real traffic, suspend,
capture, the stop/start and delete/re-create beats, the relay control and
the relay->kernel reopen upgrade), including the digest-pinned image the
scenarios share.
The spec's B table grows its fifth row: bridge add_nio_tap mints a
transient TAP when the name is free and hardens it at creation, while an
attach to an existing device (possibly a user-owned TAP with addresses)
is left untouched. The rule paragraph spells out attach-is-not-a-creation.
Delivery: uBridge fb72758, closing the gap the new e2e assertions found
(live: the switch's gns3*-0 relay TAP carried fe80::). The server issues
no link l2only of its own -- the switch call site now documents that the
TAP is minted-and-hardened by add_nio_tap, and the ESW feature doc
describes the uBridge-side close instead of a server workaround.
The feature doc covers the architecture (one irreducible userspace hop
on the unix-socket leg; the link segment kernelized through the port
bridge's swappable TAP leg), the capability gate, the anchor lifecycle,
the swap flow, the inherited capture/markers/filters, the relay
fallback and the live verification — including the ensure-then-add
lesson (tap create is strictly create-only).
The other datapath docs' roadmaps catch up: the IOL runner container
has landed (and the ethernet-switch doc's stale 'cascade deferred'
exclusion from d2f8aba58 is corrected while there); the docker doc's
adapter table no longer rejects IOLDockerVM.
QEMU is the guest side of every adapter TAP, and uBridge's tap delete
refuses a device another process holds open ('Device or resource busy') —
a best-effort delete whose failure is suppressed. QEMU's stop did the
delete first and stopped the process after, so every node stop or project
delete leaked the persistent TAPs: found by inspecting the host after a
project delete (16 taps left behind, carrier down, no holder), and the
held-tap behaviour reproduced against uBridge directly.
Stop the QEMU process first, then remove the TAPs and the per-link kernel
bridges — process side first, then the devices, the same lesson as IOU's
reverse stop order. Verified live: node stop and project delete now leave
no taps and no processes behind, while a restart recreates the TAPs.
The adapter anchor naming also goes through the shared
utils.kernel_anchor helper, like the other node types.
The Ethernet switch's kernel fast path: instead of a link crossing the
switch's per-port uBridge relay, the switch absorbs the peer's anchor into
its own kernel bridge. The peer's interface (Docker veth host end, QEMU
TAP, IOU port TAP, Dynamips port TAP) becomes the switch port, and frames
cross anchor -> switch bridge -> switch ports entirely in the kernel.
Wire format: the switch end receives nio_anchor ('join this interface to
my own bridge' — the mirror of nio_bridge), the peer end nio_bridge with a
null bridge ('your anchor is bridged elsewhere': carrier, markers and
impairments only — never bridge membership, and never a bridge delete, on
an interface this node does not own). Anchor names come from the shared
naming contract (utils.kernel_anchor), a pure function of node type, id,
adapter and port — the controller computes the peer's anchor while the
node may be stopped, which is what makes links to stopped nodes wireable.
Stopped peers defer the join; after any node of a link starts
(UDPLink.node_started) the controller re-pushes the switch NIO, and the
switch completes — or repairs, when a peer restart replaced its interface
under the same name — the join against the kernel's actual membership
instead of assuming it.
VLAN changes now edit membership in place: only the ports whose mode/VLAN
actually changed are touched, each losing exactly the previous mode's
entries the new mode does not want (the dot1q admits-all entry is one
range operation in both directions), and no port ever leaves the bridge to
change its VLANs. This replaces the old 'reset by re-enslaving' shortcut
(delif+addif on every wired port, added when the ports were the switch's
own TAPs): on absorbed anchors, re-enslaving an interface owned by another
node is where live testing watched links die — anchors occasionally lost
their bridge membership right after a ports_mapping update (later updates
then failed with delif EINVAL), reproducibly with qemu, IOU and Dynamips
peers and never with Docker's veths.
Eligibility: the switch side needs no capability of its own (its bridge
exists by construction); the peer must be able to anchor. Switch-to-switch
cascades stay on the relay — a kernel interface belongs to one bridge, and
joining two bridges needs a veth pair, deferred.
Verified live on a real server: qemu peers through four VLAN cycles with
both anchors enslaved throughout; a mixed switch (2 IOU + 2 Dynamips, four
kernel ports); Docker peers with real ICMP isolating and recovering (with
both anchors still enslaved); VPCS relay ports likewise in place.