test(e2e): relay capture and markers on both relay engines

The relay engines' observable paths had no live coverage: every
capture/start call in the suite sat on a kernel anchor (AF_PACKET
capture start_kernel) and markers were only asserted on marker
add_kernel. The two relay engines' own command shapes were unit-tested
only:
  * bridge engine  - bridge start_capture / bridge add_packet_filter ... mark
  * iol_bridge engine - iol_bridge start_capture / iol_bridge add_packet_filter ... mark

Docker relay control (bridge engine, 1 passed 33s): the capture start
REST call writes the ICMP exchange to a pcap off the relay bridge; a
mark filter attached with capture_node_id n1 signals exactly 10
marker.match events over the dedicated marker WS for 5 pings (5
requests + 5 replies, both directions), carrying node_id/link_id/tag and
the busybox frame length, writes exactly those 10 frames to the marker
pcap, and deleting the marker takes the pcap with it.

IOU relay control (iol_bridge engine, 1 passed 173s): the same two
checks on iol_bridge start_capture and the port's mark filter, driven
by the real IOS ping through the fabric socket relay.

Not duplicated in the ESW/VPCS/cloud relay controls: they issue the same
uBridge bridge commands through their own call sites, which stay covered
by unit tests; the IOL container relay goes through the same iol_bridge
engine already exercised here.

docs/features/docker-kernel-datapath.md

@@ -466,9 +466,10 @@ silence with the guests' links down), the filter matrix one type at a time
 (netem core and extensions, cls_bpf match-drop, the eBPF classifier, each
 asserted through real traffic), suspend, capture, link delete/re-create,
 container stop/start, a relay negative control — whose own filters
-(delay/frequency_drop/bpf) are asserted to really shape the relay wire as
-uBridge userspace bridge filters — and the relay→kernel upgrade on project
-reopen (the datapath choice is flipped across a server restart). The container image is digest-pinned and pulled through the
+(delay/frequency_drop/bpf) really shape the relay wire, whose capture and
+markers ride the same relay engine (`bridge start_capture`, the bridge
+`mark` filter) — and the relay→kernel upgrade on project reopen (the
+datapath choice is flipped across a server restart). The container image is digest-pinned and pulled through the
 server's own pull route on a cache miss, so collaborators test the same
 bytes.
 

docs/features/iou-kernel-datapath.md

@@ -153,7 +153,8 @@ bridge, netem delay, the TAP-anchor classifier spot check
 stop/start rewiring with a delay filter restored on the fresh anchor
 (tc and traffic both), the serial link staying on the relay, and a relay
 negative control whose real `delay`/`frequency_drop` filters ride the
-port's `iol_bridge add_packet_filter` list.
+port's `iol_bridge add_packet_filter` list — capture and markers on the
+same engine (`iol_bridge start_capture` / the port's `mark` filter).
 
 The uBridge side passed its own 30/30 (`tests/iol/test_tap_anchor.py` in the
 fork), including the DOWN-anchor resilience and kernel-bridge interop this

tests/e2e/README.md

@@ -83,7 +83,8 @@ Current scenarios:
   spot check and a filter surviving a node restart, a serial link staying
   on the relay (serial bays never anchor), and a relay negative control
   whose filters ride the IOU-specific `iol_bridge add_packet_filter` engine
-  (delay and frequency_drop measured through real traffic).
+  (delay and frequency_drop measured through real traffic, capture and
+  markers on `iol_bridge start_capture` and the port's `mark` filter).
 * `test_qemu_kernel_datapath.py` — two real IOSv routers (a real qcow2
   image from the images directory's `QEMU` subfolder, linked clone, the
   real IOS CLI on the server's telnet console): the anchor TAP is QEMU's
@@ -97,8 +98,10 @@ Current scenarios:
   (netem core and extensions, cls_bpf match-drop, the eBPF classifier —
   each measured through real traffic), plus a relay negative control whose
   own filters (delay, frequency_drop, bpf) really shape the wire as uBridge
-  userspace bridge filters, and a relay→kernel reopen upgrade (the server's
-  datapath choice is flipped across a restart).
+  userspace bridge filters and whose capture and markers ride the relay
+  engine (`bridge start_capture` / the bridge `mark` filter), and a
+  relay→kernel reopen upgrade (the server's datapath choice is flipped
+  across a restart).
 * `test_marker_kernel_datapath.py` — two real Alpine containers on a
   kernel-only link with traffic-insight markers attached to a veth anchor
   (`marker add_kernel`): `marker.match` signals collected off the dedicated

tests/e2e/test_docker_kernel_datapath.py

@@ -39,7 +39,9 @@ server itself uses. The scenarios cover:
   uBridge relay, the kernel-only filter types are hidden — and the relay's
   own filter set (delay, frequency_drop, bpf) really shapes the wire, as
   uBridge userspace filters on the filter node's bridge
-  (``bridge add_packet_filter``).
+  (``bridge add_packet_filter``); its capture (``bridge start_capture``)
+  and its markers (``bridge add_packet_filter … mark``) ride the same
+  relay engine, not the kernel datapath's AF_PACKET modules.
 * ``test_docker_relay_to_kernel_reopen_upgrade`` — the server's datapath
   choice is flipped (relay → kernel) across a restart and the project is
   reopened: the link is rebuilt on the kernel datapath and forwards.
@@ -114,6 +116,10 @@ def _filter_types(compute, pid, lid):
     return {entry["type"] for entry in compute.call("GET", f"/projects/{pid}/links/{lid}/available_filters")}
 
 
+def _matches(marker_ws, marker_name):
+    return [e for e in marker_ws.events("marker.match") if e["filter"] == marker_name]
+
+
 def _silence_guest(daemon, container_id):
     """Silence the guest for the idle-silence measurement — the docker
     analogue of shutting the IOS interfaces in the dynamips scenario. The
@@ -448,7 +454,8 @@ def test_docker_kernel_filter_matrix():
 def test_docker_relay_control():
     """Negative control: the same topology on an isolated relay-configured
     instance wires nothing into the kernel — no per-link bridge, no
-    enslavement — and still pings over the uBridge UDP relay."""
+    enslavement — and still pings over the uBridge UDP relay, its filters,
+    capture and markers all riding the relay engine."""
     server = harness.live_server(kernel=False)
     compute = server.compute
     image = harness.ensure_docker_image(server)
@@ -527,6 +534,58 @@ def test_docker_relay_control():
         assert big["loss"] == 100, big["raw"]
         _put_filters(compute, pid, link["link_id"], {})
         assert harness.docker_ping(daemon, n1["container_id"], R2_IP, count=5)["loss"] == 0
+
+        # capture on the relay: the same REST call as the kernel datapath's,
+        # hosted by the relay node's uBridge bridge (bridge start_capture)
+        # instead of AF_PACKET on an anchor
+        print(".. relay capture: bridge start_capture writes the ICMP exchange")
+        capture = compute.call(
+            "POST", f"/projects/{pid}/links/{link['link_id']}/capture/start", {"data_link_type": "DLT_EN10MB"}
+        )
+        harness.docker_ping(daemon, n1["container_id"], R2_IP)
+        time.sleep(1)
+        compute.call("POST", f"/projects/{pid}/links/{link['link_id']}/capture/stop")
+        path = capture["capture_file_path"]
+        count, ethertypes = harness.pcap_records(path)
+        assert count >= 4, f"{path}: {count} records"
+        assert "0800" in ethertypes, (path, ethertypes)
+
+        # markers on the relay are uBridge's `mark` packet filter on the
+        # relay bridge (bridge add_packet_filter ... mark), signalled over
+        # the same dedicated marker WS — the relay engine's observability
+        # path (the kernel datapath's marker add_kernel is the other one)
+        print(".. relay marker: 5 ICMP echoes through a mark filter on the relay bridge")
+        marker_ws = harness.WebSocketCollector(server, f"/projects/{pid}/notifications/markers/ws")
+        try:
+            m = compute.call(
+                "POST",
+                f"/projects/{pid}/links/{link['link_id']}/markers",
+                {"name": "m-icmp", "bpf": "icmp", "tag": 4242, "capture_node_id": n1_id},
+            )
+            assert m["capture_node_id"] == n1_id and m["enabled"] is True, m
+            result = harness.docker_ping(daemon, n1["container_id"], R2_IP, count=5)
+            assert result["loss"] == 0, result["raw"]
+            harness.wait_until(lambda: len(_matches(marker_ws, "m-icmp")) >= 10, timeout=10)
+            icmp = _matches(marker_ws, "m-icmp")
+            assert len(icmp) == 10, len(icmp)  # 5 requests + 5 replies
+            assert {e["dir"] for e in icmp} == {"tx", "rx"}, [e["dir"] for e in icmp]
+            for event in icmp:
+                assert event["node_id"] == n1_id, event
+                assert event["link_id"] == link["link_id"], event
+                assert event["tag"] == 4242, event
+                assert event["len"] == 98, event  # busybox ping frame
+            # the marker's pcap on the host holds exactly the matched frames
+            markers_dir = os.path.join(compute.call("GET", f"/projects/{pid}")["path"], "project-files", "markers")
+            pcap = os.path.join(markers_dir, f"{n1_id}_{link['link_id']}_m-icmp.pcap")
+            assert os.path.exists(pcap), os.listdir(markers_dir)
+            count, ethertypes = harness.pcap_records(pcap)
+            assert count == 10, count
+            assert set(ethertypes) == {"0800"}, ethertypes
+            # deleting the marker removes its pcap with it
+            compute.call("DELETE", f"/projects/{pid}/links/{link['link_id']}/markers/m-icmp")
+            assert harness.wait_until(lambda: not os.path.exists(pcap), timeout=10), pcap
+        finally:
+            marker_ws.close()
     except BaseException:
         harness.release(server, pid, failed=True)
         raise

tests/e2e/test_iou_kernel_datapath.py

@@ -41,8 +41,11 @@ relay-configured instance: the same topology rides the fabric socket ↔ UDP
 relay (no per-link bridge, nothing enslaved) and still pings — with its
 filters applied by the IOU-specific engine
 (``iol_bridge add_packet_filter`` on the IOL port) really shaping the wire.
+Its capture and markers ride the same engine
+(``iol_bridge start_capture`` / ``iol_bridge add_packet_filter … mark``).
 """
 
+import os
 import re
 import time
 
@@ -82,6 +85,10 @@ def _boot(server, project_id, node_id, timeout=BOOT_TIMEOUT):
     return console
 
 
+def _matches(marker_ws, marker_name):
+    return [e for e in marker_ws.events("marker.match") if e["filter"] == marker_name]
+
+
 def _eth_shutdown(console, shut):
     """Shut/unshut the guest's Ethernet0/0 — the only legitimate speaker on
     the link segment (the IOL fabric's own chatter) — for §E.2's silence
@@ -382,6 +389,52 @@ def test_iou_relay_control():
         recovered = harness.wait_ping(c1, "10.1.1.2")
         assert recovered["success"] == 100, recovered["raw"]
 
+        # capture on the relay: iol_bridge start_capture on the port's IOL
+        # location (the IOU engine's own capture path, distinct from the
+        # kernel datapath's capture start_kernel on the anchor)
+        print(".. iol_bridge relay capture: start_capture writes the ICMP exchange")
+        capture = compute.call("POST", f"/projects/{pid}/links/{lid}/capture/start", {"data_link_type": "DLT_EN10MB"})
+        harness.ping(c1, "10.1.1.2", repeat=5)
+        time.sleep(1)
+        compute.call("POST", f"/projects/{pid}/links/{lid}/capture/stop")
+        path = capture["capture_file_path"]
+        count, ethertypes = harness.pcap_records(path)
+        assert count >= 4, f"{path}: {count} records"
+        assert "0800" in ethertypes, (path, ethertypes)
+
+        # markers on the relay: iol_bridge add_packet_filter ... mark on the
+        # port's IOL location, signalled over the dedicated marker WS
+        print(".. iol_bridge relay marker: 5 ICMP echoes through a mark filter")
+        marker_ws = harness.WebSocketCollector(server, f"/projects/{pid}/notifications/markers/ws")
+        try:
+            m = compute.call(
+                "POST",
+                f"/projects/{pid}/links/{lid}/markers",
+                {"name": "m-icmp", "bpf": "icmp", "tag": 4242, "capture_node_id": n1_id},
+            )
+            assert m["capture_node_id"] == n1_id and m["enabled"] is True, m
+            result = harness.ping(c1, "10.1.1.2", repeat=5)
+            assert result["success"] == 100, result["raw"]
+            harness.wait_until(lambda: len(_matches(marker_ws, "m-icmp")) >= 10, timeout=10)
+            icmp = _matches(marker_ws, "m-icmp")
+            assert len(icmp) == 10, len(icmp)  # 5 requests + 5 replies
+            assert {e["dir"] for e in icmp} == {"tx", "rx"}, [e["dir"] for e in icmp]
+            for event in icmp:
+                assert event["node_id"] == n1_id, event
+                assert event["link_id"] == lid, event
+                assert event["tag"] == 4242, event
+            markers_dir = os.path.join(compute.call("GET", f"/projects/{pid}")["path"], "project-files", "markers")
+            pcap = os.path.join(markers_dir, f"{n1_id}_{lid}_m-icmp.pcap")
+            assert os.path.exists(pcap), os.listdir(markers_dir)
+            count, ethertypes = harness.pcap_records(pcap)
+            assert count == 10, count
+            assert set(ethertypes) == {"0800"}, ethertypes
+            # deleting the marker removes its pcap with it
+            compute.call("DELETE", f"/projects/{pid}/links/{lid}/markers/m-icmp")
+            assert harness.wait_until(lambda: not os.path.exists(pcap), timeout=10), pcap
+        finally:
+            marker_ws.close()
+
         c1.close()
         c2.close()
     except BaseException: