XGS 3300 - what has been measured on it

Numbers taken on one appliance, with how each was taken and what it does not mean. They are here rather than in octeon-tx-reference.md because a measurement belongs to a board, not to a family: a second XGS 3300 with a different kernel or a different cable would produce different figures, and the reference page should carry conclusions rather than readings.

Nothing here is a projection. Every line was read off this machine.

The machine

appliance	Sophos XGS 3300, assembly AMDA0202-0004, platform xgs1us
host	AMD Ryzen Embedded V1780B
coprocessor	Cavium OCTEON TX CN8365, PCI 177d:a300 at pci0:1:0:0
operating system	OPNsense 26.7 / FreeBSD 15.1-RELEASE-p1 stable/26.7-n283674-12334a596709 SMP until 2026-10-03; OPNsense 26.7.5 / FreeBSD 15.1-RELEASE-p3 stable/26.7-n283949-083dc7025377 SMP since after it (the update is below; the previous kernel stays on the disk as kernel.old with a module of its own); OPNsense 26.7.7 / FreeBSD 15.1-RELEASE-p4 stable/26.7-n284159-abbbe67a45e7 SMP, read 2026-10-17 - the README's badges carried 26.7.5, p3 and n283949 until that day and carry these since
driver	octep, built on the appliance against that kernel's own headers
front ports	12, all presented as oxp0..oxp11

Round trip and loss

Method. ping -f from the appliance to the router on the other end of a copper panel port's cable, 30,000 full-size frames, interface counters read either side.

elapsed 6 s
in   30022 pkts   1512 bytes/pkt    5003 pps    60 Mbit/s
out  30003 pkts   1513 bytes/pkt    5000 pps    60 Mbit/s
0.0% packet loss
round-trip  min 0.091   avg 0.193   max 0.508 ms

What it is. A latency figure and a loss figure. Nothing dropped in 30,000 full-size frames.

What it is not. A throughput figure. ping -f sends the next probe when the last reply arrives, so it is bound by the round trip: 5,000 packets a second against a 0.193 ms round trip is the same number written twice.

The transmit ceiling

Method. A UDP sender that never waits (blast.c, below), aimed at a panel port with no cable in it so the frames cross the whole host-to-coprocessor path and die at a dark switch port without touching anything anyone else is using. Five seconds, 1472-byte payloads. The coprocessor's own named counters were read either side of the run over the rpc facility.

offered and accepted by the driver     2,628,609     525,722 pps
FPCNTR_RX_KN                          +2,627,710     the coprocessor took them
FPCNTR_FROM_KN_TO_WIRE                +2,628,609     and chose the wire for every one
FPCNTR_TX_DROP_QUEUE_FULL             +2,210,885     84% refused at its egress queue
to the wire                              416,825      83,365 pps = 1,023 Mbit/s

The account closes. 1,023 Mbit/s is line rate for a gigabit port at 1534 bytes a slot with the preamble and the interframe gap, so everything the driver posted either left at line rate or was dropped by the coprocessor's egress queue - which is the correct thing for it to do.

So the number that matters is what the driver accepts:

	packets a second
the driver accepts	525,722	measured
a 1G port needs	81,486	6.5x headroom
a 10G port needs	814,863	65% of line rate

For the eight copper panel ports and the two 1G cages this driver is nowhere near the bottleneck. Only the two 10G SFP+ cages are capped.

Read that table one port at a time. It says what the driver accepts for a single interface, and that is what it was measured as. The appliance's total is a different number and a smaller one - see "the aggregate ceiling" below, which was measured afterwards and is one gigabit for all twelve ports together.

Removing a redundant write

Method. The same generator and the same dark port, before and after the change in #173, which stopped the transmit path zeroing the frame it was about to copy over.

payload	before	after
1472 bytes	525,722 pps	535,142 pps	+1.8%
64 bytes	595,558 pps	607,232 pps	+2.0%

What it tells us beyond the gain. 525,722 packets a second is 1.9 microseconds each, and a redundant 1,514-byte write is not what that is made of. The next suspect is the one MMIO write per packet - the input doorbell.

What load on one front port costs the other eleven

Method. tools/octep-fairness.sh. The generator is paced now - blast takes a packets-per-second argument and sends in bursts of about a millisecond - so the load is a known rate rather than whatever one core can manage. Each step offers six seconds of traffic at one rate on a dark front port while an ordinary ping runs underneath it on the port that carries the appliance's WAN, and the coprocessor's own named counters are read either side of every step.

with no load at all: 0.0% loss, 0.177 ms average, 0.209 ms worst

target_pps  offered_pps  wire_Mbit   wan_loss  wan_avg wan_worst  drop_queue_full
      5000         5000         61      4.2%    0.168    0.228                 0
     10000        10000        123      0.0%    0.169    0.203                 0
     25000        25000        307      8.3%    0.170    0.207                 0
     50000        50000        614     29.2%    0.173    0.225                 0
    100000       100000       1227     31.2%    0.173    0.250           100,229
    200000       200000       2454     33.3%    0.171    0.187           696,534
    400000       400000       4909     72.9%    0.183    0.228         1,904,034
   unpaced       535176       6568     60.4%    0.173    0.193         2,711,514

with no load again:  0.0% loss, 0.178 ms average, 0.225 ms worst

What it says. The twelve interfaces are not independent. A flood aimed at one of them costs the others their packets, and it starts long before anything is reported as dropped: at 50,000 packets a second the WAN loses 29% of its echoes while FPCNTR_TX_DROP_QUEUE_FULL has not moved at all. Below 10,000 packets a second the loss is indistinguishable from this appliance's ordinary background, which is why the soak script defaults to that rate.

And the latency never moves. 0.17 ms average at every rate, 0.25 ms at worst, the same as idle. A frame for another port is sent at once or thrown away; it is never delayed. Whatever is full is not a queue deep enough to wait in.

Where those packets are lost, counter by counter

Method. 200 echo requests at 50 a second, under three conditions, with five instruments read either side: the interface's own transmit count, the coprocessor's FPCNTR_RX_WIRE and FPCNTR_FROM_WIRE_TO_KN_FORCED, the driver's dp.rx_done, and the host's ICMP input histogram.

	requests posted	coprocessor took from the wire	forwarded to the host	driver delivered	echo replies at the stack	loss
no load	202	210	210	218	200	0.0%
50,000 pps elsewhere	202	161	161	153	126	37.0%
unpaced elsewhere	200	72	72	72	55	72.5%

Four things follow, and they narrow the fault to one place:

• The host posts every request in all three runs. Nothing is refused on this side, and net.inet.ip.intr_queue_drops stayed at zero throughout.
• The coprocessor forwards everything it receives. RX_WIRE and FROM_WIRE_TO_KN_FORCED are equal in every run, so no LIF gate and no TCAM entry is dropping anything.
• But it receives less and less from the wire as the load rises - 210, then 161, then 72. The replies are not coming back, because the requests are not getting out.
• A second, much smaller loss is inside the host: 161 forwarded against 153 delivered.

So the frames die after this host counted them as sent and before the gateway ever saw them, on a path every front port shares.

The suspect, found by reading the transmit path afterwards rather than by guessing: every interface copies its frame into one buffer - sc->dp_txbuf, a single page - posts an instruction carrying that buffer's physical address, rings the doorbell and releases the mutex. The coprocessor reads the memory itself, afterwards, on its own schedule. The next frame overwrites the previous one. At 1.9 microseconds a packet an echo request posted by one port is overwritten by another port's traffic before it is read, and what goes out in its place carries the other port's tag. That would produce exactly this: posted, counted, never seen, and no counter anywhere reporting a drop. It was built, and the next section is the same sweep again.

The same sweep, with a buffer per instruction slot

Method. Identical to the sweep above - same script, same rates, same appliance, same kernel - with one change in the driver: sc->dp_txbuf, one page shared by every interface, became sc->dp_txbufs, 256 buffers at the ring's own stride, and a frame is copied into the buffer belonging to the slot its instruction will occupy. Nothing else was touched. The appliance was rebooted because the datapath attaches once per coprocessor boot, and it came back with twelve interfaces, the WAN lease and the default route, unattended.

with no load at all: 0.0% loss, 0.178 ms average, 0.229 ms worst

target_pps  offered_pps  wire_Mbit   wan_loss  wan_avg wan_worst  drop_queue_full
      5000         5000         61      0.0%    0.169    0.210                 0
     10000        10000        123      0.0%    0.172    0.245                 0
     25000        25000        307      0.0%    0.170    0.219                 0
     50000        50000        614      0.0%    0.174    0.385                 0
    100000       100000       1227      0.0%    0.181    0.240           100,274
    200000       200000       2454      0.0%    0.175    0.255           698,090
    400000       400000       4909      0.0%    0.173    0.200         1,890,791
   unpaced       528885       6490      0.0%    0.173    0.203         2,660,027

with no load again:  0.0% loss, 0.180 ms average, 0.224 ms worst

offered on one port	WAN loss before	WAN loss after
5,000 pps	4.2%	0.0%
25,000 pps	8.3%	0.0%
50,000 pps	29.2%	0.0%
400,000 pps	72.9%	0.0%
unpaced, about 530,000 pps	60.4%	0.0%

Zero at every rate, including 6.5 Gbit/s offered at one port while another carries the appliance's internet. The twelve interfaces are independent now, and they were not before.

A second counter agrees without being asked. FPCNTR_TX_WIRE_ERR used to move whenever anything was loaded - 25 to 76 of them in a fifty-second window at a mere 10,000 packets a second. On the fixed driver it does not move at all, through 16 million frames in a thirty-second window at full rate. Frames that had been overwritten mid-flight were reaching the MAC malformed, and the far side was counting them; now there are none to count.

What this says about the earlier numbers. The egress ceiling is unchanged - drop_queue_full tracks the old run step for step - so the one-gigabit aggregate below stands exactly as measured. What changed is that reaching it no longer costs the other eleven ports their traffic.

The aggregate ceiling is one gigabit, not twelve

Method. Two dark front ports loaded at the same time, six seconds each, and the arithmetic of FPCNTR_FROM_KN_TO_WIRE minus FPCNTR_TX_DROP_QUEUE_FULL - what the fast path chose to send, less what it threw away for want of room. The soak was running concurrently at 10,000 packets a second on a third dark port, which is included in these figures.

offered	chosen for the wire	refused, queue full	reached the wire
one port at 50,000 pps	329,588	0	329,588	54,931 pps
two ports at 50,000 pps	664,182	159,759	504,423	84,070 pps = 1,023 Mbit/s
two ports at 200,000 pps	2,462,180	1,960,315	501,865	83,644 pps = 1,019 Mbit/s

One port at 614 Mbit/s passes untouched. Two ports at 614 Mbit/s each do not: a quarter of the frames are refused, and what gets through is 1,023 Mbit/s - the same figure the single-port ceiling run produced, and the same figure again when the offered load is quadrupled. One gigabit is not a per-port limit on this appliance. It is the whole datapath.

It is a bit rate, not a packet rate, which is worth proving because the two look identical from one frame size:

payload	ports	reached the wire
1472 bytes	two at 200,000 pps	82,582 pps	1,013 Mbit/s
64 bytes	two at 200,000 pps	405,964 pps	422 Mbit/s, nothing refused
64 bytes	one, unpaced	591,380 pps	615 Mbit/s
64 bytes	two, unpaced	1,053,385 pps	1,096 Mbit/s

A million small frames a second go through the stage that caps full-size frames at 84,000. In bytes the two agree: 129 MB/s and 137 MB/s. So the shared stage drains about a gigabit of frame bytes a second whatever the frames are.

What it is not. It is not a limit of this driver: the coprocessor accepted 6.5 Gbit/s over PCIe in the same runs - FPCNTR_RX_KN counted every frame - and put a gigabit of it on the wire. It is not a measurement of twelve ports doing useful work either; every frame here died at a dark port, and a cabled port has a link partner and flow control that a dark one does not.

And it is not the switch uplink. That was the first suspect and it is now ruled out by reading the switch itself. xgs-mdio -a 0 2 <port>.0 on the coprocessor returns each port's status register, and the uplink is the one that differs:

switch port	status	link	speed field
0, the uplink to the coprocessor	0x0f4d	up	3
two cabled copper ports	0x0e0f	up	2, which is 1000 Mbit/s
the dark copper ports	0x000f	down	-
the two 1G cages	0x0249	down	2

The only difference between the uplink and a gigabit port is one bit: speed 3 rather than 2, which on this family means above 1000 Mbit/s. The uplink also carries a different connection mode - 13 against 15 - which is what a 10G mode looks like beside a 1000BASE-X one.

So the gigabit is inside the coprocessor's fast path, not on the wire between it and the switch. The remaining candidate is a shaper: FPCNTR_TX_DROP_SHAPER_INV exists, and the rpc enumeration has QOS_SET_SHAPER_RATE at 23 and QOS_GET_SHAPER_RATE at 24. Neither request structure is in the module's DWARF, so reading the rate back is the next piece of archaeology rather than the next command.

Two counters that were wrong, and how the measuring found them

Received bytes were doubled. The first run charged an interface 3,026 bytes for every 1,514-byte frame. ether_input_internal() adds the received byte count itself unless the driver claims IFCAP_HWSTATS, which this one does not, so counting it in the driver as well doubled it. The packet count was right, because that file increments no IPACKETS. Fixed in #171; afterwards the two directions agree at 1512 and 1513 bytes a packet, which is what a ping flood should show.

Opkts is offered load, not what left the port. In the ceiling run it read 6,432 Mbit/s out of a gigabit port: the 84% the coprocessor dropped, counted as sent. The driver cannot see those drops per packet - they belong to the far side and only rpc reads them - so read Opkts as offered load and FPCNTR_TX_DROP_QUEUE_FULL as the truth about what the wire took.

The receive path, and why every number above is a transmit number

Everything on this page until now was measured by sending. A generator on a dark front port, thousands of packets a second, hours of soak, and the loss figures that came out of it are real. Not one of those measurements could ever have filled a receive ring, and a defect had been sitting in that half of the datapath the whole time.

What it looked like: the appliance held its lease, routed, resolved, and answered the first hop in 0.6 ms. traceroute ran clean to six hops. And a fetch of a 16.7 MB file returned 0 bytes in 25 s, every time, while three consecutive opnsense-update runs installed nothing at all.

Four measurements say where it is not:

the line	16.7 MB to a PC on the same ONT at 4,354 kB/s
the host stack and igb0	64 MB in 737 ms, 90,933 kB/s, with no octep in the path
the coprocessor's own ledger	six counters moved during a failing download and balanced exactly: RX_WIRE +180, TX_KN +180, FROM_WIRE_TO_KN_FORCED +180, and no drop counter moved
the front port	Ierrs 0 Idrop 0 Oerrs 0

So the wire was fine, the host was fine, and the coprocessor believed it had delivered everything.

The two numbers that had never been put side by side

The far side's count of what it handed over, against the host's count of what it took:

FPCNTR_TX_KN              +19      handed to the host
driver dp.rx_done          +0      taken by the host
all oxp interfaces Ipkts   +0

dp.state then showed real WAN frames sitting in host memory, addressed to the WAN port's own MAC, with OUT_CONTROL reading idle - the block had written them and finished. R_OUT_CNTS low half read 1,532: packets the host had never acknowledged. And all eight MSI-X vectors were frozen, their sum exactly equal to dp.intr_taken, so not one interrupt had fired since the stall began.

One dp.service=1 by hand delivered 1,550 frames in a single call. With a service loop running, the first hop answered 8/8 at 0.0% loss, and the interrupts resumed on their own.

The mechanism

octep_dp_intr took one pass at the ring. A pass reads R_OUT_CNTS, takes at most RSIZE packets, and acknowledges those. The block raises its interrupt when the count crosses the interrupt level - not while it sits above it. So a burst bigger than one pass leaves the count above the level with nothing left to re-cross it, and the ring is dead.

And octep_dp_service() had exactly two callers: that handler, and a sysctl written by hand. There was no periodic receive path, so a lost edge was not a hiccup - it was permanent, until a reboot.

That is the whole symptom set. A ping never fills a ring, so a ping always worked. A download fills one immediately, so a download never worked - and from that moment reception was dead for everything, which is why the ping that had just succeeded started failing too.

After the fix

The handler now drains until the ring reads empty, bounded at OCTEP_DP_OQ_DRAIN_ROUNDS, and a callout services every armed ring twenty times a second whatever the interrupts are doing. (Every servicer drains now and the bound is in packets; and that callout, whose service was one pass, turned out to be a way of leaving a ring with packets in it: a visit is not a pass.)

The same fetch, on the same appliance, over the same front port:

16,697,068 bytes in 29 s, fetch reporting 1,049 kB/s

and the instruments for that one download:

dp.intr_drained	1,824	handler entries that needed a second pass
dp.rxwd_runs	11	interrupts genuinely lost, and caught by the timer instead
R_OUT_CNTS low half	0	the ring fully drained

rxwd_runs is the number worth keeping an eye on. Eleven lost edges in one 16 MB download is not a rounding error, and before there was a timer under them, eleven was ten more than it took.

And rxwd_runs was read too strongly too. Eleven is how many times the timer found packets in a ring, and that is all it is. It finds some in a ring whose handler is on its way; and for as long as its own service was one pass it was itself what left a ring with packets in it, to be found on its next tick. No lost interrupt has been shown on this block. The number to keep an eye on is dp.rxwd_stalls: a visit is not a pass.

And intr_drained was read too strongly when it was first published here. This page said every one of those 1,824 second passes was a stall under the old code. It was not: a second pass that finds three more frames is an ordinary second pass, and a stall needs the count left above the interrupt level after a pass. What the counter measures is how often one handler entry needed more than one pass, which under load is most of them. The evidence for the stall is the measurement above - TX_KN +19 against dp.rx_done +0, 1,532 unacknowledged, eight vectors frozen - and the figure itself was also inflated by a second defect, fixed separately: a pass used to report the register's reading rather than what it had taken, so a round that delivered nothing still counted. (It counts the passes beyond the first of any servicer's hold of a ring now, not only the handler's.)

Then the thing the stall had been blocking for three attempts: pkg fetch -u took the cache from 40 packages to 336 in 52 seconds, having previously stalled at 41 over several minutes each time, and pkg upgrade installed all 84 - OPNsense 26.7 to 26.7.5 - with the datapath carrying it and all twelve front ports still up afterwards. That download alone needed 59,675 second passes.

And with the appliance warm, three 16.7 MB fetches back to back:

run 1: 16,697,068 bytes in 7 s = 2,038 kB/s
run 2: 16,697,068 bytes in 3 s = 4,076 kB/s
run 3: 16,697,068 bytes in 3 s = 4,076 kB/s

4,076 kB/s is the best of the three, against the 4,354 kB/s the same file reaches on a PC on the same ONT. That is 94% of it: the gap narrowed, it did not close. And the runs do not agree closely enough to place it even that precisely - the first of the three was 2,038 kB/s, 47% of the PC figure, and the fetch above them 1,049 kB/s, so one file spread 3,027 kB/s across four readings, eleven times the 278 kB/s between the best run and the PC.

The seconds printed beside those rates are not the rates' own timebase. 16,697,068 bytes at 4,076 kB/s is 4.00 s and at 2,038 kB/s is 8.00 s - both exact, with kB read as 1,024 bytes - against the 3 s and 7 s printed beside them, and the fetch above them reads 29 s where its rate implies 15.5 s. So the rates are averages over the transfer and the seconds are something coarser. No transcript was kept to settle which, so read the rates and treat the seconds as indicative.

What these runs establish is that a download now completes, repeatedly, at the same order as the line - #197 is gone. What they do not establish is a throughput figure for the datapath: 4,076 kB/s is 33 Mbit/s on a gigabit front port, so what bounds these runs is the ONT and the far server. Nor a clean receive path - the driver was still returning sixteen credits per packet where the block debits about one, and a twenty-download loop still stalled on the tenth until #205 held the credit to the grant.

A kernel update must not cost the front ports

This is the circle, and it closed on this appliance on 2026-10-01:

new kernel -> the module's stamp no longer matches -> bring-up refuses it -> no front ports
-> no WAN -> nothing can fetch the kernel sources that would rebuild the module

The refusal itself is right. A FreeBSD module declares its kernel dependency as a range, so within 15.x the loader accepts a module built against a different 15.x kernel rather than refusing it, and any structure that moved is then read at the wrong offset inside the kernel of a machine carrying traffic. There is no loud failure to wait for. So the module is stamped with the kernel it was built against, and a mismatch is refused.

And the refusal costs more than the link. OPNsense configures its interfaces from the devices that exist at that point in the boot, and the assignments whose devices are absent are dropped out of /conf/config.xml outright. One boot with the module refused removed wan, opt1, opt2 and opt3 from the configuration here - the WAN's own DHCP assignment with them - and getting the module right afterwards did not put them back. They were restored from /conf/backup/, which is the only reason this is a paragraph and not a reinstall.

The window

Between a kernel being installed and the host rebooting onto it, everything needed is present at the same time:

the old kernel is still running	so the module is loaded and the WAN is alive
the new kernel is already on disk	where strings -a /boot/kernel/kernel finds the source commit it was built from

That is the only moment a kernel change can be followed, and install/kernel-follow.sh is that moment made automatic. Where to call it from took two attempts, and the first would never have run.

The hook level, which was checked rather than assumed

rc.syshook.d/update looks like the obvious home: the directory exists and OPNsense ships a script in it. Nothing on this appliance passes update to rc.syshook. The callers pass facility, stop, start, monitor, upgrade, import, early and carp. A hook placed at that level is read by nobody - which is the most expensive kind of wrong, because it looks like a guarantee and it is a file.

upgrade is called, from scripts/firmware/upgrade.sh, and reading that script settles where in the sequence it sits:

opnsense-update -u                 the packages
rc.syshook upgrade                 <-- the hook
opnsense-update -K -c ; -K         the pending kernel is applied to /boot HERE
reboot

So the new kernel is not in /boot when the hook runs. It is a set waiting in /var/cache/opnsense-update/.sets.pending/ with a marker beside it, and once it is applied the reboot follows with no hook in between. Reading /boot/kernel/kernel is therefore too late at the one place anything calls us.

So the follower looks in both places, staged first:

a staged set, kernel-<release>-<arch>.txz	tar -xOf <set> boot/kernel/kernel \| strings reads its version without unpacking it; the pipe closes at the first match
/boot/kernel/kernel	the kernel the next boot will use, for when -K has already run

And a cron entry, which needs nobody's cooperation

Five minutes, from src/etc/cron.d/octep. When the stamp already matches - almost always - it is two file reads and an exit. It is there because a hook depends on somebody calling it and this does not, and because the update level proved that assumption worth paying for once. That FreeBSD's cron reads /usr/local/etc/cron.d on this appliance was also tested rather than assumed: a probe entry written at 19:06:30 fired at 19:08:00.

Then one reboot brings the kernel and the module up together and nothing is done by hand.

Run by hand it is a no-op whenever the stamp already matches, and DRY=1 says what it would do:

sh install/kernel-follow.sh
DRY=1 sh install/kernel-follow.sh

It fails towards the ports, not away from them

• the build happens before anything is replaced, and a build that fails leaves the installed module and its stamp exactly where they were
• the hook exits 0 on every path, because an update that stops half way is worse than a module that needs rebuilding by hand; what a failure costs is a loud warning in the log
• contrib/octep/build.sh takes KVER so the stamp records the kernel being built for rather than the one running, which is the whole point before a reboot

And following forwards must not mean falling off backwards

The new module is stamped for the kernel on disk, which is not the kernel running while it is built. If the host boots the old kernel again - a rollback, a boot menu choice, an update that did not take

• that stamp is wrong in the other direction and the ports would be refused for the opposite reason to the one they were being saved from.

So the module being replaced is kept beside it as octep.ko.prev with its own stamp, and the bring-up script loads that one when its stamp is an exact match for the running kernel. This is not FORCE: nothing is loaded on a guess, and a module that matches neither kernel is still refused.

Tested

Proving the refusal branch used to mean unloading the driver, which on this appliance takes the twelve interfaces, the WAN lease and the default route with it and needs a host reboot to undo. Two small changes make it reachable instead: bringup.sh takes CHECK=1, which stops after the module decision and says which one it would load, and the loader commands are variables so a test can report the module absent without it being absent.

All seven on the appliance, with the datapath carrying traffic throughout:

stamp already matches	no-op
DRY=1 on a mismatch	reports, changes nothing
mismatch, for real	the previous module kept with its stamp, the installed one stamped for the kernel on disk - and the loaded driver was never touched, WAN at 0.0% loss during
octep.ko wrong, octep.ko.prev an exact match	CHECK=1: would load /boot/modules/octep.ko.prev, exit 0
neither stamp matches	refused, exit 1
a staged kernel set present	version read out of the .txz, reported as read from: staged, waiting to be applied, and the decision correct with both a matching and a stale stamp
cron reads /usr/local/etc/cron.d	a probe entry fired on the next minute boundary

And that row said rebuilt more strongly than the run could show. Nothing in the follower as it was merged could tell a rebuild from a reinstall. It built in /root/npu/octep-sdp, a tree assembled by hand that persisted between builds, and the make clean meant to empty it was given no SYSDIR: with no /usr/src/sys to fall back on, bsd.sysdir.mk stops with Unable to locate the kernel source tree, and that error went to /dev/null. kmod.mk never re-points an existing machine, x86 or i386 link, and the generated bus_if.h, device_if.h and pci_if.h are remade only when their .m file is newer. So once the objects were newer than anything make could see, make would compile nothing and the module already in place would be installed and stamped for the new kernel - the silent wrong-offset load the stamp exists to prevent - and the only checks were that make exited 0 and that an octep.ko was there. #205 found exactly that on the appliance, by review: a module stamped p3 whose machine link pointed into the p1 tree.

And the one case the follower exists for was not among the seven. fetch-sources.sh printed its SYSDIR= line at column zero only on the path where the tree was already there; where it had just fetched one it printed it inside an indented hint, and the follower read it with an anchored sed. So a run whose sources had to be fetched could only end in could not get kernel sources, and the seven runs above all had the tree present already. The next day, on the one kernel it had to fetch for, it fetched 341 MB, verified the tree, reported that it could not get the sources and left the module unbuilt. Both ends were fixed in #205.

Stopping the receive path before an interface is freed

Found by review rather than by a crash, which is the only reason it is written down calmly.

A servicer reads dif->ifp, stores it in each mbuf's rcvif, and hands the chain to if_input(). Detaching an interface calls ether_ifdetach() and then if_free() on that same ifnet. Nothing connected the two, so a detach could free an ifnet while a servicer still held mbufs pointing at it - inside the kernel of a machine carrying traffic.

Two things made it reachable. octep_dp_stop() detached the interfaces before releasing the interrupt handlers and the watchdog, so both servicers were live for the whole of the detach. And if_input() was called after the ring's busy flag was released - deliberately, so the ring was free while the stack worked - which meant the busy flag could not be used to tell whether a servicer was still holding an ifnet.

Every path that reached it is deliberate: dp.stop, dp.if_del, a device detach, kldunload octep. Normal operation never goes near it. "You have to ask for it" is not the same as safe.

The quiesce

Two halves, and both are needed:

a servicer that has not started	takes the ring's busy flag, sees the quiesce, and gives the flag straight back. The check is after the acquire - before it, one could pass the check and take the flag after the wait had finished looking
a servicer already inside	is waited for, by reading the busy flags. For that wait to mean anything, if_input() moved back inside the flag

Moving if_input() costs nothing. The buffers are re-poisoned and the doorbell is credited before the frames go up, so the block has its credits back either way, and the only thing held out is another servicer of the same ring - which is the one thing that must be held out.

The watchdog is drained rather than flagged, and its handler declines to re-arm while a quiesce is set, so callout_drain() is final instead of racing a re-arm. octep_dp_if_detach_all() quiesces at its top and resumes at its bottom, so both callers get it and dp.if_del - which leaves the datapath up - can carry on afterwards.

Testing it without paying for it

Reaching the quiesce the way the driver does takes the twelve front ports with it, one of which is the WAN, and the last time those ports went away for a boot OPNsense dropped four interface assignments out of its configuration. That is too much to spend on a test, so the mechanism is reachable on its own: dp.rx_quiesce runs exactly the quiesce the detach path runs, and dp.oq_busy shows the flag and one busy flag per served ring.

dp.oq_busy also carries the two numbers that answer the question a sample cannot. A servicer is inside a ring for microseconds, so a reading taken next to one is almost always clear - which is what the first run of this test showed, and it proves nothing either way. waits counts the quiesces that had to spin at all, and longest the worst one in microseconds. If waits stays at zero under load, the wait is decoration; it does not.

What the test found, which was not what it was for

The first version of the quiesce was written to close a use-after-free, and the knob was added only so the mechanism could be exercised without taking the front ports away. Hammering it found a second defect, in the resume rather than the quiesce.

5,285 quiesce/resume cycles over ten seconds, with a download offering 1,510 frames a second underneath, delivered four frames. The receive path was dead for the whole run, and the appliance was at 100% loss to the internet immediately afterwards.

The reason is the same hardware behaviour the drain loop exists for. A quiesce turns a servicer away before it clears R_OUT_INT_STATUS, so every interrupt arriving while the quiesce is set is taken and discarded with the status still latched - and the block raises its interrupt on the count crossing the level, not on its sitting above it. Once the quiesce lifts there is nothing left to raise one. Clearing the status on the way out instead would lose the edge just the same.

So resume has to look at the rings itself, and the first version did not: it armed the watchdog 50 ticks out and returned, and the next quiesce two milliseconds later drained that timer before it could fire. Nothing ever looked.

Resume now drains every ring before it returns, bounded like the handler's own loop, because a quiesce taken under load leaves a backlog one pass cannot clear and no interrupt is coming to finish it. (Sixteen sweeps of one ring's worth from every ring, now; and it arms the watchdog the way the watchdog arms itself, because arming it for its period threw away a request for its next tick. The same test was run again there, 2,782 cycles under a stream that did not slow: a visit is not a pass.)

5,285 cycles, before	4 frames delivered, 100% loss after
the same test, after	21,990 frames delivered, 0.0% loss after, and a 16.7 MB fetch in 4 s

The lesson is narrow and worth keeping: on this block, anything that declines to service a ring owes a service when it stops declining. The drain loop learned it, the watchdog learned it, and the quiesce had to learn it separately because it looked like a locking problem rather than an interrupt one.

The watchdog had not learned it. Its one pass over a ring the handler was working left that ring with packets in it and nobody inside, fifty milliseconds at a time, and the handler that found the ring held was declining with nobody owing. It showed as one TCP stream through the host that would not go faster than 75 Mbit/s: a visit is not a pass.

And the number the mechanism needed

waits 74 on that run, longest 60 us. Seventy-four times in ten seconds a quiesce found a servicer already inside a ring and had to wait for it - which is the measurement that says the wait is load-bearing rather than decoration, and the one a sample of the busy flags could never have given. Without it, those seventy-four interfaces would have been freed under a servicer holding mbufs that pointed at them.

A bridged front port, and the filter that made one look alive while dropping every reply

A front port's hardware filter admits unicast only for the one address that port was given. That is right for a routed port. Put the port in a bridge and it is wrong for every frame that matters: the replies to the machines behind it carry the bridge's address, so none of them is admitted - while broadcast still arrives, so ARP answers and DHCP hands out leases and the port looks perfectly healthy. The symptom points at the firewall rules, the switch, the driver, the cable; not at the one thing that is filtering.

The attribute that lifts it is NetAgent 0x45, and the appliance had been getting it from a line in the bring-up script that turned it on for ten ports unconditionally at boot. That worked and was wrong in both directions: a routed port paid for a filter it did not need, and the setting lived in a shell script where nothing in the system could see it or reverse it.

The mechanism

octep now follows IFF_PROMISC. The link poll already runs on taskqueue_thread because the NetAgent request sleeps; each tick it reads the flag off every front-port interface and sends the attribute only where the answer changed. The flag is read from the interface rather than recorded from SIOCSIFFLAGS, so there is no transition to miss and no second copy to go stale.

Two sizing decisions, both measured rather than assumed:

• The link read stays one port per tick - it is paid every second forever, and twelve NetAgent round trips a second on a control channel this project has watched go down under one unlucky request is not worth a faster answer to a question that moves at the speed of somebody plugging in a cable.
• The filters sweep all twelve every tick. Their conditions are false unless the stack has just changed its mind, so in the steady state the sweep costs twelve comparisons and no mailbox traffic at all - and a bridge is built by adding every member at once, so round-robin would have left the last member deaf for twelve seconds.

A refusal is reported once and not retried: a firmware without the attribute answers with a failed status, and asking again every second for the life of the machine is the wrong answer to a clear no.

What it did, on a bridge of eleven of the twelve ports

The appliance's own configuration: bridge0 with every front port in it except the one that is the WAN. So eleven ports want the filter open, one does not, and the driver is told nothing by anybody - it reads IFF_PROMISC and acts.

[86] oxp0: promiscuous on
[87] oxp1: promiscuous on
[88] oxp2: promiscuous on
[89] oxp3: all-multicast on
[90] oxp4: promiscuous on
[91] oxp5: promiscuous on
[92] oxp6: promiscuous on
[93] oxp7: promiscuous on
[94] oxp8: promiscuous on
[95] oxp9: promiscuous on
[96] oxp10: promiscuous on
[98] oxp11: promiscuous on

front-port interfaces present	12
bridge members	11
ports carrying IFF_PROMISC	11 - the members, and only those
promiscuous requests sent	11
requests refused	0
requests abandoned after retries	0
oxp3, the WAN: a routed port	no promiscuous request, all-multicast only, lease held, default route on it

One port per second, and that is the bound rather than a coincidence. The timestamps are consecutive ticks: one request per invocation of the link poll, which is what stops twelve ports from becoming twenty-four sleeping mailbox transactions in a single pass of taskqueue_thread. The one gap, at [97], is a tick that spent itself on the link read instead.

The same message buffer still held the boot before it, which ran the unbounded version, and the contrast is the measurement:

[88]  oxp0: promiscuous on
[88]  oxp5: promiscuous on
[842] oxp1: promiscuous on      nine ports, one second, one task invocation
[842] oxp2: promiscuous on
[842] oxp4: promiscuous on
[842] oxp6 .. oxp11: promiscuous on

Nine requests inside one tick. Healthy, each costs about 12 ms and nobody notices; on the timeout path each costs up to four seconds, and that is a shared single-threaded taskqueue held for most of a minute by a driver polling a mailbox. The defect was found by review before it was seen, and the log of the boot that had it is the reason the bound is not negotiable.

The attribute was also confirmed to work in both directions before any of this was trusted - status 0x00000000 for param=0 as well as param=1, on a port with no cable, with the state put back as found. A mechanism that can only turn a filter on would leave a port permissive forever after it left a bridge.

Then 30,000 frames through it, to see what the filter costs

Method. ping -f -c 30000 -s 1472 from the appliance to a PC behind the bridged port, so every frame is full-size and every reply is addressed to the bridge. The coprocessor's named counters read before and after; elapsed time from the shell.

30000 packets transmitted, 30000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.197/0.328/1.095/0.034 ms
elapsed 10 s

counter	delta	per second
FPCNTR_RX_WIRE	29,790	2,979
FPCNTR_TX_KN	29,790	2,979
FPCNTR_FROM_WIRE_TO_KN_FORCED	29,790	2,979
FPCNTR_RX_KN	29,788	2,979
FPCNTR_FROM_KN_TO_WIRE	29,788	2,979
FPCNTR_TX_WIRE	29,788	2,979
FPCNTR_TX_DROP	0	-

Interface counters over the same window: oxp0 51.9 MB in and 50.1 MB out at 1,485 bytes a frame, Idrop 0; dp.tx_iq_full 0; dp.oq_busy reports no ring contention and no quiesce waits.

Three things this says, and one it does not.

• Nothing is lost on either hop. Every counter in the six-stage chain moves by the same number, twice - once per direction - and the host's own dp.rx_done and dp.tx_posted had read exactly equal to FPCNTR_RX_WIRE and FPCNTR_RX_KN before the run began. Host and coprocessor agree to the frame.
• Promiscuous mode costs nothing measurable. 0.328 ms average with the filter open, against 0.459 ms on a lighter run and 0.63 ms on the first three frames; the spread is scheduling, not the filter. There is no per-frame work in a filter that has been told to admit everything.
• FROM_WIRE_TO_KN_FORCED equals RX_WIRE exactly, every frame, which is the same reading that settled #221: with no flow installed, the fast path forwards everything to the host and accelerates nothing. A bridge does not change that.

It does not establish a ceiling. 2,979 frames a second is lower than the 5,003 further up this page, and the difference is the far end: this run answers a Windows PC rather than a router, 0% was lost, and not one of the driver's pressure counters moved. Nothing here was stressed, so this is a floor on what the path can do and not a limit.

FPCNTR_TX_DROP is 146, and it is from start-up

The first reading after the boot showed FPCNTR_TX_DROP and FPCNTR_TX_DROP_QUEUE_FULL both at 146, which reads like a 4.6% loss toward the host. It is not: across a 3,000-frame run and then a 30,000-frame flood, neither counter moved at all. The 146 was spent once, during bring-up, while the rings were being armed and before the logical interfaces existed - which is where a frame has nowhere to be delivered by construction.

Worth recording because the snapshot alone would have been read as an ongoing loss, and the only thing that distinguishes the two is a second reading under load. A counter with no derivative is a fact about the past.

Jumbo frames are not available, and the number is 1506

Jumbo frames are not available, and the number is 1506. Struck 2026-10-12: a port's MTU is any value from 72 to 9,000 since that day, the coprocessor having always written a long frame over as many buffers as it needs - see the MTU is the LIF's and the rows below. The arithmetic that follows was right about one buffer and wrong about the limit; it stays as it was written.

Asked for directly, and the answer is in the ring rather than in the hardware. The receive ring is programmed with one buffer size, which the live register dump confirms as BSIZE 1602:

OCTEP_DP_BUF_SIZE     = 1536 + 66 (the private tag)   = 1602
OCTEP_RX_PREFIX_LEN   = 82         (the far side writes its own prefix into the same buffer)
Ethernet header       = 14
OCTEP_DP_IF_MTU_MAX   = 1602 - 82 - 14                = 1506

So ifconfig oxpN mtu 1506 is accepted and 1507 is not, and mtu 9000 cannot be honoured by asking for it: the far side would write a 9,000-byte frame plus an 82-byte prefix into a 1,602-byte buffer. Jumbo frames on these ports are a ring-sizing change - a larger BSIZE published to the coprocessor at ring-arm time, which is latched once per coprocessor boot - and not a configuration setting.

Recorded because the clamp has been wrong once already in the other direction: it read OCTEP_DP_BUF_SIZE - 128 = 1474, below the 1500 every one of these interfaces actually runs at, so mtu 1500 failed with EINVAL and a port lowered to 1474 could not be raised again without a reboot.

Ten hours at light load

2026-10-08 16:24 to 2026-10-09 02:24 by the appliance's clock, module 73d533b9 - main after a ring that fills. The trigger on (dp.auto 1), with making between polls, reviving and the keep-alive at their defaults, and ipsec.flows 0. Those four had never run together for longer than 33 minutes, and it was the first long run of the receive path of the two changes before it. It is written here some hours after it ended and not when it ended; the first of what it found had been posted on two issues the evening it began.

How. A script on the appliance read the driver's counters, five of the coprocessor's and pf's state-mismatch count once a minute, 600 times, and would have switched the trigger off had five commands timed out or twenty been refused. A machine on the LAN, out through NAT by the WAN front port, ran one cycle every ten minutes: a download from a public server cut after six seconds, five small fetches, twenty DNS questions. One connection at a time, sixty-four cycles.

What else used the appliance meanwhile, because it shows in the lines. The measuring machine's own traffic, which reaches the internet through the appliance's management port. About twenty runs of the candidate-table tests of the next section, in the first hour and a half: they are most of the 565 connections made in the first hour. And three compiles of a branch, none of them installed.

samples	600 of 600; no panic; the guard never tripped
commands to the coprocessor	0 timed out, 0 refused; dp.flow_rc_refused 0, dp.auto_full 0
connections made (dp.auto_made)	1,958, of them 321 between polls (dp.fast_made); 374 directions attached later (dp.flow_attached)
taken out	681 because pf no longer had their state (dp.auto_gone), 903 found reclaim-pending on the far side (dp.flow_pending); the far side agreed to reclaim 1,584 (dp.flow_reclaimed). The most held at one sample: 87
keep-alive touches (dp.ka_touched)	5,596
attempts put off for a pf state not settled (dp.flow_unsettled)	20,736
frames the coprocessor forwarded without the host	2,166,821
frames the host took (dp.rx_done)	629,352; the most the host carried at a sample 7 Mbit/s
dp.rxwd_stalls, dp.oq_lapped, dp.rx_resync, dp.rx_short, dp.oq_unseen	0
dp.oq_bound, dp.credit_capped	did not move (3 and 1,151 when it began)
rings the watchdog found work in unasked (dp.rxwd_rescues)	2,035
given back by the retransmission rule	1 (the coprocessor's own count), revived in place (dp.flow_revived 1)
dp.flow_tuple_mismatch	0 to 30, every one an ICMP echo read from its reply: issue 326
pf's state-mismatch counter	+81
the candidate table's side entry	armed 721 times, read 42, let go at its limit 83: issue 322

Per hour, as differences:

hour beginning	made	between polls	attached	pf's state gone	reclaimed	kept alive	forwarded without the host
16:24	565	82	32	440	533	1,062	294,195
17:24	97	14	11	18	89	375	203,977
18:24	124	25	18	26	104	415	246,751
19:24	153	31	55	25	99	428	225,816
20:24	186	33	51	30	133	540	274,995
21:24	146	29	57	21	90	403	257,918
22:24	183	26	60	28	123	542	128,808
23:24	179	23	23	33	156	660	177,067
00:24	166	25	26	27	140	564	176,192
01:24	159	33	41	33	117	607	181,102

The load, as the machine on the LAN saw it.

downloads	64 begun; 60 carried data until the six-second cut, 0.9 to 83 MB each, 45 MB on average; 4 received nothing - three never connected, one connected and received nothing in its six seconds
small fetches	320; 5 failed
DNS questions	1,280; 9 unanswered
a TLS failure of the kind in issue 279	none
downloads in which a connection was given back and revived	1 of 64: issue 315 had read one in ten

The four downloads that received nothing were at 20:26, 22:32, 23:11 and 00:38. The router this appliance's WAN is a client of keeps a monitor on its own upstream gateway, and its log has an alarm standing at each of those four moments: delay rising to down at 20:26; down from 22:30 to 22:33; down from 23:10 to 23:12; down from 00:38 to 00:42. All five failed fetches are in the last two of those cycles. The appliance's counters show nothing in those minutes but dp.flow_unsettled rising, which is opening frames being sent again. So they are put down to the line, on that evidence and no other. That monitor raised an alarm thirty-three times in those ten hours and read down in thirty of them.

(The private record of this run said "64 downloads, every one ended by the cut" and named two of the four. It was read from curl's exit status, which is the same for a download that is cut and for one that never connects. The byte counts were in the same log.)

What it does not say. It is ten hours at a handful of connections, on a line of about sixty megabits. It says nothing about load. And it could not have seen a receive ring that was behind the block - the fault of a packet is not a buffer, which this build had: no counter moved for that, and dp.state was not read when the run ended. One new connection in eight failing would have shown in the fetches, and did not; with eighty-seven connections at the most the messages that cause it are seldom long enough. That is an argument, not a reading.

The candidate table with more than one user

2026-10-08, module 73d533b9, during the first hours of the run above. Issue 322, which was read and not run when it was written. These rows are what main does; the page that will describe what replaces the side entry is not written, because the change is not merged. The first of them were posted on the issue the same evening and the rest waited for that page, which is how they came to be late.

How. UDP through NAT from a machine on the LAN to a public resolver, a question and its answer every twenty milliseconds on each flow that is driven. Two or three hundred flows are opened first; the appliance is asked what port pf gave each; the slot of each direction and its bucket in the table of live tuples are computed with the driver's own functions; and the flows that fit the row are driven for eight seconds while dp.flow_table is read every half second. "Made by" is the first reading that showed the connection.

A connection that collides with itself, with a held connection in each of its two buckets. Two more flows are opened from ports chosen so that each one's tuple counts in the same bucket as one of the collider's, driven until the table holds them, and kept in use.

	runs	made
collides with itself, a held neighbour in each bucket	7	0
collides with itself, no neighbour, in the same runs	6	4

In the seven the side entry was armed 11 times and read 0.

The test of a connection that collides with itself again, two colliding flows and a control, one after another: 10 of 12 colliding flows made, where that page has 7 of 7 and 6 of 6; the six controls all made.

The side entry at rest. Three minutes with no test running: armed once, read 0 times, and 1,338 attempts refused for want of a next hop - connections that come in by the management port, whose opening direction no ring will ever carry. At the start of that run the counters since boot read 93 armings, 2 reads, 106 askings while it was another tuple's, 29 let go at the limit; nine minutes of the tests later, 117, 10, 126 and 34.

The bar. A flow that collides with itself and is driven one way only, so that it can never be made, and a second colliding flow driven both ways for three seconds from the fourteenth:

second
4	the entry is armed for the one-way flow
12	it has held it its eight seconds and is barred
16	the second flow has been given the entry, read from it and made
18	the one-way flow is armed again - six seconds after it was barred for thirty
28	barred again
58	armed again, thirty seconds after that

The bar is the tuple still written in the entry, so any other tuple's turn lifts it.

Several at once. Every flow found to collide with itself, all driven together:

colliding flows	made by, seconds
6	0.5, 1.2, 2.6, 3.9, 4.6, 5.9
2	0.5, 1.9
4	1.2, 1.9, 2.5, 3.9
6	1.2, 4.0, 5.3, 6.0, 6.7, 8.0
3	1.2, 2.5, 3.2

One a second or so, which is the entry's one place. Across those five runs the entry was wanted 35 times while it was another tuple's.

Several tuples in one slot. Flows from source ports chosen so that their opening tuples share a slot, never used before the drive, all driven together. (A first set of these rows was thrown away: the flows had been used to find them, and were already in the table when the drive began.)

flows in the slot	made, of them	made by, seconds	candidates overwritten unread
3	3	0.5, 1.2, 1.2	88
5	5	1.9, 2.6, 4.6, 6.0, 8.0	890
8	8	0.5, 1.9, 2.5, 3.9, 3.9, 5.2, 5.2, 5.9	1,684
12	8	0.5, 1.9, 3.3, 3.9, 5.3, 5.9, 7.3, 8.0	2,476

How old the other direction's candidate may be. One direction of a UDP flow is punted once and goes quiet; after a gap the flow is driven both ways.

gap, seconds	made	"the other direction has not been punted"	a direction attached afterwards
4	yes	0	-
8	yes	0	-
12	yes	0	-
20	no	7	-
40	no	0 (7 attempts put off, pf's state not settled)	-
4, one question after the drive	yes	0	0
8, the same	no	6	0
12, the same	yes	0	1
16, the same	yes	0	1

The second set does not agree with the first at eight seconds and was not repeated on this build.

Through the lab tunnel, the next morning on the same boot, ipsec.flows 2: flows whose two tuples share a slot, found by their ports, and a control, all at once.

	made by, seconds	the control	decrypted frames handed to the host
six UDP flows	1.2, 2.6, 3.3, 4.7, 5.3, 6.7	0.5	963
six UDP flows	1.2, 1.9, 3.2, 3.9, 5.3, 6.0	0.5	843
three TCP flows, 203 to 259 Mbit/s each	0.5, 0.5, 1.9	0.5	33,839
three TCP flows, 178 to 319 Mbit/s each	1.2, 1.2, 1.9	0.5	95,652

(The first attempt at each of the two kinds made nothing and echoed nothing: the tool had failed to bind its ports. Not rows.)

Nine hundred connections held, the same morning: nine hundred UDP flows through the tunnel to an echo at the far end, made fifty at a time at fifty datagrams a second each, then each kept in use with a datagram every four seconds.

the table stopped growing	110 s after the first flow was opened; 112 s when it was run again from a clean boot
connections held	895
runs between polls that made nothing	637
creates refused by the coprocessor	0
one TCP download through the tunnel meanwhile	794 Mbit/s; 774 from the clean boot

What these do not say. The tunnel rows and the first of the nine-hundred rows were taken at the end of the boot the ten-hour run above had just finished on; whether a receive ring was behind the block by then was not read. The row from a clean boot left ring 0 seventy-three buffers behind: that is the next section but one.

The same rows on a build that was not merged - withdrawn

2026-10-09, module 6896a1cc: main with the candidate table given four places a slot in place of the side entry, and the reply tuple of an ICMP echo derived as its frames are read (issue 326). None of this is a result. At the end of the boot these rows were taken on, receive ring 0 was found 268 buffers behind the block, and one new connection in eight was timing out. When in the boot it began to fall behind is not known. The fault is main's and not this build's, but every row below was taken with it present or possibly present, and each has to be taken again on a build without it. They are here because they were taken, and so that the next set can be read against them. (Taken again on 2026-10-11, on f78812a8 - the change over a main that has the fix - from a clean boot with the rings read level throughout: a slot holds both directions has every row.)

row	as it read	on main, above
collides with itself, a held neighbour in each bucket	3 of 3 made	0 of 7
the same runs, no neighbour	2 of 2	4 of 6
the test of the side entry's page	4 of 4	10 of 12
nine colliding flows at once	every one by 1.2 s	one a second
six, and seven	every one by 1.2 s; every one by 0.5 s	one a second
three, and four, in one slot	all by 0.5 s	0.5 to 1.2 s for three
five in one slot	0.5 to 1.2 s	1.9 to 8.0 s
eight in one slot	0.5 to 1.2 s	0.5 to 5.9 s
twelve in one slot	12 of 12, by 2.6 s	8 of 12 in 8 s
the other direction's candidate 4, 6 and 8 s old	made, nothing attached afterwards	-
9, 10, 12, 14, 20 and 26 s old	made, one direction attached afterwards (two at 9 s); dp.flow_rc_refused 0	not made at 20 and 40
11 s old	no row: the one question after the drive was not answered	-
a discard of the flows	256 places of the candidate table written before it, 0 after	-
twelve pings through NAT	11 answered; dp.flow_tuple_mismatch +0, dp.flow_wait_other +0	12 answered; +1 and +1
six UDP flows through the tunnel, twice	every one by 0.5 s; 33 and 123 frames handed to the host	1.2 to 6.7 s; 963 and 843
three TCP flows through the tunnel, twice	every one by 0.5 s; 3,528 and 3,869 handed up	0.5 to 1.9 s; 33,839 and 95,652
nine hundred held: the table stopped growing	33 s	110 s
the same: runs that made nothing	18	637
the same: connections held; the download	913; 760 Mbit/s	895; 794

And what was seen on that boot that belongs to the fault and not to the change:

five more sets of fifteen pings through NAT	14, 15, 15, 15 and 15 answered
eight pings to each of six addresses not used before	five answered all eight; one answered none, with dp.flow_wait_other +9
the three-mode tunnel suite	three rows with a stream that timed out: four down and four up with ipsec.flows 0 (one stream of four at 0 in each), one down with ipsec.flows 1 (0.0 Mbit/s); the other nine rows 739 to 964
the suite's second check, three pings of the largest size that fits	0 of 3 answered
forty-two connections to the appliance's own address, source ports in a row	36 opened, 6 did not; one that did not, five more times: never; one that did: every time
the tunnel	found down, no association installed, about an hour into the boot; why was not looked for
three hundred of the nine hundred flows, tried again then	none made; nothing echoed

A ring behind the block, build by build

2026-10-09. A packet is not a buffer has the account; this is every row, on every build, including the ones that page gives in a sentence.

On the build it was first seen on, 6896a1cc, an hour into its boot:

buffers written and not taken	268, all on ring 0; the same figure in every reading for half an hour
ring 0's doorbell, of 2,032	1,748 to 1,763; the seven siblings 2,016 to 2,031
what the block had written to each ring since boot	ring 0: 28,263 packets; rings 1 to 7: 1.0 to 4.1 million each
interrupts each ring's vector had taken, read a little later	ring 0: 22,253; the others 320,007 to 1,236,048
the eight rings' counters together against dp.rx_done	17,016,890 and 17,008,395 - 8,495 apart, which settled nothing: the driver's total is added to by eight threads without a lock
177 opening frames from one failing port in 8 s	ring 0 took 188 packets in 185 interrupts, and 102 of what it took carried a tag no interface owns; the coprocessor counted 282 frames from the wire and 282 to the host, none dropped
sixty-four UDP tuples, a datagram each	56 seen on the interface's tap within a second, 8 not
ring 0 at rest, ten seconds	20 packets, 13 of them with a tag no interface owns; the one kept carried the control tag, 254
four pings of 1,478, 1,479, 1,500 and 1,550 bytes from the far end	none answered; the coprocessor's FPCNTR_TX_DROP +4 each time; nothing written to a ring
four pings of 1,472 bytes	none answered, and no drop counted: not explained, and not tried on another build

The scale row from a clean boot, on main and on each build of the change:

build	packets of more than one buffer	buffers they used after their first	the longest, bytes	buffers left written	tuples of 64 seen	the table stopped growing	download
73d533b9, main	not counted	-	-	73	56	112 s	774 Mbit/s
049f58da	72	77	not kept	0	64	112 s	774
7544df3a	60	60	2,736	0	64	112 s	765
8e5cc09b, merged	62	62	3,114	0	64	114 s	810

b69959d1, between the first two, was not given this row. On it the control frame at rest was read with the longer capture: type 2, two entries, the same in twelve readings a second apart.

The instrument, dp.oq_lose, on ring 0 - packets acknowledged and not taken, and how they came back:

build	5, the ring at rest	40, under 300 datagrams back to back	40, under 300 datagrams 4 ms apart	numbered datagrams	dp.rxwd_stalls
049f58da	5 in 5 passes	40 in 10	40 in 23	600 of 600, in order	0
7544df3a	5 in 5	40 in 7	40 in 23	600 of 600, in order	0
8e5cc09b	5 in 5	40 in 3	40 in 40	600 of 600, in order	1

dp.oq_unseen and dp.rx_resync 0 on all three. The first two builds took every packet up to the first empty buffer; the last takes as many as were there when the wait began. Its walks under the back-to-back burst were fewer and larger, and one of them brought back enough at once to be counted a stall. Whether that is the change or the moment was not established.

The three settings of ipsec.flows, twelve seconds a row, Mbit/s:

build	setting	four down	four up	one down	one up
049f58da	0	966.2	957.3	789.7	771.1
	1	964.1	954.0	797.7	769.0
	2	940.2	893.7	780.5	793.9
7544df3a	0	956.6	939.3	797.9	769.3
	1	956.1	942.1	776.2	762.0
	2	952.4	945.5	769.7	778.0
8e5cc09b	0	954.9	944.2	807.6	761.6
	1	925.7	937.6	820.4	791.8
	2	951.3	946.3	789.3	767.7

The six checks and the clear-text probe read on all three as they had before the change. The 893.7 on the first build was not looked into: nothing in that row's counters stands out - no frame refused by the coprocessor, six retransmissions at the far end.

The hand-back rows, downloads with ipsec.flows 2, Mbit/s:

build	making between polls and reviving	four, 12 s	one, 12 s	four, 30 s
b69959d1	both off	966.0	806.6	920.2
	making on	959.8	914.7	949.6
	both on	930.3	805.2	927.9
7544df3a	both off	950.3	807.9	938.5
	making on	954.4	792.0	934.9
	both on	957.2	769.0	938.4
8e5cc09b	both off	945.9	776.2	942.4
	making on	966.5	906.5	940.6
	both on	956.3	781.5	944.7

Three rekeys under a four-stream upload, sixteen seconds each, ipsec.flows 0:

build	Mbit/s	dropped by the far end as replays	dropped while an association installed
b69959d1	920.6, 935.2, 938.9	0, 0, 0	0
7544df3a	938.5, 938.7, 928.8	0, 0, 1	0
8e5cc09b	949.2, 948.3, 944.6	0, 0, 0	0

One frame in nine rekeys, where installed first, taken second counted none in twenty-three. Not looked into: the appliance sent nothing by the kernel's cipher on an association the driver did not yet have, dropped nothing while one installed, and counted no replay of its own.

The receive path's counters over each boot:

build	what ran	frames	dp.rxwd_stalls	dp.oq_unseen	dp.rx_resync, dp.rx_short, dp.oq_lapped, dp.rx_long
049f58da	scale, instrument, tunnel suite	11.1 million	0	0	0
b69959d1	hand-back, rekey	9.6 million	2	2	0
7544df3a	scale, instrument, all three suites	21.3 million	0	0	0
8e5cc09b	the same	22.1 million	3, one of them the instrument's	0	0

No packet was taken without a count on any of them except under the instrument.

The way back, and the stalls, build by build

2026-10-11. The way back keeps up has the account. f78812a8 is main with the fix above; c12b6dfd is main with dp.oq_lose_burst and dp.rxwd_stall_log.

The instrument on ring 0, a numbered UDP stream from the LAN, the interface's tap open for the whole stream:

build	stream	to lose	lost	back	walks	written and not taken, read every second	handed up
f78812a8	3,000 at 4 ms	2,100	1,489	1,489	1,489	never above 1	3,000 of 3,000, in order
f78812a8	42 s as fast as the sender makes them, 1,400 a second	2,100	1,463	1,463	469	never above 1	in order
f78812a8	3,000 at 4 ms, nothing lost	0	0	0	0	0	3,000 of 3,000, in order
f78812a8	3,000 at 4 ms	300	300	300	300		in order

A first run of the first row read 337 missing: sent after the tap had closed, the tap fourteen seconds and the stream fifteen and a half. One earlier run of the 2,100 row, before the tool counted inversions, printed that the order was not kept; the run after it, the same row, that it was. Not reproduced.

The instrument on the download's ring, under a one-stream download through the tunnel, ipsec.flows 0, the ring found while the download ran:

build	download	to lose	a pass may spend	spent	back, in walks	written and not taken at a reading	dp.rxwd_stalls
c12b6dfd	780 Mbit/s, 105,000 frames a second, 30 s	3,000	64	3,000 within 4 s	3,000 in 18	18 before; 42 at most under it; 0 at the end	+4, the walks of 32 or more

A second download after it, 775 Mbit/s. dp.oq_unseen and dp.rx_resync 0. A first run of this row put the instrument on a ring found with a separate download, which hashes to a ring of its own: the row was void and the tool rewritten.

The stall log, dp.rxwd_stall_log, everything it held on c12b6dfd over the rows above, the three standing suites run twice, and four rekeys run for it:

what ran	ring	first pass	ticks nobody inside	the visit
the download row, before the instrument was armed	0	50	4	50
the rekey suite, first run	7	237	12	603
the rekey suite, second run	1	96	5	270
under the instrument, the way back's walks	2	195, 347, 300, 252	18, 48, 43, 45	195, 347, 302, 338
four rekeys, at the first	2	257	12	624

The three-mode tunnel suite and the hand-back suite, twice each: none. The four rekeys, sixteen seconds each under a four-stream upload with ipsec.flows 0: 932.8, 949.6, 928.3, 907.8 Mbit/s; dropped by the far end as replays 0, 0, 0, 0; packets the kernel numbered before the coprocessor took the association 697, 945, 33, 943. A sampler on the appliance every 100 ms across the fourth rekey, 406 readings: written and not taken on a ring 1 to 29, the 29 on ring 4 in the upload's first second, single digits after. The samplers of the first three rekeys were overwritten, each by the next, and the log's readouts between rekeys printed an error in place of the log - the tool sent $(...) to the appliance's csh; the log itself keeps the last eight and was read after.

Six more rekeys with the tool fixed, 2026-10-12, every sampler kept, same upload and setting:

rekey	Mbit/s	replay drops at the far end	install, microseconds	stall	the sampler's most written and not taken on a ring, at one reading
1	931.5	1	5,008	none	15 on ring 6
2	953.3	0	81	none	203 on ring 2, a servicer inside it
3	948.4	0	95	none	35 on ring 1
4	953.7	0	1,176	ring 1, first pass 109, 6 ticks nobody inside, visit 320	95 on ring 1, at the install's reading
5	950.4	0	91	none	12 on ring 2
6	948.6	0	42	none	28 on ring 6

Four stalls at a rekey in twelve rekeys now, and the one replay drop of the ring-fault builds (1 in 9) is 1 in 10 since. The install's hold of the driver's lock, which the termination of every decrypted frame also takes, was checked against these durations: a 1.2 ms install beside a stall of six ticks, a 5 ms install beside none. Not it alone.

The NUL bytes in the sampler's log, and where they came from. The sampler's log of the fourth rekey, 36,870 bytes in 406 lines, carried 1,896 NUL bytes, and the first reading of that was that dp.state's output carries them. It does not: read three times at rest on 2026-10-12, sysctl -n dev.octep.0.dp.state gave 1,115 bytes each time and no NUL in any. The NULs are one contiguous run at offset 34,812 of the log, with the fourth rekey's last lines before it and the third rekey's last lines after it: the tool ran every rekey's sampler into the same file, the fourth truncated it while the third was still writing at its old offset, and the bytes between the two writers' positions are the hole a file system leaves. The tool writes a file a rekey now. The sampler's readings themselves - written and not taken on a ring, 1 to 29 - are unaffected, the hole being after the last of them.

An address change under a mirrored association

2026-10-12, module c12b6dfd, for issue 317: the first time one end of the tunnel changed its address under an association the coprocessor was encrypting. Installed first, taken second has the account.

How. The lab tunnel on the 10G path, the appliance made the IKE initiator (only the initiator moves its address, RFC 4555), dp.auto 1 and ipsec.flows 0, so every frame crosses the host and the coprocessor is the only cipher. A second address, in a second /30, on the tunnel's port, with the connection's local addresses set to both; on the peer a second runtime address in that /30, so the moved address is on-link. A stream of 200 datagrams a second from the LAN machine through the tunnel for 40 s, the peer capturing the ESP from either address and the IKE; eight seconds in, the first address taken away from the port. Then, in a second run, put back.

	the address taken away	the address put back
what charon did	MOBIKE moved the IKE SA to the second address pair, then unable to update SAD entry with SPI c2184f81: reqid change is not supported, and rekeyed the child: new SPIs, a fresh exchange	nothing: it stays on a path that works
the kernel's clone, key_updateaddresses	not called	not called
the driver	sa_installed 36 to 38, sa_removed 34 to 36, sa_failed 0, out_drop 0, tx_nosa 0, rx_nosa 0, rx_bad 0	unchanged
the old association's last packet, the new one's first	00:19:47.408 and 00:19:47.528: 120 ms	-
ESP captured at the peer	1,621 on the old association, 6,145 on the new; sequence numbers consecutive on each; IV equal to the sequence number on all but the capture's cut last line	7,772 on the one association, consecutive, IV equal to the sequence number
the kernel's cipher	sent nothing: no lower-half IV, no upper-half IV	the same
dropped by the peer as a replay	0	0
the stream	8,000 sent; the capture opened about a second after it began	8,000 sent

Two things the run itself did that are worth knowing. The host route to the peer's first address by the tunnel's port was not installed by the tool (route add -host ... -iface is not the flag), so for the seconds between the address going and MOBIKE settling, the appliance's route to that address was its default, and charon's path probes to it left by the WAN - three IKE probe packets to a lab address, out of the upstream port, before the probe of the second pair answered. And the capture's last line is cut by the capture's own timeout, which is the one packet on each run whose IV could not be read.

The first connection through the tunnel, and where the MSS clamp has to sit

2026-10-12, module c12b6dfd, for issue 321: the first TCP connection a LAN machine opens through the tunnel stands still for 0.3 s while it learns the path's MTU - ten 1,460-byte segments with DF, ten "fragmentation needed" answers from the appliance, a 300 ms retransmission timer. The issue said what would remove it, a maximum MSS on the tunnel's traffic, and did not try it. Tried, in three forms, on the lab tunnel over the 10G path, one connection from the LAN machine each time with the far end capturing the decrypted side and the appliance's ipsec.out_needfrag read before and after:

setting on the appliance	the SYN's MSS as the far end received it	ipsec.out_needfrag	the connection
none	-	+11	the machine learned a path MTU of 1,446
a normalisation rule, scrub in on <the member port> proto tcp to <the far end> max-mss 1406	1460	+0	1 MB in 10 ms - the machine's own memory of the path, not the rule: the rule matched no packet at all (evaluated 102,713 times, 0 packets), because OPNsense's scrub in all comes after it and the last matching normalisation rule wins
the interface's MSS field on the member port the machine is cabled to: scrub on oxp5 inet all max-mss 1406	1460	+0	the same; the rule matched 38 packets and clamped nothing: the machine's packets reach pf on the bridge, not on the member
the interface's MSS field on the bridge, the LAN: scrub on bridge0 inet all max-mss 1406	1406	+0	1 MB in 10 to 30 ms, no pause over 100 ms, no segment over 1,406

So the remedy is the MSS field of the interface pf sees the LAN machine on - the bridge here - set to the tunnel's MTU plus forty, 1,446. Nothing in the driver.

Three things the measuring found about itself. The Windows machine's memory of a path's MTU outlives what netsh interface ipv4 show destinationcache lists, and the peer's keep-alive pings to it refreshed that memory, so two "after" runs read 1,406-byte segments from a machine that had not been clamped; the SYN's MSS at the far end is the reading that cannot be fooled that way, and is the column above. The first run's client sent no mode byte to the far end's server and was reset before it could time anything, so the 0.3 s itself is the issue's own reading of 2026-10-08 and not this day's. And out_needfrag rose by ten between two runs while nothing of the tool's was running and the pinger was stopped: a first connection from the machine that was not this tool's; which, not known.

An association behind NAT, on the coprocessor

2026-10-12, for issue 294's first item; the UDP header is the host's to write has the account. The lab tunnel over the 10G path with encapsulation forced at both ends (ESP inside UDP 4500), dp.auto 1, ipsec.flows 0.

build	the association	inbound, the peer's frames	outbound, the appliance's frames	rates, Mbit/s
b2564f90: taken, ports and udp_enable in the request, envelope plain ESP	mirrored: esp-udp in the kernel, two table rows	decrypted and terminated, rx_done +106, rx_bad 0	refused by the coprocessor: FROM_KN_TO_IPSEC_ENCR +5, CRYPTO_DROP_PROTO_ERR +5, nothing on the wire	0 - every stream timed out
1707f625: the envelope carries the UDP header	the same	rx_done 4,538,392, rx_bad 0, rx_nosa 0, rx_nokey 0; FROM_WIRE_TO_IPSEC_DECR +285,858 over a burst	accepted: the peer's ESP in +2,526,537, replay drops unchanged, its capture shows the appliance's ESP inside UDP from sequence 0x100001; FROM_KN_TO_IPSEC_ENCR +285,787, no CRYPTO_DROP_*; out_drop 0, tx_nosa 0	four streams 929.2 up, 957.9 down; one stream 798.5 up, 826.4 down

A 1,472-byte ping with DF is answered "needs to be fragmented"; 1,410 bytes, eight less than the plain tunnel's largest, gets three of three replies. Lab raised 01:24, down 01:36; the peer's encap turned on and off again, its configuration otherwise untouched, zero leaves differing after.

An association with extended sequence numbers

2026-10-12, for issue 294's second item; under ESN, what arrives stays with the kernel has the account. The lab tunnel over the 10G path with esp_proposals = aes128gcm16-esn at both ends (the pair shows ESP:AES_GCM_16-128/ESN, the kernel flags=0x400), dp.auto 1, ipsec.flows 0. The sequence is four streams up 16 s, four down 16 s, one up and one down 12 s, pings, then a rekey under a four-stream upload with the peer capturing the numbers.

build	the association	inbound, the peer's frames	outbound, the appliance's frames	four streams up / down, one stream up / down, Mbit/s
c92ca5dd: both directions taken, ext_seq in the request	mirrored, two table rows	decrypted and terminated for 1.36 to 3 million frames, then every frame CRYPTO_DROP_AUTH until a rekey - three runs in four, each after the four-stream upload; across one dead download FROM_WIRE_TO_IPSEC_DECR +13,550, CRYPTO_DROP_AUTH +1,379, TX_DROP +1,379; rx_bad 0, rx_nosa 0, rx_nokey 0	accepted throughout: the peer's replay drops 0, its capture clean at the rekey	935.5 / 21.3 with four timeouts / 0 / 0; the second run 952.9 / 13.3 / 0 / 0; the run with a 5 s pause before the download 953.8 / 824.2 (one stream)
a4c1091e: the arriving association refused, the leaving one taken	one table row, encrypt; the kernel holds both; 23 refusals per pair in sa_refused (every association is offered to all twelve interfaces)	decrypted by the kernel's cipher: its ESP input 151 to 5,861,941, replay 0; rx_done unchanged at 13; no FROM_WIRE_TO_IPSEC_DECR, no CRYPTO_DROP_*	tx_encrypt 3,876,458 = FROM_KN_TO_IPSEC_ENCR +3,876,458; the peer's ESP input +3,876,460, replay drops unchanged; out_drop 0, tx_nosa 0	933.9 / 955.6 / 796.4 / 822.8; pings 3/3 after; the rekey under load 940.7, replay 0, the kernel's cipher numbered 1 and 2 and the coprocessor's begin at 0x100002

The host over the second build's run, the kernel's cipher taking a gigabit arriving: irq88: octep0 45.9%, swi1: netisr 0 30.0%, four more octep0 interrupt threads at 16.7 to 17.4%. The coprocessor over that run: TX_DROP_QUEUE_FULL +126 (the first build's runs saw +1), FROM_WIRE_TO_KN_MFLOW_NOT_ACTIVE +9,739,497. A 1,472-byte DF ping is answered "needs to be fragmented", 1,418 bytes gets three of three. Lab raised 02:30, down 02:34 for the second build (02:13 to 02:27 for the first); the peer's generated swanctl.conf edited and regenerated from its template after, its configuration otherwise untouched, zero leaves differing after each.

Replay windows other than 32

2026-10-12, for issue 294's last item; a window wider than 256 has the account. The lab tunnel over the 10G path with replay_window = N in both children, dp.auto 1, ipsec.flows 0; each row is four streams up 16 s, four down 16 s, one each way 12 s, five pings, the coprocessor's counters read across all of it. Module a4c1091e.

N	the association	inbound, the peer's frames	outbound, the appliance's frames	up x4 / down x4 / up x1 / down x1, Mbit/s
64	mirrored, win 64; kernel replay=8	rx_done +4,500,152, rx_bad 0; FROM_WIRE_TO_IPSEC_DECR +4,500,902, CRYPTO_DROP_REPLAY_OOW +19, TX_DROP_QUEUE_FULL +398	tx_encrypt +2,519,961 = FROM_KN_TO_IPSEC_ENCR; the peer's ESP input +2,519,961, its replay drops unchanged	933.4 / 960.8 / 788.4 / 802.3
128	win 128; replay=16	+4,511,552, 0; +4,512,334, REPLAY_OOW +14, QUEUE_FULL +155	+2,550,740, the peer the same	949.4 / 961.1 / 790.1 / 791.8
256	win 256; replay=32	+4,535,721, 0; +4,536,328, REPLAY_OOW +11, QUEUE_FULL +147	+2,585,429, the same	944.5 / 933.5 / 828.5 / 819.6
1,024	win 1024; replay=128	+4,542,081, 0; +4,542,939, no REPLAY_OOW, QUEUE_FULL +209	+2,568,113, the same	954.7 / 959.7 / 795.2 / 808.8
4,096	win 4096; replay=512	+4,500,636, 0; +4,501,227, no REPLAY_OOW, QUEUE_FULL +128	+2,554,122, the same	947.4 / 957.0 / 788.9 / 791.4
48	refused (not a power of two): one row; kernel replay=6	the kernel's cipher: its ESP input +4,534,348, its replay drops 0 to 32; rx_done unchanged; no FROM_WIRE_TO_IPSEC_DECR	+2,564,493, the same	952.6 / 956.2 / 802.7 / 801.9; host irq84: octep0 45.2 %, netisr 30.4 %
8,192, module ca92801b (the cap at 4,096)	refused: one row; kernel replay=1024	the kernel's cipher: +4,532,578, its replay drops 0; rx_done unchanged; no FROM_WIRE_TO_IPSEC_DECR	+2,558,312, the same	934.4 / 956.3 / 803.6 / 822.4; host irq88: octep0 43.4 %, netisr 27.5 %
4,096, module ca92801b	win 4096; replay=512	+4,474,873, 0; +4,475,369, no REPLAY_OOW, QUEUE_FULL +265	+2,534,903, the same	955.0 / 961.0 / 780.7 / 761.1

Every ping 5/5. The host with the coprocessor's cipher, every row: netisr 30 to 32 %, four octep0 interrupt threads at 25 to 26 %. Lab raised 05:36, the second module installed by reboot at 05:46 with the lab up (the pinger stopped first), down 05:50; the peer's generated swanctl.conf edited and regenerated from its template after, zero leaves differing.

AES-CBC with an HMAC on the coprocessor

2026-10-12, for issue 294's block-cipher item; AES-CBC with an HMAC runs at two thirds of GCM's rate has the account. Module b282ce62, the lab tunnel over the 10G path, the proposal hand-set at both ends, dp.auto 1, ipsec.flows 0; each row four streams up 16 s, four down 16 s, one each way 12 s, five pings, the coprocessor's counters read across it.

proposal	the association	inbound, the peer's frames	outbound, the appliance's frames	up x4 / down x4 / up x1 / down x1, Mbit/s
aes128-sha256 (IV 16, ICV 16)	mirrored, two rows; kernel aes-cbc + hmac-sha2-256	rx_done +3,140,345, rx_bad 0; FROM_WIRE_TO_IPSEC_DECR +3,140,965, CRYPTO_DROP_REPLAY_OOW +45	tx_encrypt +1,965,655 = FROM_KN_TO_IPSEC_ENCR; the peer's ESP input +1,965,655, its replay and authentication drops unchanged	669.9 / 609.4 / 602.4 / 561.1; 1,402-byte ping 3/3
aes256-sha1 (IV 16, ICV 12)	the same; hmac-sha1	+3,173,430, 0; +3,174,098, REPLAY_OOW +44	+1,997,046 = ENCR; the peer's input +1,995,655 (1,391 short; its authentication drops 0, its "queue full" 3,131 cumulative, unread before)	688.4 / 612.1 / 608.9 / 556.9; 1,402 3/3
aes256-sha512 (IV 16, ICV 32)	the same; hmac-sha2-512	+3,286,180, 0; +3,287,001, REPLAY_OOW +40	+2,097,933 = ENCR; the peer's input +2,097,933	714.2 / 632.5 / 633.0 / 556.9; 1,402 refused (DF), the largest inner 1,422
aes128gcm16 the same hour, for comparison				947.4 / 957.0 / 788.9 / 791.4

Every ping 5/5; a 1,472-byte DF ping "needs to be fragmented" on all. The same day, module ca92801b and b282ce62, the two items the driver does not decide (GCM-12 is the kernel's no, transport mode the vendor's):

item	what happens
esp_proposals = aes128gcm12	the appliance selects AES_GCM_12_128, its kernel refuses both SAs (Invalid argument (22)), NO_PROPOSAL_CHOSEN to the peer, no CHILD_SA; sa_refused unchanged
mode = transport, the endpoints as selectors	INSTALLED, TRANSPORT at both ends; no table row, sa_refused +48; pings 5/5 at 0.27 ms; one stream sourced at the appliance 1,526.7 down / 1,303.6 up through the kernel's cipher, its ESP in +2,096,721 and out +2,095,716; rx_done and tx_encrypt unchanged

Lab raised 06:12, the module installed by reboot before it, down 06:17; the earlier GCM-12 cycle 06:03 to 06:04; the peer's generated swanctl.conf edited and regenerated from its template after, zero leaves differing after each.

An IPv6 outer header

2026-10-12, for issue 294's IPv6 item, the outer half; the outer header can be IPv6 has the account. Module c644d91b; the lab tunnel's endpoints on a ULA /64 laid on the 10G path at both ends, the selectors IPv4, dp.auto 1, ipsec.flows 0; each row four streams up 16 s, four down 16 s, one each way 12 s, five pings, the coprocessor's counters read across it.

proposal	the association	inbound, the peer's frames	outbound, the appliance's frames	up x4 / down x4 / up x1 / down x1, Mbit/s
aes128gcm16	mirrored, the table's two rows with the v6 addresses	rx_done +4,502,986, rx_bad 0; FROM_WIRE_TO_IPSEC_DECR +4,503,704, CRYPTO_DROP_REPLAY_OOW +47, TX_DROP_QUEUE_FULL +145	tx_encrypt +2,551,889 = FROM_KN_TO_IPSEC_ENCR = the peer's ESP input; out_nonhop 0	935.9 / 941.1 / 786.4 / 773.2; 1,398-byte ping 3/3
aes128-sha256	the same	+3,102,597, 0; +3,103,086, REPLAY_OOW +41	+1,989,485, the peer the same	674.6 / 589.4 / 594.0 / 502.8
aes256-sha1	the same	+3,156,423, 0; +3,156,998, REPLAY_OOW +45	+2,024,805; the peer's input +2,023,733, its authentication drops 0	691.7 / 589.6 / 604.7 / 517.5

Five of the appliance's CBC frames at the peer: sequence 0x100001 to 0x100005, a different sixteen-byte IV in each. Lab raised 06:30, the module installed by reboot under it twice (06:44 instrumented, 06:54 fixed; the peer's pinger stopped first), down 07:01; the peer's alias and generated swanctl.conf back as they were, zero leaves differing; the three void attempts before the fix are in the finding.

IPv6 inside the tunnel

2026-10-12, for issue 294's IPv6 item, the inner half; IPv6 inside the tunnel, delivered has the account. Module f9f5ba7e; the lab tunnel over its IPv4 10G path with IPv6 selectors (a ULA /64 on the appliance's LAN bridge, the peer's ULA /128), AES-GCM-16, dp.auto 1, ipsec.flows 0; the traffic sourced at the appliance itself over the 10G path.

	reading
pings through the tunnel	5/5, 0.34 ms
one stream up / down 10 s; four streams up / down 12 s	1,820.8 / 1,325.5; 2,136.0 / 1,699.7 Mbit/s
inbound	rx_v6 +7,395,177 delivered to IPv6 input, rx_bad 0, rx_blocked 0; FROM_WIRE_TO_IPSEC_DECR +7,395,602; rx_queuefail ~210,000 (the IPv6 netisr queue at 1.7 Gbit/s into the host's own stack)
outbound	tx_encrypt +7,397,169 = out_taken = FROM_KN_TO_IPSEC_ENCR = the peer's ESP input; its authentication drops 0; out_orig 0
the largest packet	1,398-byte ping passes (inner 1,446), 1,402 refused, not fragmented
the coprocessor; the host	REPLAY_OOW +94, QUEUE_FULL +3; netisr 68.6 %, the load generator 63.8 %, irq85: octep0 54.9 %

The forwarded path the same afternoon (14:40, the machine behind the appliance with a temporary address in the LAN /64): pings 5/5; one stream 760.0 / 802.7, four streams 935.3 / 946.6 Mbit/s; rx_v6 +3,613,105 with rx_bad 0, tx_encrypt +2,084,625 = the peer's ESP input; a 1,398-byte ping lost from the machine where it passes from the appliance. Bisected at 15:00 and 15:18: 1,335 bytes of ping data pass and 1,336 do not - an inner packet of 1,383, which is 1,500 less the kernel's worst-case IPsec header estimate of 117 (ipsec6_check_pmtu with ipsec_hdrsiz_internal: 8 + 64 + 9 + 16 + 20) on every forwarded IPv6 packet; purging the host cache changes nothing (the check writes the route's MTU back into it), the IPv4 twin is off by default (net.inet.ipsec.dfbit 0), and the appliance's own packets skip it. The driver's counters do not move for a refused one; the kernel counts a Packet Too Big per attempt.

Four raises (07:15 to 07:25); the three void ones are the kernel's route-before-policy and the port's ifdisabled, in the finding. The peer's temporary pass in quick on enc0 inet6 rule (its owner's word) was a runtime addition and is gone (0 after the filter reload), its route and alias removed, zero leaves differing.

Two hours of TLS through the tunnel, with the load at the lab's far end

2026-10-12, for issue 279, module f9f5ba7e. The lab peer serves a 1 GiB file of random bytes over HTTPS from its lab alias on the 10G path; the PC downloads it through the appliance and the IPsec tunnel - the coprocessor's AES-GCM both ways, dp.auto 1 - in three concurrent loops for 120 minutes. Two oracles per pass: curl's TLS stack and the file's SHA-256 against the server's.

passes	651, every one rc=0, HTTP 200, 1,073,741,824 bytes, the server's hash
TLS errors, different hashes, other exits	0, 0, 0
per-stream rate	90 to 659 Mbit/s, 365 on average; about 780 Mbit/s aggregate over the two hours
the driver over the run	rx_bad 0, out_drop 0, tx_nosa 0; rx_done to 546.8 million
the coprocessor	RX_IPSEC +521.4 million, TX_DROP_QUEUE_FULL +23,491 (its queue to the host under three downloads; TCP covered it), FROM_WIRE_TO_KN_MFLOW_NOT_ACTIVE +601 million; flows made 156

The first run of the same tool was void from its second pass: the loop clobbered its stream id, three loops wrote one file, and one pass hashed DIFFERENT with a clean TLS exit - the tool's own race, caught in two minutes; the numbers are the restarted run's. Lab raised 12:33, down 14:40 (the longest raise of the lab so far); the server, the file and the certificate lived in the peer's /root/lab, removed with it, zero leaves differing after.

The MTU, from 72 to 9,000, on the spot

2026-10-12, modules dac136a1 and d93f152c; the MTU is the LIF's has the account. The 10G port's MTU set with ifconfig, which now tells the port's LIF; the peer's bridge at 9,000 on the 10G path; the peer's DF pings and the appliance's own TCP.

	reading
what stopped a 1,522-byte frame before	the fast path, at the LIF's MTU of 1,500: FROM_WIRE_DROP_MTU_EXCEEDED +1 per frame, the 10G MAC having passed a 9,014-byte one
the LIF told 9,000 through the instruments	the drops stop; a 9,014-byte frame arrives in six buffers: rx_spans 12, rx_span_bufs 24, rx_span_max 9,088 after twelve pings
ifconfig oxp0 mtu 9000 on d93f152c, DF pings of 1,472 to 8,972 data bytes	all answered; 8,973 (a 9,015-byte frame) refused; rx_spans 2,029,355 over the run, rx_span_nobuf 0
TCP from the appliance, one stream 8 s; four streams 10 s	7,386.6 / 7,178.4; 8,703.6 / 8,876.7 Mbit/s
the port back at 1,500	1,853.3 / 2,323.5 one stream; a 1,522-byte frame refused again
through the tunnel at 9,000: the largest DF ping; TCP one stream / four; the driver	8,918 answered, 8,922 refused (inner 8,946 = 9,000 less 54); 2,007.9 / 1,502.8 and 2,366.4 / 1,856.8 (one up stream timed out); rx_bad 0, tx_encrypt = ENCR +6,605,231, rx_spans +2 - the peer's MSS clamp of 1,406 on its bridge keeps the segments small
the first build, dac136a1	the slot's physical address on the old stride: every transmit read from the wrong slot, no port passed a frame, the appliance's WAN gone for three minutes until d93f152c

The lab's alias on the peer added and removed around each run, the tunnel raised 16:02 and down 16:04, zero leaves differing.

The clear-text path at a gigabit, through the host and with the flow table

2026-10-12, 17:15 to 17:20, module d93f152c, for issue 320 (plain forwarding through the host had never been run: every gigabit row before this was through the tunnel) and issue 315 (a plain connection given back by the fast path, and whether it is revived). A far end OUTSIDE the tunnel's selectors: a second /30 on the lab wire, the peer's alias on its bridge, the appliance's on the 10G port, the peer routing the LAN behind the appliance back by it, and a TCP load generator's server on the peer's alias. The LAN machine's traffic to it is routed through the host in the clear - the peer's capture shows the echo request and reply as plain ICMP on its bridge - and NAT is not in it. The first attempt of the day (01:47) had stopped at the peer's default block: its LAN pass rule names its own network only. A temporary pass rule on the peer for the one source network to the one address, runtime only and dropped by a filter reload at the end, was the whole difference.

Four streams for 16 s and one stream for 12 s, each way, from the LAN machine (gigabit port behind the switch) to the peer (10G port); the driver's receive counters and the coprocessor's across each regime:

dp.auto	four up / four down	one up / one down	the driver	the coprocessor
0, through the host	957.1 / 964.0 Mbit/s (238-241 and 183-312 a stream)	783.8 / 810.7	rx_done +6,746,841; oq_behind, oq_unseen, rx_resync, rx_short, oq_lapped, rxwd_stalls, credit_capped all 0	RX_WIRE +6,747,465, TX_WIRE +6,747,213, MFLOW_NOT_ACTIVE +6,747,196 (every frame to the host), TX_DROP_QUEUE_FULL +48
1, the flow table	955.1 / 964.8 (238-240 and 195-268)	771.8 / 789.4	rx_done +28,703; flow_revived +42, revive_refused 0, flow_wait_other 0	FROM_WIRE_TO_WIRE +6,630,528 of RX_WIRE +6,657,897 (99.6 %), TCP_MAX_RETRANS +43, CONN_RECLAIM_PENDING +25,181, TX_DROP_QUEUE_FULL +75

So the host forwards a gigabit of plain TCP by itself, four streams at the line's rate and one stream at 78 to 81 % of it, with the receive path's fault counters at zero - the row #319's rows through the tunnel had only implied. The flow table adds nothing to the rate here (the line is the LAN machine's gigabit port) and takes the host out of 99.6 % of the frames.

A plain connection given back, and revived (#315's second item). One stream for 30 s each way with the trigger on, the counters across each:

	rate	the fast path's rule	the driver	frames handed up while it waited	in hardware
download (the peer sends, 10G into the gigabit port)	795.6 Mbit/s	TCP_MAX_RETRANS +17	flow_revived +17, revive_refused 0	CONN_RECLAIM_PENDING +17,920 - about 1,050 a give-back, 16 ms at this rate	2,291,280 of 2,310,063, 99.2 %
upload (the gigabit port sends)	753.8	0	0, 0	+3	3,872,719 of 3,873,314, 99.98 %

The rule fires on the download only, where the 10G side pours into the gigabit port and the coprocessor's queue to that port drops a frame now and then (TX_DROP_QUEUE_FULL +4 in the 30 s); the retransmissions that follow are what the rule counts. Every one of the seventeen was revived in place, none refused, and the connection was back in hardware about a thousand frames later each time. The 42 of the four-stream run above are the same thing, 43 fired and 42 revived - the one not revived was a connection at its end. On the upload nothing retransmits and nothing is given back. That is the plain connection's behaviour, and it is the tunnelled one's.

Thirty downloads across the WAN, the real round trip (#315's first item). The lab has no delay element at its far end and a pipe on the appliance would delay the host path only, so the one path with a real round trip is the appliance's WAN: the cycle of the ten-hour run (a download from a public speed-test server, one at a time, cut at six seconds, the trigger on), thirty of them back to back, 17:22 to 17:25, the counters read after each.

downloads	30, 6.5 to 68.4 MB in the six seconds, 9 to 95 Mbit/s, three servers in turn
TCP_MAX_RETRANS, flow_revived, revive_refused	0, 0, 0 - no download was given back for the rule
CONN_RECLAIM_PENDING	on 15 of the 30, +4 to +564 frames, 3,553 in all, each with TCP_FIN_SYN_RST +1 at the cut: the connection handed back on curl's close, and the server's segments still in flight arriving while it waited to be reclaimed
the ten-hour run of 2026-10-08, same cycle	1 of 60 given back

At 40 ms and under 100 Mbit/s the rule does not fire at all now; the issue's "one in ten" was two builds ago. What a long burst of duplicate acknowledgements does to the wait that doubles stays unmeasured, because no path here makes one: it needs tens of milliseconds AND a rate that retransmits, and the lab's far end is the owner's production firewall, which takes no pipe.

The peer: alias, route, server and the temporary rule in place 17:15 to the end of the TLS soak that followed on the same path; the rule dropped by a filter reload, zero leaves differing after.

Three hours of TLS on the clear-text path, with the trigger on and then off

2026-10-12, 17:27 to 20:28, for issue 279, module d93f152c: the soak of the two hours through the tunnel above, run on the clear-text path of the section before this one - plain forwarding through the host, which is the path of the two failures of 2026-10-06 - for 90 minutes with the trigger on and 90 with it off. Three concurrent download loops of the 1 GiB random file over HTTPS from the peer's second alias; curl's TLS stack and the file's SHA-256 the oracles, every pass recorded.

	trigger on, 17:27 to 18:57	trigger off, 18:57 to 20:28
passes	524, every one rc=0, HTTP 200, 1,073,741,824 bytes, the server's hash	524, the same
TLS errors, different hashes, other exits	0, 0, 0	0, 0, 0
per-stream rate	105 to 697 Mbit/s, 397 on average; about 830 aggregate (the LAN machine's gigabit port)	92 to 765, 402; about 830
the coprocessor	FROM_WIRE_TO_WIRE +439,136,341; TCP_MAX_RETRANS +3,741; CONN_RECLAIM_PENDING +843,479; MFLOW_NOT_ACTIVE +297,721; TX_DROP_QUEUE_FULL +19,917; 2,626 flows made	MFLOW_NOT_ACTIVE +442,890,333 - every frame to the host; TX_DROP_QUEUE_FULL +20,504
the driver	flow_revived +3,740, revive_refused 0	rx_done +439.1 million, intr_taken +75.7 million; the three busiest ring handlers at 21, 20 and 11 % of a core
the receive path's fault counters	oq_lapped, rxwd_stalls, oq_behind, oq_unseen, rx_resync, credit_capped all 0	all 0

1,048 GiB in the clear through the appliance at the LAN machine's line rate, half of it through the host frame by frame, and nothing changed a byte. With the 651 passes through the tunnel that morning, 1,699 passes and no failure on this build. The two failures of 2026-10-06 were at 200 to 300 Mbit/s on the WAN's NAT path, through the host; the second half here covers that path and that rate, NAT aside, three streams at once, and does not reproduce them. The issue's criterion is a month; this is three hours of it. The month was a number set on the day of the failures and derived from nothing; it was withdrawn the evening of 2026-10-12 on the owner's word for the criterion the failures themselves set - their own path and rate, translated through the host at 200 to 300 Mbit/s a stream, the trigger on and then off, clean for two hours - and that run is the next section.

Two hours of TLS on the failures' own path: translated, through the host

2026-10-12, 21:17 to 23:18, module d93f152c, issue 279's closing run under the criterion above. The path of the two failures of 2026-10-06, rebuilt on the lab wire: the appliance translates the LAN behind it to its own address on the 10G port (a runtime nat on oxp0 rule in the translation section of the loaded ruleset, dropped by a filter reload after), the peer sees one host and its temporary rule passes that one host to the one alias - the peer's capture shows the echo request from the translated address, the appliance's state table shows the translation, the flow table's connections carry do_snat. The same three download loops of the 1 GiB random file over HTTPS, the same two oracles, an hour with the trigger on and an hour with it off.

	trigger on, 21:17 to 22:17	trigger off, 22:17 to 23:18
passes	349, every one rc=0, HTTP 200, 1,073,741,824 bytes, the server's hash	343, the same
TLS errors, different hashes, other exits	0, 0, 0	0, 0, 0
per-stream rate	95 to 742 Mbit/s, 398 on average; about 830 aggregate	82 to 722, 396; about 810
the coprocessor	FROM_WIRE_TO_WIRE +292,202,734; TCP_MAX_RETRANS +2,368; CONN_RECLAIM_PENDING +479,228; MFLOW_NOT_ACTIVE +130,967; TX_DROP_QUEUE_FULL +13,028	MFLOW_NOT_ACTIVE +289,592,336 - every frame to the host; TX_DROP_QUEUE_FULL +13,598
the driver	flow_revived +2,368 = the give-backs, revive_refused 0	rx_done +289.6 million
the receive path's fault counters	oq_lapped, oq_behind, oq_unseen, rx_resync, credit_capped 0; rxwd_stalls 3 - below	all 0, rxwd_stalls +0

692 GiB through the failures' own path at the LAN machine's line rate, 200 to 740 Mbit/s a stream, translated by the host for an hour and translated and accelerated for another, and nothing changed a byte. With the day's other runs, 2,391 passes and no failure on this build. The criterion is met and the issue is closed on it; what would reopen it is the thing itself, recurring, with the capture its body asks for.

Three rings left with a burst, all with the trigger on (issue 361). rxwd_stalls, zero through every run of the day until then, counted three in the first hour:

when	ring	first pass	idle before	the visit took
21:33:37	7	53 packets	234 ticks	53
21:57:01	5	32	927	47
21:59:50	0	92	67	99

Each is a burst that landed on a ring nobody was inside - idle for 67 to 927 ticks - and raised no handler until the watchdog's visit, at most one period later. None at a rekey (the association was seventeen minutes old at the first and its rekey forty minutes away); none in the hour with every frame through the host, where no ring is ever idle; none in the three hours on the clear-text path. So it is not the load but an idle ring that a burst finds, and the trigger on is what leaves rings idle. TCP covered the delay every time - the passes around each were clean - and oq_bound did not move. It is the shape the fourth section of #320 left as a reading, and it is now an issue with a log of its own.

The peer: alias, rule, server in place 21:16 to 23:19; both temporary rules dropped by a filter reload at each end (0 left at both), the aliases gone, /root/lab gone, zero leaves differing, its WAN-path tunnel re-initiated.

A ring left with a burst: two builds that asked the block what it saw

2026-10-13, for issue 361, the same trigger-on soak on the translated path as above, an hour at a time, on two builds of the working tree (neither committed):

build	what it adds	passes	stalls in the hour
d93f152c (main), the night before	nothing	349	3
c3d2f133	RESEND (R_OUT_CNTS bit 59) written when a visit leaves a ring, the vendor host's end-of-poll write; dp.resend, dp.resend_writes (on by default in the three instrument builds; off by default in the committed driver, 2026-10-14, because of this row)	338, 923,193 writes	5 - rings 5, 4, 2, 7, 4; idle 33 to 1,179 ticks; first passes 37 to 49
9df38c7b	R_OUT_INT_STATUS, R_OUT_CNTS and R_OUT_INT_LEVELS read before every watchdog visit and kept with the stall record	362	1 - ring 0, first pass 56, idle 128 ticks, visit 67
e13e7292	the ring's MSI-X vector read the same way - table mask bit, PBA pending bit, index, handler entries - and dp.istat_reset, passes that found the status set again right after clearing it	133 in 22 minutes (cut by the bench)	1 - ring 5, first pass 33, idle 309 ticks, visit 33

So RESEND changes nothing, which says the count was not sitting at the level when a burst began - the ring was empty, the burst crossed the level, and still no handler ran within a watchdog period. And the one stall the third build caught says why, in the block's own registers as they stood before the watchdog touched the ring:

before it: int_status 0x1  cnts 0x4000143c00000038  levels 0x5600000001

int_status 0x1: the block's latched interrupt status was set. cnts: 0x38 = 56 packets in the low word, the register's own "output interrupt" flag (bit 62) set in the high one. levels: a packet threshold of 1. The condition was there, latched, for 128 ticks with 56 packets behind it, and the handler did not run. This is not the block failing to raise the interrupt; it is the interrupt raised and not acted on - the MSI-X message not delivered, or delivered to a vector that did not run the handler. The next instrument is on that side: the vector's MSI-X mask bit and its PBA pending bit and the vector's count from the kernel, read at the same moment, and a second read of the status right after a visit's clear, for a clear that lost a race with the block setting it. The fourth build read exactly that, and its one stall says:

before it: int_status 0x20  cnts 0x40001ccf00000021  levels 0x5600000001
           vector 21  mask 0  pending 0  handler entries 88541

Ring 5's status set, 33 packets against a threshold of 1; the ring's vector (16 + 5) not masked and nothing pending in the PBA. No message was in flight and none was held back by the host. And the new counter says why a set status is not by itself a message: over the 37 minutes to the end-read, istat_reset rose by 328,303 against 256,384 interrupts taken - more passes found the status set again right after clearing it than there were interrupts. The write-one-to-clear is not final while the count stands at or above the level: the block sets it again at once, or swallows the clear. The message comes on the status's edge, and a status that never went down gives no edge. So the stall is a ring whose status never went down between a visit's last clear and the burst - where in the visit that happens is what the next instrument has to catch: at every visit's end, the status and the count as the visitor leaves them, and the vector's entry count, kept per ring and printed with the stall, so the record says whether the handler ran at all between that visit and the watchdog's. Every pass of all four runs was clean, so the net under this is still a net.

The lab for these: raised 20:12, 21:28 and 22:48 (PC clock) with the two temporary rules, down 21:18, 22:29 and 23:26, zero leaves differing each time, the WAN-path tunnel re-initiated. The fourth run was cut at 23:10 by the bench - the soak's source address was moved to another appliance's port for that appliance's work - so its 133 passes are 22 minutes, not an hour.

The fast path's give-backs across the first half (#315): 3,741 times the retransmission rule gave a connection back and 3,740 times the driver revived it in place - the one not revived a connection at its end - with 843,479 frames handed up while the connections waited, 225 a give-back on average. 2,626 flows were made for 524 downloads over three loops: about five a download, which is the give-backs' share - one new flow for a connection taken out and attached to again - against the 3,741 revivals that cost no new flow.

The watchdog under a gigabit through the host (#320's second and third sections). With the trigger off and three streams through the host - 81,000 frames a second - oq_declined rose at 13 a second (+70,306 over the half): the watchdog finding a ring's handler at work and leaving its note. With the trigger on it was 0.7 a second. rxwd_runs +42,263 and rxwd_rescues +50,752 over the same half are the watchdog arriving on a loaded ring between an interrupt and its handler, and rxwd_stalls stayed 0: no ring was ever left with packets in it. oq_handed +4,031 is a servicer finding such a note on its way out and going round again. oq_bound did not move all day, 126 before the first gigabit row and after the last: no visit reached its bound, so the regime of the third section was not entered at a gigabit, and the RESEND question of the fourth stays a reading.

The peer: alias, route, the two servers and the temporary rule in place 17:15 to 20:29; the rule dropped by a filter reload (0 left, 0 states to the alias), the alias and the route gone, /root/lab gone, zero leaves differing, its WAN-path tunnel re-initiated from its side. The longest raise of the lab so far, three hours and fourteen minutes.

How to take these again

Everything here is taken with four things in tools/, which are meant to be copied to the appliance and run there:

blast.c	a UDP sender that does not wait, and paces itself when given a rate
octep-fpcnt.sh	the coprocessor's 182 counters, with their names
octep-fairness.sh	the sweep above: what load on one port costs the others
octep-soak.sh	hours of traffic, with a leak, a stall and a starvation test in the log

fpcntr-names.txt is the list those names come from - the enum in the shipped module's DWARF, in the array's own order. Nine of its indices were already known from live traffic before the list existed and every one of them lands on the right name, which is the reason to trust the other 173. Without it a counter read is a column of integers: index 18 means nothing until it reads FPCNTR_TX_WIRE_ERR.

The generator binds a source address, sends to a destination port nothing listens on, and never waits:

blast <bind-ip> <dst-ip> <payload-bytes> <seconds> [packets-per-second]

Point it at a port with no cable, with an address on that interface and a static ARP entry for a neighbour that does not exist, so nothing is ever asked for and nothing leaves the appliance:

ifconfig oxpN inet 198.51.100.1/24 alias
arp -s 198.51.100.2 02:00:00:00:00:02
...
arp -d 198.51.100.2 ; ifconfig oxpN inet 198.51.100.1 -alias

Read the counters with netstat -i -b -n, and mind the columns: Ipkts is $5, Ibytes is $8, Opkts is $9, Obytes is $11. $7 is Idrop, and reading it as Opkts once produced a confident wrong conclusion that the transmit counters were dead.

The far side's named counters come over rpc command 44 with rpc.num_entries set - see octeon-tx.md for the reader and for the 182 names in order.

Discussion

The discussion of this page is held in the project's GitHub Discussions. Showing it here loads giscus from giscus.app and GitHub, and writing in it needs a GitHub account - what that means.

Show the discussion