Skip to content

Measurements

What the coprocessor counts of a connection, measured, 2026-10-19

Once a connection is in hardware the host stops seeing it: its frames go from port to port inside the coprocessor, and pf's state, the interface counters and every graph built on them stop growing - which is why every throughput on these pages is read at the far end, not on the appliance. The coprocessor itself counts every connection it forwards, and the XGS 3300's sends those counts to the host unasked, about once a second, as control messages on tag 254 - which the driver had dropped unread since the receive path was written. Since this change it reads them (octep_dp_cmsg_rx in contrib/octep/octep_dp.c): each connection in the table keeps the packets and bytes the coprocessor counted for each direction, dev.octep.0.dp.flow_table prints them, and the Offloaded Flows page shows them as Counted in hardware.

Connections cut by their TLS fingerprint (JA4), measured, 2026-10-19

XGS: Settings - Cut connections by TLS fingerprint. JA4 is a fingerprint of a TLS ClientHello - the version, ciphers, extensions and protocols a client offers - so it names the software that opened a connection rather than the address it went to. OPNsense's Intrusion Detection (Suricata 8) writes one for every TLS connection it sees once its TLS log carries the JA4 field. With fingerprints listed here, scripts/xgs/ja4_cut.py reads that log every two seconds (started each minute by cron) and when the settings are saved, and kills by id every pf state of a connection it shows with a listed fingerprint; the driver then takes the connection out of hardware, as it does for a block list (lists-cut-2026-10-19.md).

A block list that cuts the connections already open, measured, 2026-10-19

XGS: Settings - Cut open connections when listed. A block list in OPNsense is an alias - a pf table filled from a URL, a feed or by hand - and a rule that blocks it blocks new connections only. A connection opened before its address was listed keeps its pf state, and with offloading on its flow in the coprocessor, for as long as it carries traffic. With a list chosen here, scripts/xgs/lists_cut.py compares it every minute (cron) and when the settings are saved with what it held before, and kills by id every pf state with either end in what was added; each driver then takes the connection out of hardware. Choosing a list cuts what it already names. Since 2026-10-23 the connection is dropped on the coprocessor itself first, by its verdict, and kept there while its sender goes on - XGS: Settings Cut in hardware, on by default (measured).

Slow minutes on the XGS 136, looked for, 2026-10-19

Four times on 2026-10-19 a stream from the bench PC through the XGS 136 to the second XGS 3300 ran at a quarter to a third of its speed for a short while (#531, open). This page is what was done to find the cause, and what was found on the way. The cause has not been found: in 28 minutes of watching and one boot watched, it did not happen again. 2026-10-20: it is not the XGS 136. After an idle hour the 136 forwarded at the link's rate from the first second both ways with the PC out of the path, while the PC was slow on two paths at once - one of which does not cross the 136 at all (the runs).

What the XGS 136's coprocessor says about the traffic it forwards, measured, 2026-10-19

The XGS 3300's coprocessor sends the host the counts of every connection it forwards, and of every port, as control messages on tag 254 - which the driver now reads and gives to pf and the interfaces (hw-counters-2026-10-19.md). The XGS 136 is as blind to its own hardware traffic as the XGS 3300 was: once a connection is in hardware, pf's state and the interface counters stop growing. This page is the first step of the same work on the XGS 136: does its coprocessor send such messages at all, and in what form.

A connection given back by the XGS 136's fast path, and made again at once, measured, 2026-10-19

When the second XGS 3300 sent to the bench PC through the XGS 136, a stream in hardware ran slower than the same stream through the host, and retransmitted far more. The fast path kept giving the connection back, and the driver kept making it again at once - and making it again was what gave it back. The driver now waits, with the connection left with the host, before it makes one the fast path gave back soon after it was made. In that direction the hardware path now runs level with the host's, and a minute of it retransmits a quarter of what it did (#529).

Three appliances on one bench, 2026-10-17

The first measurements with a hardware peer at both ends of every tunnel: until this night the far end of a measured tunnel was a firewall doing ESP in software, so a figure could be that firewall's rather than the appliance's. Every window a number came from was photographed when the run finished, and the pictures are below each table.