Skip to content

Measurements

Unusual traffic, reported, measured, 2026-10-19

XGS: Settings - Watch for unusual traffic. Off until it is turned on, and it never blocks: it says, in the XGS log, what no normal day looks like. Every minute scripts/xgs/anomaly.py (cron) reads pf's state table every ten seconds through a window of fifty, keeps every connection any reading saw, and for each host that opened connections works out what it did in the window - bytes sent and received, connections opened, new addresses reached, and the most ports it tried on any one address. Then:

Does the coprocessor drop an address it is given? Measured, 2026-10-19

No - not on the XGS 3300's firmware (v22). The question came from the vendor's own design: its command list has three commands between the QoS ones and the first IPsec one - DOS_SET_POLICY, DOS_ADD_BLACKLIST_ENTRY and DOS_CLEAR_BLACKLIST, 27, 28 and 29 - with their structures in dos_table.h: a policy (a packets-per-second ceiling, a type, whether a source over the ceiling is blacklisted, and for how long) and an entry (a family, a type, an address, a duration). The XGS 136's helper module was built with two more facts about them - an even type is a source and the next odd one its destination; an entry lasts 30 seconds by default and 3,600 at most - and with an error code for a full DoS table. If the coprocessor kept such a table, a block list could be enforced there, at the port, before a frame reached the host.

What the coprocessor counts of a connection, measured, 2026-10-19

Once a connection is in hardware the host stops seeing it: its frames go from port to port inside the coprocessor, and pf's state, the interface counters and every graph built on them stop growing - which is why every throughput on these pages is read at the far end, not on the appliance. The coprocessor itself counts every connection it forwards, and the XGS 3300's sends those counts to the host unasked, about once a second, as control messages on tag 254 - which the driver had dropped unread since the receive path was written. Since this change it reads them (octep_dp_cmsg_rx in contrib/octep/octep_dp.c): each connection in the table keeps the packets and bytes the coprocessor counted for each direction, dev.octep.0.dp.flow_table prints them, and the Offloaded Flows page shows them as Counted in hardware.

Connections cut by their TLS fingerprint (JA4), measured, 2026-10-19

XGS: Settings - Cut connections by TLS fingerprint. JA4 is a fingerprint of a TLS ClientHello - the version, ciphers, extensions and protocols a client offers - so it names the software that opened a connection rather than the address it went to. OPNsense's Intrusion Detection (Suricata 8) writes one for every TLS connection it sees once its TLS log carries the JA4 field. With fingerprints listed here, scripts/xgs/ja4_cut.py reads that log every two seconds (started each minute by cron) and when the settings are saved, and kills by id every pf state of a connection it shows with a listed fingerprint; the driver then takes the connection out of hardware, as it does for a block list (lists-cut-2026-10-19.md).

A block list that cuts the connections already open, measured, 2026-10-19

XGS: Settings - Cut open connections when listed. A block list in OPNsense is an alias - a pf table filled from a URL, a feed or by hand - and a rule that blocks it blocks new connections only. A connection opened before its address was listed keeps its pf state, and with offloading on its flow in the coprocessor, for as long as it carries traffic. With a list chosen here, scripts/xgs/lists_cut.py compares it every minute (cron) and when the settings are saved with what it held before, and kills by id every pf state with either end in what was added; each driver then takes the connection out of hardware. Choosing a list cuts what it already names.

Slow minutes on the XGS 136, looked for, 2026-10-19

Four times on 2026-10-19 a stream from the bench PC through the XGS 136 to the second XGS 3300 ran at a quarter to a third of its speed for a short while (#531, open). This page is what was done to find the cause, and what was found on the way. The cause has not been found: in 28 minutes of watching and one boot watched, it did not happen again.

What the XGS 136's coprocessor says about the traffic it forwards, measured, 2026-10-19

The XGS 3300's coprocessor sends the host the counts of every connection it forwards, and of every port, as control messages on tag 254 - which the driver now reads and gives to pf and the interfaces (hw-counters-2026-10-19.md). The XGS 136 is as blind to its own hardware traffic as the XGS 3300 was: once a connection is in hardware, pf's state and the interface counters stop growing. This page is the first step of the same work on the XGS 136: does its coprocessor send such messages at all, and in what form.

A connection given back by the XGS 136's fast path, and made again at once, measured, 2026-10-19

When the second XGS 3300 sent to the bench PC through the XGS 136, a stream in hardware ran slower than the same stream through the host, and retransmitted far more. The fast path kept giving the connection back, and the driver kept making it again at once - and making it again was what gave it back. The driver now waits, with the connection left with the host, before it makes one the fast path gave back soon after it was made. In that direction the hardware path now runs level with the host's, and a minute of it retransmits a quarter of what it did (#529).

Three appliances on one bench, 2026-10-17

The first measurements with a hardware peer at both ends of every tunnel: until this night the far end of a measured tunnel was a firewall doing ESP in software, so a figure could be that firewall's rather than the appliance's. Every window a number came from was photographed when the run finished, and the pictures are below each table.