Where a connection sent to inspection goes, measured - and then the inspector made, 2026-10-23

A connection on the coprocessor has a third verdict beside forward and drop: send to inspection (0). The fast path then hands the connection's frames to an inspector instead of the wire, and the inspector hands each one back to be sent on - the shape an IPS needs to see a connection's start and let the rest run in hardware (the discovery, 680). Before building anything on it, this is where those frames go on an appliance running OPNsense.

The fast path hands them over, on both families, and nothing on this host receives them. Every frame of a connection set to inspection was counted as sent to the inspector, with no error; none reached the far end, none came back, and none reached the host's own receive rings. The inspector the fast path sends to is a PCIe virtual function, and OPNsense makes none made none - until the afternoon, when the XGS 3300's driver made one and Suricata inspected connections in hardware through it (below).

	stream	sent to the inspector in six seconds	errors sending to it	back from it	to the far end meanwhile	set back to forward
XGS 3300	TCP	30	0	0	0 Mbit/s	3,720 Mbit/s
XGS 3300	UDP, 300 Mbit/s	167,677	0	0	0 Mbit/s	338 Mbit/s
XGS 136	TCP	48	0	0	0 Mbit/s	1,006 Mbit/s
XGS 136	UDP, 300 Mbit/s	151,867	0	0	0 Mbit/s	323 Mbit/s

The modules measured: octep.ko e0dd1809 on the second XGS 3300 and npuep.ko b1cc86a8 on the XGS 136 - the builds of the verdict's measurement with one change, their dp.flow_verdict and flow.verdict instruments taking 0 as well as 1 and 2. Nothing else in either driver sets a connection to 0. Since the afternoon the XGS 3300's driver makes a connection with 0 when it is asked to and something inspects (below).

Where the inspector is

In the vendor's own system the inspector is a program on the host that reads the coprocessor's virtual functions from user space:

• On the XGS 3300 the vendor's host driver enables SR-IOV on the coprocessor's PCIe endpoint and makes eight virtual functions (177d:a303, sriov_numvfs 8), one ring each - which is why its own rings start at 8 (read under the vendor's system on 2026-09-29).
• On the XGS 136 the coprocessor's startup script sets up four (giu-1 to giu-4, fixed for the XGS 116 to 138), and the vendor's host makes them as SR-IOV functions (11ab:7081) and gives them to vfio-pci, for DPDK.
• The metadata the fast path puts in front of each frame for it is in the vendor's metadata_dpi.h: sixty-four bytes headed by the magic 0xC352A8B1, the session's identity and revision, the direction, and room for the inspector to say it rewrote the frame and wants new checksums. "USFP metadata structures and defines shared with DPI", "USFP <-> DPI (DPDK DAQ)": the inspector reads it through a DPDK packet-acquisition module. A frame comes back with forwarding metadata, and the fast path sends it on (FROM_IPS_TO_WIRE, proc_from_ips).
• The vendor's host picks the function for each connection from a table of inspector instances (read_ips_lb_tbl), and writes its number into the connection's ips_vf field.

On OPNsense neither driver makes a virtual function. octep tells the coprocessor it has none (num_vfs 0 in the endpoint handshake); telling it eight without making them took the coprocessor's side down on 2026-09-29. The XGS 136's coprocessor sets up its four regardless, and nothing on the host services them. So a frame sent to inspection is sent to a function that is not there, and is lost: today until the XGS 3300's driver made one, the verdict 0 is was a drop by another way, and nothing here uses used it.

What using it would take

1. The virtual functions themselves, on both families: SR-IOV in octep and npuep (FreeBSD's pci_iov), and something on the host that services their rings.
2. A go-between for Suricata, which knows nothing of the coprocessor's metadata: take the sixty-four bytes off each frame, give Suricata the Ethernet frame (netmap, as OPNsense runs it), and send each frame Suricata passes back with its metadata - matched to it, since Suricata may drop or hold frames.
3. A way for Suricata to say it is done with a connection, so the connection's verdict can be set to forward and the rest of it run in hardware. Suricata's bypass has no outside hook in the netmap capture OPNsense uses.

None of that is started. All three were made on the XGS 3300 the same day - below; the XGS 136 is next. There is a nearer way to the same end with what OPNsense already has - a connection kept on the host until the IPS has seen what it inspects, then put in hardware - and it has not been measured either: Suricata's IPS mode, over netmap, has never been run on these drivers' interfaces. Suricata's IPS mode over netmap now runs on the inspector's interface below.

Then made, on the XGS 3300

The same afternoon, on the second XGS 3300, as the vendor's source does it - and off unless it is asked for (the marker /var/db/os-xgs-npu/inspector-vf, read at boot):

1. One virtual function, on ring 0. The PF publishes one function of one ring in SDP_EPF_RINFO, gives ring 0 to it in R_VF_NUM (which this driver never wrote), switches SR-IOV on and lets the function master the bus, and starts its own rings at 1, its pkind at 41 (40 plus the functions). The coprocessor took it - two host ports started where there had been one, and its fast path, its RPC and its heartbeat all answered. On 2026-09-29 eight functions published without being made had taken it down; made, one did not.
2. The frames arrive. With the function's ring armed, a connection set to inspection has its frames on this host, each behind the vendor's usfp_dp_md exactly as metadata_dpi.h lays it out: the magic 0xC352A8B1, both checksums good, IPS and forwarding blocks valid, the connection's own index as the session, the IP header at 14, and in the forwarding block the next hop's two MACs, the egress port and its MTU. The frame behind it is the one that arrived from the wire.
3. Handed back, they go on. Posting the same sixty-six bytes and the frame on the function's input ring, with pkind 40, the fast path takes them back (RX_IPS) and sends them to the wire (FROM_IPS_TO_WIRE), nothing dropped.
4. Suricata reads them. The driver makes oxips0, the inspector's interface: a frame for the inspector goes to it as input with its sixty-six bytes kept by the driver, keyed by the frame's addresses and ports. Suricata in IPS mode over netmap takes the interface's input over while it runs, and netmap hands each frame it lets through to the interface's own input function - the driver's, which puts the bytes back in front and hands it back. Nothing in Suricata changes.
5. Then hardware. With dp.ips_new, a new connection is made with the verdict "send to inspection" while something reads oxips0; once the inspector has let dp.ips_depth bytes of it through - 1 MiB, Suricata's own stream depth - the driver sets it to forward and the rest runs in hardware. With nothing reading oxips0, a frame for the inspector is handed straight back and its connection set to forward: an inspector that stops costs its connections nothing.

on the XGS 3300, octep.ko
946415d4	a TCP stream set to inspection, every frame through this host and back	4,105 to 4,856 Mbit/s, nothing dropped
946415d4	a UDP stream at 300 Mbit/s, the same	341 to 344 Mbit/s - all of it
4d04264a	Suricata 8.0.7 on oxips0, one rule dropping UDP to port 5204: the UDP stream set to inspection	blocked - 203,608 drop events, nothing delivered
4d04264a	the same, a TCP stream no rule names	2,943 and 3,230 Mbit/s through Suricata
1d1f2d08	dp.ips_new: a new TCP stream	731 to 811 frames through Suricata, then forward: 3.82 to 3.85 Gbit/s overall
1d1f2d08	dp.ips_new: a new UDP stream to the dropped port	dropped by Suricata from the moment it was in hardware - 205,338 and 205,642

The function costs the forwarding nothing measured. With the same module after a reboot each way, a TCP stream through the XGS 3300 ran at 3.82 Gbit/s without the function and 3.97 with it, UDP offered at 2 Gbit/s 1.44 and 1.66. One boot with the function ran slow - 532 Mbit/s plain, about 100 inspected - and the same steps after the next boot did not; why, is not known.

How to turn it on, while it is an experiment

There is no setting yet. On an XGS 3300 with this driver:

touch /var/db/os-xgs-npu/inspector-vf     # then reboot: the function is made at boot
ifconfig oxips0 up
sysctl dev.octep.0.dp.ips_new=1           # new connections inspected while Suricata reads oxips0

and Suricata in IPS mode on oxips0, its netmap section pairing the interface with its host side:

netmap:
  - interface: oxips0
    copy-mode: ips
    copy-iface: oxips0^
  - interface: oxips0^
    copy-mode: ips
    copy-iface: oxips0
stream:
  midstream: true

midstream, because a connection's first frames cross the host before it is made in hardware, so Suricata on oxips0 meets it already open; the measurements ran Suricata by hand beside OPNsense's own IDS service rather than through it (images/deploy/inspect-suricata.yaml, images/deploy/lab-inspect-then.sh). Counters: dp.ips_rx, ips_up, ips_passed, ips_back, ips_nomd, ips_dropped, ips_made, ips_cut.

What it does not do yet

• The XGS 136: its coprocessor sets up four functions of its own; the host makes none yet.
• A setting, and OPNsense's Intrusion Detection service reading oxips0: the interface is not assigned, and the service only lists assigned ones.
• A connection's first frames, before it is in hardware, cross the host and are not on oxips0; Suricata sees them only if it also inspects the front ports.
• A frame longer than one receive buffer, and a frame that is not IP, is not inspected - it is dropped and counted; a later fragment of a packet is keyed without ports and comes back without its bytes.
• A tunnel's connections are made forward, as before.

How it was measured

images/deploy/lab-inspect.sh, each run in a window of its own and photographed (images/deploy/lab-inspect-series.ps1); the text of every run, the ones before the photographs among them, is in the run log. One iperf3 stream from the PC through the appliance, made in hardware with the action IPS and the fast path's IPS switch on - the "Cut in hardware" default - so its verdict is asked for every frame. At second 10 its verdict set to 0 with the driver's instrument, at second 16 back to forward (2). Read before and after: the fast path's own counters for the inspector's path (FROM_WIRE_TO_IPS_INSPECT, TX_IPS, TX_IPS_ERR, RX_IPS), what reached the far end, frames the host received with a port tag no interface claims - where a frame for a function the host does not know would land - and the last such frame itself. Through the six seconds those were only the coprocessor's control messages (tag 254), at their usual rate of about one a second.

The figures in the table are from the runs before the photographs, in the run log; the photographed runs are the same measurement made again.

What it found on the way

A run can pick the last run's connection. The script takes the stream's connection as the one with the most frames, and a TCP stream finished seconds before was still in the XGS 136's table with more frames than the new UDP stream - so the first UDP run set the old connection to inspection, and the stream went on at 326 Mbit/s. The script now passes over every connection that was in the table before its stream started.

Discussion

The discussion of this page is held in the project's GitHub Discussions. Showing it here loads giscus from giscus.app and GitHub, and writing in it needs a GitHub account - what that means.

Show the discussion