Design

The shape of the machine

A Sophos XGS appliance of this family - Marvell ARMADA CN913x, PCI 11ab:7080, fourteen front ports - looks like one computer and is two. Not all XGS models are: two of the six families carry no coprocessor at all.

  +-----------------------------+            +--------------------------------+
  |  x86 host                   |            |  Marvell CN913x "NPU"          |
  |  AMD Ryzen Embedded R1606G  |            |  its own Linux, its own eMMC   |
  |  runs OPNsense              |            |  owns ALL 14 front ports       |
  |                             |            |                                |
  |            PCIe root -------+---- x4 ----+--- PCIe endpoint 11ab:7080     |
  |                             |   Gen3     |                                |
  |            USB -------------+------------+--- MCP2210 bridge -> reset     |
  |            uart2 0x3E8 -----+------------+--- the NPU's console           |
  |            SMBus -----------+------------+--- board VPD EEPROM            |
  +-----------------------------+            +--------------------------------+

The host has no network hardware of its own. Every port on the front panel belongs to the coprocessor, and the only way to reach them is to get the coprocessor running and then speak to it over PCIe.

The stages, and where the line is now

 1   release the NPU from reset           DONE     src/etc/rc.syshook.d/early/06-npuctl
 2   complete the facility handshake      DONE     contrib/npuep/npuep.c
 3   the management interface, mvmgmt0    WORKING  contrib/npuep/npumgmt.c
 4a  the AGNIC command channel            WORKING  contrib/npuep/npugiu.c
 4b  traffic classes, queues, buffers     WORKING  contrib/npuep/npugiu.c
 4c  fourteen netdevs on the trunk        WORKING  contrib/npuep/npugiu.c
 4d  the 66-byte header, both directions  WORKING  contrib/npuep/npugiu.c
 4e  per-port control, the nwa mailbox    WORKING  contrib/npuep/npunwa.c
 4f  the rpc channel and the tables       WORKING  contrib/npuep/npurpc.c
 5   loading at boot                      WORKING  src/etc/rc.syshook.d/early/07-npuep
 6   assignment in OPNsense               not started

The front ports carry traffic in both directions. Measured with three cables in, frames arriving on three ports at once and each landing on its own interface, and a full ARP exchange completing over a switch port and over a SoC port. That is the line that moved.

Stage 4e's open question - how a request is signalled to a far side with no doorbell - is answered: a turn register the host writes rather than a doorbell, NWA_TURN at offset 0x18.

Stage 4f did not exist when this was first written. Receive needs the coprocessor's own forwarding tables filled in, and that is a fifth facility with its own protocol - see docs/npuep/rpc.md. It is where most of the difficulty turned out to be.

What is left is stage 6: the interfaces exist and carry traffic, but until they are assigned in OPNsense they are outside the firewall's own configuration and pf has no rules for them. That work is parked rather than under way - the appliance is powered off.

Five limits that are properties of the hardware

The datapath attaches once per coprocessor boot. The device waits for HOST_MGMT_READY once, answers once, and then spends the rest of its life in its command loop. A module reload on its own cannot be answered. Three remedies were tried and measured not to help: PF_DISABLE/PF_CLOSE, which the device accepts and which changes nothing; retracting the stale handshake; and waiting thirty seconds instead of four.

What restores it is a coprocessor reboot, and the host owns the means: 06-npuctl pulses the reset line, and the pulse is a reset rather than a release. Verified: a pulse followed by a reload brings back all fourteen interfaces, the forwarding tables and the network agent, with no power cycle - nwa: 14 of 14 ports up, and a front port brought up with no address counted forty frames off the wire in twelve seconds. A cold power cycle is verified too. A warm reboot runs that same hook, so it very probably works as well - still untested as a whole, though the mechanism it would rely on no longer is.

This file used to say a power cycle was the only thing that worked. That was wrong, and it was wrong for a reason worth keeping, because both causes looked exactly like hardware:

• reload.sh parsed the PCI selector with awk -F'[@ ]', and pciconf -l separates the selector from the class with a tab. So the selector came out as pci0:1:0:0:<tab>class=0x020000, every probe failed to parse, the script waited its full ninety seconds and reported a dead endpoint that had been answering from the first second.
• The barmap parser read an entry the coprocessor had not written yet as a real one. All four fields are zero in an unwritten entry, and zero is MV_FACILITY_CONTROL, so a half-published table came back as "control facility is not on BAR2" and failed attach outright.

The facility table is published about fourteen seconds after a reset, and not atomically. Measured here: the cookie reads zero for thirteen seconds, the entries then appear in order, and the table is complete at fourteen. The cookie is not a commit - it is in place before the entries are - so npuep_wait_barmap polls for the facilities it needs rather than for the cookie, up to two minutes, and goes on with whatever is published if it runs out.

That timing is also why none of this showed at boot: 06-npuctl pulses reset there too, but a minute of other boot work happens before the module loads, so the table is long finished. The defect was invisible on the only path that was ever run.

Loading the driver while the endpoint is in reset hangs the host. Measured: a reset pulse, a forty-five second wait and a kldload stopped the machine dead - no panic, no console output, nothing. A PCIe read to an endpoint in reset neither completes nor times out.

Guarded in two places now. Attach asks config space whether the device answers before it reads any memory, which is the safe question: a configuration read to a device that is not answering is completed by the root complex as all-ones rather than left outstanding, so it returns instead of stopping the machine. reload.sh asks the same question before it loads. Measured after the fix, the endpoint answers config space immediately after a pulse - so the guard costs nothing on the path that works, and only the facility table needs waiting for.

Programming the forwarding tables races the coprocessor's own startup. Its userspace fastpath starts in response to this host's handshake and zeroes the whole logical-interface table about thirteen seconds later. Commands sent before that are accepted, answered rc 0, and erased. So programming runs on a thread, reads back, and repeats until the read agrees.

The device's own packet counters are unavailable. GET_STATISTICS is answered at full length with zeros - it is the physical packet processor's block, and a function with no physical port has none. GET_GP_STATS is not implemented by this firmware at all. The driver reports its own counts and says so, because a zero that reads like a measurement is worse than an admission.

Stage 1 - reset

The NPU's reset line is not on a GPIO controller or a CPLD. It is on an MCP2210 USB-to-SPI bridge that enumerates as an ordinary USB HID device, which FreeBSD attaches without any help. kldload hidraw gives a device node and the rest is 64-byte reports.

The bridge powers up loading pin states from its own NVRAM, and on this board that state holds the NPU in reset. Nothing in OPNsense changes it, which is the entire reason a stock install finds no ports.

Two details matter and both were measured rather than assumed:

• It has to be a pulse, not a level. Writing the release values alone does nothing - the pin states read back correct and the NPU stays dead. Driving the pins to the hold state and then to the release state starts it every time. The part wants an edge.
• The polarity is per board. AMDA0200 is the exact inverse of AMDA0201. The module reads the assembly number from the bridge's EEPROM rather than assuming.

Stage 2 - the handshake

Once running, the NPU publishes a small map into BAR2 and then blocks. Its own startup script polls one 32-bit word and does nothing until it reads 0x0b:

bit 0  TRGT_INIT        the NPU sets this
bit 1  HOST_INIT        the host sets this
bit 2  TRGT_H2T_DBELL   the NPU sets this
bit 3  HOST_ALIVE       the host sets this, repeatedly - the NPU clears it on every scan

So the whole gate is two bits. But they cannot simply be written, because setting them tells the NPU that a host driver is present and ready - and it immediately starts raising doorbells, which are MSI-X interrupts, into vectors that only a kernel can allocate. It also begins using the host memory it has mapped.

That ordering is the single most important thing in this repository:

  allocate the five MSI-X vectors
  install their handlers
  THEN set HOST_INIT
  THEN set HOST_ALIVE, and keep setting it

Doing it the other way round - completing the handshake with nothing behind it - is a coprocessor with bus mastering enabled, told a driver is ready, writing into host memory that nobody vetted. It produces a general protection fault in an unrelated kernel subsystem some minutes later, with nothing in any log to connect the two.

Stage 3 - mvmgmt0, the management interface

A virtual Ethernet link between host and coprocessor. No wire: two rings in host memory, which the NPU reaches through its inbound window using physical addresses the host publishes into a shared structure at BAR2 + 0x1000.

This is the first point at which the host hands the coprocessor addresses in its own RAM. Up to stage 2 everything was the host reading and writing the endpoint's BARs, where the worst case is a confused endpoint.

The specification it was written against is docs/npuep/mvmgmt.md, which also records three places where a straight transcription of the vendor driver would be wrong.

It was the right step to take before the datapath, for three reasons: it is one facility rather than the whole GIU machinery, it is the smallest thing that can carry a packet and therefore prove the model, and it is the link the vendor's own diagnostic tools use - on the appliance, xgs-cpld, xgs-ports, xgs-sff and the rest are thin wrappers that ssh across it.

Stage 4 - the datapath

Not a ring pair like stage 3. GIU is a whole NIC: thirty management commands over a 64-byte descriptor channel, up to eight traffic classes each with its own queues, a buffer pool per receive queue, checksum offload in both directions, VLAN filtering, and a per-queue MSI-X vector.

The split is also inverted. In stage 3 both rings and both index pairs live in host memory. Here the configuration structure and every ring's producer and consumer index live in the device's BAR0, while the descriptors and buffers live in host memory - so the indices are MMIO accesses, not loads and stores.

contrib/npuep/npugiu.c implements it. docs/npuep/giu.md is the specification it was written from, read out of the vendor's GPL giu_nic source, including three defects the vendor fixed after publishing it.

How the fourteen ports are told apart is settled: a 66-byte private header at the head of every frame - a two-byte port identifier in network order, then sixty-four bytes of metadata - in front of the Ethernet header. Not the descriptor's port_num field, which the vendor's host driver never reads, and not a VLAN tag - an earlier reading of the harvested port map blamed the VLAN 4095 subinterface, and that was wrong, because 4095 is the same on all fourteen and so cannot distinguish them.

That split stage 4 into two pieces, and they were done in that order. Fourteen interfaces that carry traffic need the GIU trunk and the two-byte tag, both fully specified. Fourteen interfaces whose link state, speed and MTU can be read and set need Sophos's NetAgent message set, which rides the AGNIC custom channel and is not published - it was recovered from mv_nwa_host the way the MCP2210 command map was recovered from xgs-usb-spi-flash, and it lives in contrib/npuep/npunwa.c.

Proving the tag table, one port at a time

An ARP exchange with a device on the far end proves one path. It proves nothing about the other eleven, and it proves nothing either way when the far end declines to answer - which cost half an hour here, pinging a gateway that had been unplugged from that wire hours earlier.

tcpdump on the sending interface does not help. That is a BPF tap taken before the frame is handed to the coprocessor, so it says the driver transmitted. It cannot say anything came out of the connector.

contrib/npuep/portmap.sh settles it without a far end at all: transmit out of every port in turn, with a loopback cable between pairs, and record which port hears it. Ten of the fourteen, in one sweep:

  npup3 ↔ npup4     0x8300 ↔ 0x8400   switch
  npup5 ↔ npup6     0x8500 ↔ 0x8600   switch
  npup7 ↔ npup8     0x8700 ↔ 0x8800   switch
  npup9 ↔ npup10    0x0001 ↔ 0x0003   SoC
  npup11 ↔ npup12   0x0004 ↔ 0x0002   SoC

with npup1 ↔ npup2 measured separately first. Three results come out of it.

Egress reaches the connector. Nothing before this had shown that; it was inferred from the frame format and from one ARP exchange.

The SoC tag order is right. Those four are tagged 0x0001, 0x0003, 0x0004, 0x0002 in connector order rather than sequentially, and that ordering was read out of a disassembly, never documented. If 0x0002 and 0x0003 were the wrong way round, a frame leaving npup10 would have come out at the connector next to npup11 and been heard there. It was heard on npup9.

The internal switch does not forward between front ports. The sending port's own counter never moved, in any of the twelve. So every frame crossing between two front ports goes up to the host and back down, and pf sees all of it. Had the switch forwarded on its own, rules would be bypassed by traffic the firewall never saw - which is the kind of thing that is discovered after it matters rather than before.

PortF1 and PortF2 are the SFP cages, and the sweep could not cover them: a copper patch lead cannot loop a fibre cage, and no module was to hand at the time. Modules and a fibre have since been used on the XGS 3300, so this is an open test rather than a missing part of the design.

Link state, and the one line that hid it

The network agent reported carrier for the four SoC ports and never once for any of the ten switch ports - not while a switch port was linked and passing frames, and not when a cable was plugged into one. nwa: 14 of 14 ports up at attach showed the agent was addressing all fourteen for the bring-up command, so it was specific to link state.

It was arithmetic, in npunwa_command. The reply length counts bytes and includes an eight-byte header, so a one-byte answer is nine. Dividing the payload by four and truncating gave zero words: nothing was read and the caller was handed a zero, which is indistinguishable from a definite answer of "no carrier".

The two families are answered by different code on the far side, and that is why only one of them worked. The ten switch ports go through UMSD, whose npu_port_state_get sets ret_data.size = sizeof(param.state) on a u8 - nine bytes, zero words, always down. The four SoC ports are answered by NetAgent itself with a four-byte state - twelve bytes, one word, fine. Port9 reporting carrier while Port1 never did was not a property of the hardware.

The vendor's own host rounds up: NWA_NUM_DATA_CHUNCK(len) is NWA_PCI_ALIGN(len) / 4.

Rounding up then requires masking. The last word read may contain bytes the far side never wrote, the window belongs to another processor, and the link poll tests the whole word as (v != 0) - so a port would have read as up on the strength of somebody else's leftovers. The mask is applied in npunwa_command, which fixes every caller at once and needs no per-attribute knowledge of field widths.

Knowing is not telling

Reading the carrier correctly only put it in the log. Nothing told the network stack.

npugiu_init_locked had been declaring every port LINK_STATE_UP the moment the datapath came up, which is how all fourteen showed a green plug in OPNsense's interface list from the moment the driver loaded, including the ten with nothing plugged into them. That facility moves frames; it has no idea whether a cable is in the socket. The claim is gone, the state now starts LINK_STATE_DOWN - it started at LINK_STATE_UNKNOWN until the path-cost measurement below showed why that was worse - and npugiu_link_change is the seam the agent calls when it learns something. Verified with a temporary printf: indices 0, 2, 3 and 8 - exactly the four cabled ports

• reached if_link_state 2, and the ten empty ones did not.

This is not cosmetic. OPNsense drives gateway monitoring, failover and its rc.linkup hooks off link state, and a port that is always up is a port those mechanisms are blind to.

The media layer, and what it finally made visible

ifconfig prints both its status: line and its media: line from SIOCGIFMEDIA, and this driver answered no media ioctl at all - so for most of the project's life none of the fourteen showed a status or a speed, and OPNsense's interface list had nothing to colour a plug icon from.

With ifmedia(9) in place:

npup4   0x1008843  UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP    status: active
npup2   0x8802     BROADCAST,SIMPLEX,MULTICAST                        status: no carrier

Only IFM_AUTO is offered and the change callback does nothing but succeed: the coprocessor negotiates and the host is not in that conversation, so a menu of forced speeds it would ignore would be a lie with a menu. The speed is read only while carrier is up, because the ten switch ports answer with their capability rather than with nothing when they are dark.

A correction. An earlier version of this section said that FreeBSD has no IFF_LOWER_UP, that it is a Linux flag, and that link state is not visible in the flags word at all. The first part is half true and the rest is wrong. 0x1000000 is indeed IFF_NETLINK_1 in this kernel's if.h - but it is set when the interface has carrier, and ifconfig prints it as LOWER_UP, as the two lines above show. Two hours went into chasing that indicator and concluding it meant nothing; it meant something, and the reason it was absent was that nothing in the driver was reporting carrier yet.

A speed the stack can use, not just one it can print

Knowing the speed and putting it in SIOCGIFMEDIA tells a human. It does not tell the kernel.

ether_ifattach stamps IF_Mbps(10) on any Ethernet interface whose driver has not set its own figure - "just a default", says the comment beside it in if_ethersubr.c - and nothing here ever had. So all fourteen front ports told the kernel they were 10 Mbit/s links.

That is not cosmetic once the ports are bridged, because if_bridge costs its RSTP paths from if_baudrate using the formula in 802.1D-2004 section 17.14:

link	if_baudrate	RSTP path cost
what a 1G port should report	1000000000	20000
what one of the 2.5G ports should report	2500000000	8000
what every port actually reported	10000000	2000000
unknown, or link down	0	55 (BSTP_DEFAULT_PATH_COST)

A factor of a hundred, in the direction that makes the fastest link look worst. On this bridge it was visible immediately: the dark ports showed 55 and the live ones 2000000, which reads like the opposite of sense until you know that 55 is the default and 2000000 is a computed cost for 10 Mbit/s. Put a 1G port and one of the two 2.5G ports in one bridge with a loop behind them and RSTP elects the wrong root port.

The call itself is one line. It uses ifmedia_baudrate() so the figure and the media word cannot drift apart, and it runs before if_link_state_change(), because the recalculation happens from inside that call. ifmedia_baudrate() answers 0 for IFM_AUTO, which is what this driver reports when no speed is known, and 0 is the right answer: RSTP reads it as unknown and uses its own default, where 10 Mbit/s was a confident lie.

And it changed nothing, which is the part worth writing down.

The cost is latched, and UNKNOWN is what let a wrong one stand

After the fix the ports still read 2000000. The driver was reporting 1000baseT through the media layer at the same moment, so the speed was known and the call was running. Deleting a member and re-adding it by hand produced the right 20000 immediately - and that is the whole answer:

if_bridge computes a member's path cost once, in bstp_create, when the member is added. OPNsense adds all eight of ours seconds after the driver loads, and the network agent does not publish its window for about nine - the sampling further down is that measurement. So the cost was latched from whatever the driver claimed during those first seconds, and nothing ever revisited it.

bstp_calc_path_cost does arrange to try again - it sets BSTP_PORT_PNDCOST, which bstp_ifupdstatus acts on - but only when it is asked about a link that is LINK_STATE_DOWN. LINK_STATE_UNKNOWN is not down. It falls straight through to the baudrate, computes a cost, and sets no flag.

This driver had been starting every port at LINK_STATE_UNKNOWN deliberately, and an earlier version of this file called that "the honest answer" for a carrier nobody had reported yet. It bought a number that was wrong by a factor of a hundred and shut the door on ever correcting it. DOWN is the better claim on both counts: it is the conservative one - we have no evidence of a link - and it is the one that leaves the door open. So attach now clears the 10 Mbit/s guess and says DOWN.

That fixed the seven bridged ports with nothing plugged in. They latch 55, the neutral default, where they used to latch 2000000.

It did not fix the port with a cable in it, and the reason is worth the measurement it took. Sampling what the agent reports, once a second, from the moment the module loads:

t+9    10baseT/UTP        <- first carrier report, and the only moment RSTP will look
t+10   10baseT/UTP
t+11   10baseT/UTP
t+12   none               <- the PHY renegotiates
t+15   1000baseT          <- settled, and what it has been ever since

The switch reports 10baseT for three seconds while autonegotiating a gigabit link. RSTP spent its one recalculation on that reading: cost 2000000, PNDCOST cleared, never revisited. The carrier drop at t+12 does not help - bstp_ifupdstatus does not call bstp_calc_path_cost on the way down, so the flag is not re-armed.

An earlier version of this section said the DOWN-at-attach change would correct the cost once carrier arrived. It does not, and the table it came with was wrong. What is actually true:

when the member is added	latched cost	later corrected?
UNKNOWN, baudrate 10 Mbit/s (before any of this)	2000000	no
DOWN, baudrate 0, port dark (now)	55	only if carrier ever arrives
DOWN, baudrate 0, then carrier at a transient 10 Mbit/s (now)	2000000	no
member deleted and re-added by hand, once settled	20000	n/a

So the driver cannot make this number right. FreeBSD fixes a member's cost before the hardware is capable of knowing what it negotiated, and offers no way to ask for another look.

What the fix does buy is a correct if_baudrate - confirmed as 1000000000 on both ports that had carrier during that run, read straight out of the kernel with getifaddrs - which is what routing metrics and anything else asking about link speed will read.

Setting it from outside, once the link means something

ifconfig <bridge> ifpathcost <member> <cost> is the one lever that works, so src/opnsense/scripts/npuctl/bridge-pathcost.sh pulls it. It computes the same 802.1D figure the kernel would have - 20000000 / megabits - reads what is there, and writes only what differs.

Three decisions in it are worth stating, because each is a thing deliberately not done:

• Dark ports are left alone. Setting a cost sets BSTP_PORT_ADMCOST, which tells RSTP never to compute one again, so pinning a guess for a port with no cable would outlast the guess. A port that has never had carrier therefore sits at 55; one that had carrier and lost it keeps whatever was latched then, which on this bench was a port briefly cabled and left at 2000000. Neither matters: RSTP gives a member with no carrier role disabled and does not consult its cost, and a cable returning brings the right figure within a minute.
• Only this driver's ports. Every other NIC on this appliance reports its speed before OPNsense gets round to bridging it, so the figure FreeBSD worked out is already right and meddling would be worse.
• One ifconfig -a and one awk for the whole decision, and no further process at all unless something needs changing. That is what makes it cheap enough to run on a timer: 0.01 s measured.

Measured: npup1 went 2000000 -> 20000 (1000 Mb/s), its member flags gained ADMCOST, the seven dark ports stayed at 55, and a second run printed nothing and changed nothing.

Why a timer and not a link-up hook

The obvious trigger is IFNET / LINK_UP through devd, and it cannot be used. devd.conf(5) is explicit: "If two statements match the same event, only the action of the statement with highest priority will be executed." OPNsense already claims that event at priority 101, to run configctl interface linkup start.

So a rule below that never fires - which is exactly what a rule at 50 did here. It was installed, devd restarted, the module reloaded, and the cost stayed at 2000000 with nothing in the log. And a rule above 101 would fire and silence OPNsense's own linkup handling for these ports, which is not a trade worth making on a firewall to correct a number that only matters when a bridge has a redundant path.

/usr/local/etc/rc.linkup has no extension point of its own, and no periodic syshook stage runs on this appliance. So it is cron, in /usr/local/etc/cron.d/npuctl - which this cron reads - rather than /etc/crontab, which OPNsense regenerates from its own configuration and would drop.

Two things about devd cost a live service to learn, and are recorded in install.sh so they are not learned twice. service devd reload does not exist - rc.d/devd answers unknown directive. And devd installs no SIGHUP handler, so sending one applies the default action and kills it; doing that left this appliance with no devd at all, which means no DHCP renewal on a link event and none of OPNsense's own linkup handling. service devd restart is the only supported way.

One more, smaller, and it is a correction to something written two paragraphs earlier in this same session. A first version of the script logged at daemon.info and appeared not to run at all. The conclusion drawn was that this box does not collect daemon.*; that was wrong. The collector filters by severity, not by facility - one message was sent at each of info, notice and warn and /var/log grepped for all three: notice and warn both arrived, info went nowhere. So the script logs at daemon.notice, which is also what 06-npuctl and 07-npuep already use.

Promiscuous mode belongs to the switch

Arming each port's "this is my address" entry is what an ordinary interface wants: the switch delivers unicast addressed to that port and drops the rest.

A bridge wants the opposite. Its function is to receive frames addressed to other machines and forward them, so a bridged front port that only accepts its own address forwards nothing but broadcast - and looks like it is half working, which is the worst way for this to fail.

UMSD's TCAM entry 1 is the catch-all and is written dead, with a source-port vector mask of 0x7FF that can never match; umsd_port_promisc_set arms it by clearing that port's bit. NWA_SUB_PROMISC (0x45) is what reaches that code, and if_bridge setting IFF_PROMISC on a member is what reaches this driver. Measured on the 3-4 loopback, with frames addressed to a MAC the receiving port does not own:

before promiscuous   0 of 20 arrived
after  promiscuous  20 of 20 arrived

It is sent from the ioctl path with no driver lock held, because the agent's mailbox sleeps and this driver has already panicked once on sleeping under that mutex.

Asking once is not enough

The first version of that sent the request when SIOCSIFFLAGS arrived and took the answer as final. On the next reload the log said this, eight times:

nwa: Port1 would not open its catch-all (22)
...
nwa: mailbox ready after 10000 ms

if_bridge arms promiscuous mode on a member the instant it is added, and OPNsense adds its members seconds after the driver loads - which was eight seconds before the agent's mailbox became usable. Every request was refused with EINVAL, nothing retried, and the bridge forwarded broadcast and nothing else while all eight ports reported themselves healthy, at the right speed, with carrier. A fault that looks like working hardware is the worst shape one can take here, and it was introduced by the same change that made bridging possible.

So the wanted state is no longer pushed and forgotten; it is reconciled. The agent's link poll already walks one port per tick, and it now also reads back what that port's ifnet is asking for and compares it with what the switch was last successfully told. They agree on almost every sweep, so the cost is a comparison; when they do not, one mailbox round trip fixes it and says so:

nwa: Port1 catch-all opened on retry

The record of what the switch was told lives in the agent's own per-port state, not in the datapath layer, because the agent is the only thing that can set it and the only thing that will notice a failure. The ioctl is a nudge that makes the common case immediate; its error is deliberately ignored. p->promisc starts at 0 rather than unknown, which is not laziness - it is what UMSD leaves TCAM entry 1 in after a reset, so a fresh sweep has nothing to reconcile unless something really is asking.

This is the pattern the rest of the per-port state wants too. The address a port is told - the thing that makes unicast work at all - is sent once from the same bring-up path and is not reconciled, and it would fail the same way for the same reason.

Why the module is loaded from a boot hook and never preloaded

It is loaded at boot by 07-npuep and never preloaded from loader.conf, and both halves of that are deliberate.

A module that panics at boot gives you a machine that panics at boot. During development that costs a power cycle each time, and on a firewall it costs the firewall.

loader.conf preload would not work anyway. Resetting the NPU clears the endpoint's BAR registers, and FreeBSD keeps serving its boot-time cached values, so every read comes back as 0xFFFFFFFF. What repairs it is pci_driver_added() calling pci_cfg_restore(), which happens on the kldload path and not on preload. This was measured, not reasoned: BARs read as all zeros immediately before the load and correct immediately after.

The reset hook and the module have different risk profiles. The hook touches a USB bridge, does nothing on unrecognised hardware, and cannot hurt the host. The module allocates interrupt vectors and invites a coprocessor to use host memory. Both run unattended, so the module's hook is written to match that risk: it checks kldstat first, treats an absent module as a normal state, and exits 0 on every failure, where the reset hook needs none of that care.

Where the hooks sit in the boot sequence

The hooks are 06-npuctl and 07-npuep, and both numbers are chosen rather than inherited. They were 01 and 02 until an audit of the update path asked what runs between them and OPNsense configuring its interfaces. The answer is OPNsense's own 05-upgrade, which is this, in full:

for STAGE in K B P; do
    if opnsense-update -${STAGE}; then echo "Rebooting now."; reboot; fi
done

It finalises a pending firmware set and reboots from inside the early sequence. Numbered ahead of it, this project brought the coprocessor up, loaded the driver and let the endpoint start bus mastering - and then that hook rebooted the machine underneath it, unattended, with nothing given the chance to unload. Numbered after it the interaction does not exist, and it costs nothing on an ordinary boot, because opnsense-update finds nothing pending and returns.

Shutdown is not detach

device_shutdown was missing for the whole life of this driver, and nothing revealed it.

FreeBSD calls device_detach on kldunload. It does not call it on reboot - it calls device_shutdown, and a driver that declares none is simply skipped. So every reboot left the endpoint bus mastering, with its MSI-X vectors armed and the ring addresses this kernel had published still live, writing received frames and doorbell messages into physical memory the next kernel was about to hand to something else.

docs/npuep/porting-notes.md already described that symptom class under a different trigger: a fault in an unrelated subsystem, some minutes later, with no device errors logged in between. The teardown that prevents it had been written, carefully, and left reachable only from a path that a reboot never takes.

And the obvious fix hung the machine

The first attempt made the quiescing half of detach a function of its own and had both device_detach and device_shutdown call it: withdraw the facilities newest first, drain the heartbeat, clear HOST_INIT and HOST_ALIVE, disable bus mastering. It reads correctly. The first reboot after it went in never came back.

06:16:43  reboot: rebooted by root
06:16:43  syslog-ng: syslog-ng shutting down
          ... thirty minutes of nothing ...
06:47:00  kernel: ---<<BOOT>>---        <- and kern.boottime is the power cycle

No ---<<BOOT>>--- in between, and the boot time afterwards is when the power was pulled. The machine entered shutdown and never reached the reset.

Every one of those withdrawals waits. They wait for a coprocessor to acknowledge, and they drain taskqueue threads. That is correct in kldunload, where the system is running underneath them. Device shutdown methods run late in kern_reboot, after the filesystems have been flushed, and waiting on anything there is a request to be hung.

So device_shutdown now does only what actually stops the endpoint writing into host memory, and only in register writes that return:

• clear HOST_INIT and HOST_ALIVE, so the far side is told;
• clear the bus master bit, so it is stopped whether or not it was listening. Every DMA write and every MSI-X message is a memory write from the endpoint, so this covers the interrupts too - and it is enforced by the root complex rather than by the coprocessor's cooperation, which is why it is the one step that matters.

callout_stop rather than callout_drain, because stop does not wait for a callout already running and the heartbeat's whole job is one register write that is harmless at that point. The facility teardown stays in device_detach, where there is a system to wait on.

The general lesson is not about this driver. A teardown written for module unload is not a shutdown handler, however similar the two look, and the difference does not show up in review - it shows up as a machine that goes quiet and never comes back.

Which kernel sources the module is built against

OPNsense ships no kernel sources and no package provides them, so this appliance had a 333MB /usr/src/sys that somebody had copied there once. It built, so nobody asked what it was.

It was the wrong tree. It was stock FreeBSD 15.1-RELEASE, BRANCH="RELEASE", while the kernel is OPNsense's own build of 15.1-RELEASE-p1 from github.com/opnsense/src. The two differ in 156 files.

That the module worked anyway was luck, and the only way to know it was luck rather than correctness was to fetch the right tree and rebuild against it. __FreeBSD_version is 1501000 in both, none of the 156 differing files is in a path this driver includes, and the two builds came out byte-identical. A good outcome, and not a reason to go on guessing - the next kernel is under no obligation to be as kind.

The kernel names its own commit, so the sources can be pinned to exactly what is running with no version file to keep in step:

FreeBSD 15.1-RELEASE-p1 stable/26.7-n283674-12334a596709 SMP
                                             ^^^^^^^^^^^^ the commit in opnsense/src

contrib/npuep/fetch-sources.sh does that, and prints the SYSDIR to build with. Run it after the reboot that brings a new kernel up, not before: uname -v reports the running kernel, so beforehand it pins the old one perfectly and uselessly.

A mismatch does not announce itself

The assumption underneath compat.json, upstream.yml and verify.sh was that a module built for the wrong kernel would be refused at load time, loudly, with a message naming the cause.

It would not. A module's kernel dependency is a range - from the __FreeBSD_version it was compiled against to the end of that branch - so a module built for 15.1 loads cleanly into any later 15.x kernel. Nothing is printed, nothing fails, and any structure that moved is read at the wrong offset.

So the mismatch is caught out of band instead. contrib/npuep/build.sh writes the kernel it built against into npuep.ko.kernel, the installer carries that stamp to /boot/modules beside the module, and verify.sh compares strings. compat.json records kern_version for the same reason: OPNsense ships kernel sets within a series, so 26.7 -> 26.7.4 replaces /boot/kernel while leaving both "OPNsense 26.7" and "FreeBSD 15.1-RELEASE-p1" untouched. The labels do not move. The commit does.