IPsec associations on SFOS 23's coprocessor software: refused, then taken, 2026-10-25
Found while starting on discovery 14 (IPsec state kept in step):
on the first XGS 3300, whose coprocessor runs SFOS 23 EAP 1's software, no association had been
offloaded since it came back to OPNsense. Its lab tunnel to the second XGS 3300 - the PC's network
to the first board's own address, IKEv2, AES-GCM-256 - ran in the kernel's cipher, and the driver
said why at every install: the coprocessor refused the association, rc 1, SP2FP_RC_INVALID_PARAMETER.
SFOS 23 lengthened SA_ADD's request. Its host's fw_fp_add_egress_sa zeroes and sends 0xc8 bytes
where SFOS 21's sends 0xc0, and its populate_add_req stores every field from the key on 8 bytes
further - the window at 0x78 instead of 0x70, the SPI at 0x7c instead of 0x74, the sequence at 0x80,
the revision at 0xc0 - with a word at 8, taken from the kernel's xfrm_state, and four bytes of
padding before them. The far side refuses a request shorter than its own, so every 192-byte request
was refused (rpc-numbering.md). The other IPsec
commands kept their sizes. Both drivers now lay SA_ADD out by the version - 192 bytes on SFOS 21
and 22, 200 on 23 with the word sent 0, as SFOS 23's host sends it for a tunnel that is not
route-based - and send no association while the version is not known (sfosnum_sa_add_shift).
octep.ko 4c844f00 on both XGS 3300s, npuep.ko 590205fb on the XGS 136
(images/deploy/lab-sa23.sh).
The line under the table in the first picture names ipsec.sa_refused the coprocessor's refusals.
It is not: it counts the offers the driver itself declined - mostly another interface's association,
22 at every boot of this board, before the fix and after it. The coprocessor's refusals are the
driver's lines below it. The second picture names it right.
| associations in the coprocessor | 40 pings through the tunnel | the engine decrypted / encrypted | the kernel's ESP in / out | |
|---|---|---|---|---|
before, fdc4e281 |
none - both refused, rc 1 | 40 answered | 0 / 0 | 40 / 40 |
after, 4c844f00 |
the inbound at index 1, the outbound at 2 | 40 answered | 40 / 40 | 0 / 0 |
The other two boards kept theirs with the same build: the second XGS 3300 (SFOS 22) its four, the XGS 136 (SFOS 21) its two, accepted by the engine.
A stream through it
TCP from the PC to the first board through the tunnel - the second board encrypting, the first decrypting and delivering to its own host - and back the other way, read by hand after the pictures, not photographed:
| runs | Mbit/s | |
|---|---|---|
| into the first board (its engine decrypts) | 5, of 6 to 10 s | 343, 1,770, 967, 196, 231 |
| out of it (its engine encrypts) | 3, of 6 to 10 s | 3,190, 3,580, 3,370 |
In the 1,770 Mbit/s run the first board's engine decrypted 152,032 frames and its fast path counted
no drop, nor did the driver's receive path (rx_bad, rx_nokey, rx_queuefail unmoved); the
counters that moved are the ones the SFOS 23 list names RX_IPSEC, FROM_WIRE_TO_IPSEC_DECR and
FROM_KN_TO_IPSEC_ENCR - 152,032, 152,032 and 152,041.
What it does not say
- Why the stream into the first board varied from 196 to 1,770 Mbit/s while the other direction held at 3.2 to 3.6 Gbit/s. Later the same night, after the tunnel had been rekeyed, the same stream ran at 7.65, 9.77 and 9.68 Gbit/s with the first board's engine decrypting all of it (ipsec-step-2026-10-25); what changed between the two was not found.
- The ARMADA driver's SFOS 23 layout is the same table's, built and run on SFOS 21 only: no ARMADA board here runs SFOS 23's software. The XGS 126 of #736 does.
- What the word at 8 does for a route-based tunnel.
The discussion of this page is held in the project's GitHub Discussions. Showing it here loads giscus from giscus.app and GitHub, and writing in it needs a GitHub account - what that means.