The MTU, from 576 to 9000, on the three-appliance bench

Measured 2026-10-17 on the bench: one TCP stream for ten seconds in each direction, in clear text and through the IPsec tunnels (AES-GCM-256, in the engines at both ends), at eight MTUs from the smallest to the largest. Every run's window was photographed when it finished; the 52 pictures are in docs/images/measurements/mtu-2026-10-17/ and below each table.

How

• The MTU was set on both ends of each far link through OPNsense - the <mtu> of the interface's entry, applied with OPNsense's own reconfigure, the way the GUI's interface page does it - and read back from the interface: the 10G fibre between the two XGS 3300s (module port A1 and F1), and the gigabit link from the second XGS 3300's Port8 to the XGS 136's Port8.
• The far link is the narrowest hop. Above 1500 the PC's own 10G link runs at 9000 at both ends (jumbo packets 9014 on the PC's NIC), so the far link's MTU decides the segment size; the TCP maximum segment size each connection actually got is printed in every window.
• The gigabit link stops at 2000 was opened to 9000. Port8 is behind the switch chip on both appliances, which carried frames of at most 2048 bytes (the MTU is the LIF's), so the XGS 136 was measured up to 2000 and the fibre up to 9000 - so the XGS 136 was first measured up to 2000. Then Port Control 2 of port 8 was written from 0x1480 (2,048 bytes) to 0x2480 (10,240) on both switch chips, with mvsw on each coprocessor, and the 136 was measured at 4000, 6000 and 9000 as well - after the fix for what that found, issue 422. Written by hand for these measurements; since the same night a start hook writes all ten ports of both chips at every boot (scripts/xgs/switch-jumbo.sh).
• What each far host had learned was forgotten at every step (net.inet.tcp.hostcache.purgenow), and the PC's own path-MTU cache before every run.
• Two clients on the PC. The direction in which the PC sends was measured with iperf3 3.20 under WSL; the direction in which it receives with iperf3 3.22 for Windows - see what went wrong first.

The first XGS 3300, over the 10G fibre

MTU	segment	PC to it, clear	PC to it, IPsec	it to the PC, clear	it to the PC, IPsec
576	524 / 536	757 Mbit/s	531 Mbit/s	1.28 Gbit/s	1.05 Gbit/s
1000	948 / 960	1.12 Gbit/s	1.23 Gbit/s	2.10 Gbit/s	2.03 Gbit/s
1280	1228 / 1240	1.48 Gbit/s	1.49 Gbit/s	2.64 Gbit/s	2.71 Gbit/s
1500	1448 / 1460	1.73 Gbit/s	1.66 Gbit/s	3.40 Gbit/s	3.16 Gbit/s
2000	1948 / 1960	2.14 Gbit/s	1.96 Gbit/s	3.16 Gbit/s	3.85 Gbit/s
4000	3948 / 3960	3.79 Gbit/s	3.71 Gbit/s	4.83 Gbit/s	5.56 Gbit/s
6000	5948 / 5960	5.29 Gbit/s	4.97 Gbit/s	4.77 Gbit/s	5.41 Gbit/s
9000	8948 / 8960	7.11 Gbit/s	7.41 Gbit/s	3.55 Gbit/s	4.46 Gbit/s

The segment is the maximum segment size the connection got, first as the PC sending (Linux, with TCP timestamps, 12 bytes less) and then as the PC receiving (Windows).

The XGS 136, over the gigabit link

MTU	PC to it, clear	PC to it, IPsec	it to the PC, clear	it to the PC, IPsec
576	726 Mbit/s	411 Mbit/s	867 Mbit/s	755 Mbit/s
1000	908 Mbit/s	859 Mbit/s	924 Mbit/s	847 Mbit/s
1280	907 Mbit/s	885 Mbit/s	926 Mbit/s	892 Mbit/s
1500	936 Mbit/s	901 Mbit/s	949 Mbit/s	908 Mbit/s
2000	950 Mbit/s	926 Mbit/s	961 Mbit/s	931 Mbit/s
4000	976 Mbit/s	962 Mbit/s	948 Mbit/s	966 Mbit/s
6000	966 Mbit/s	975 Mbit/s	987 Mbit/s	977 Mbit/s
9000	989 Mbit/s	982 Mbit/s	991 Mbit/s	968 Mbit/s

From 4000 on, with jumbo frames through both switch chips and the driver of issue 422's fix (module 31def415); rx_nobuf and the tunnel's tx_nobuf stayed at 0.

What the tables say

• Into a 3300's host the rate follows the frame: from 757 Mbit/s at 576 to 7.11 Gbit/s at 9000 in clear text - 9.4 times the rate for 17 times the segment - and the tunnel keeps up all the way: 7.41 Gbit/s at 9000, with the engines encrypting on the second 3300 and decrypting on the first.
• The tunnel costs most at the smallest frame: 531 against 757 Mbit/s at 576, where ESP's envelope is the largest share of every packet. From 1000 upwards, towards the 3300, IPsec is within ten per cent of clear text and at 1000, 1280 and 9000 above it. In the other direction it is faster than clear text from 2000 on - 3.85 against 3.16 Gbit/s at 2000, 4.46 against 3.55 at 9000 - not yet explained.
• Out of the first 3300's host the curve turns over above 4000: 4.83 and 5.56 Gbit/s at 4000, then 3.55 and 4.46 at 9000. Not yet taken apart: the sender here is that appliance's host, and the receiver is a different client from the other direction. Taken apart: the PC drops them. Every segment the sender retransmitted is one the PC's NIC counted as discarded on arrival, to the segment - 864 and 864 at 1500, 1717 and 1717 at 4000, 1322 and 1328 at 9000 - and with four streams the same path carries 9.91 Gbit/s of 9000-byte frames with 79 discards. One stream lands on one receive queue of the PC's X722, which has 512 receive buffers; the appliances drop nothing. The same holds through the tunnel, so its apparent advantage in this direction is the noise of those drops. The window it was read in:

- The gigabit link is full from 1000 on: 908 to 961 Mbit/s in clear text, the line rate less the headers; IPsec is 2 to 8 per cent below it . up to 2000, and within 1 to 3 per cent from 4000, where 989 and 991 Mbit/s at 9000 are the line rate. At 576 the 136 sends 867 Mbit/s and takes 726, and the tunnel at 576 costs it the most of any row - 411 Mbit/s in.

What went wrong first

Two things in the measuring rather than in the appliances, and both are why the tables above were measured the way the How section says - and a third in the XGS 136's driver.

1. The PC kept a path MTU it had learned. The first pass from 1000 to 1500 ran with a 500-byte segment at every step: Linux holds a learned path MTU for ten minutes, the one learned through the tunnel at 576 was still held, and the flush in the script ran as a user that cannot flush it. Those runs were thrown away and measured again with the cache flushed as root before every run.
2. WSL drops a received frame above 1500 bytes. Every run in which the PC received, above 1500, measured 0.00 bits/sec, through the tunnel and without it. A 9000-byte ping from Windows to the second 3300 was answered; the same ping from WSL was not, from 1600 bytes up, with WSL's MTU at 9000 - its mirrored networking passes large frames out and not in. Between the two XGS 3300s over the fibre, 9000-byte frames ran at 8.87 Gbit/s one way and 8.19 the other. So every run in which the PC receives, at every MTU, was measured again with iperf3 for Windows.
3. The XGS 136 stopped receiving at the first frame above 4096 bytes - issue 422. With both switch chips carrying jumbo frames, DF pings of 1472, 2000 and 4000 bytes answered; one of 8000 did not, and from then on no TCP connection to the 136 worked, on any port. Its console showed why: receive queue 0 stopped at one descriptor (rx 200/56) and rx_nobuf climbing a thousand a second. The driver took the frame's mbuf with m_get2, which serves at most a page and answers NULL for anything larger, every time - and a failure there leaves the frame for the next tick. The same call built the tunnel's ESP envelopes, so above 4096 bytes not one segment left the 136 encrypted (6000 and 9000: 0.00 bits/sec, tx_nobuf counting). Both now take a cluster of the frame's size.

The windows

MTU 576

MTU 1000

MTU 1280

MTU 1500

MTU 2000

MTU 4000

MTU 6000

MTU 9000

Discussion

The discussion of this page is held in the project's GitHub Discussions. Showing it here loads giscus from giscus.app and GitHub, and writing in it needs a GitHub account - what that means.

Show the discussion