How it was found

A map of the work, and it is deliberately not a list of achievements. The dead ends are the expensive part - they are where the months went - so they are drawn at the same size as the things that worked, and the page is more useful for them.

If you want what is true now, read families/octeon-tx-reference.md. If you want the numbers, measurements/xgs3300.md. This page is about the shape of getting there.

The size of it

	on 2026-10-01	on 2026-10-12
commits in this repository	166, across 2026-09-25 to 2026-10-01	272
changes merged by pull request	157	263
issues opened	32, of which 27 are closed	73, of which 62 are closed
lines of driver	7,903 for octep, 16,441 across all of contrib/	20,009 for octep, 28,556 across contrib/
lines of documentation	7,651	18,355
things tried against the last hop and recorded as not working	19	19
claims published and later withdrawn on this page's subject	10	not recounted

The second column was added on 2026-10-12, taken with the same commands and the limit set; the first stays as the account of the week this page is about.

One of those figures was wrong here until it was recounted. This page said 107 changes merged; it was 154 when that was caught and 157 by the end of the same evening, which is why the table above and this sentence do not match - the table is a count and this is an account. gh pr list returns thirty rows unless it is told otherwise, and the first count took the default and believed it. A page about not guessing had a guess in its own table.

The tenth withdrawal is this page's own neighbour. Hours after the front panel's micro-controller was photographed, this project published that the panel's protocol is the bridge's own and that the 0xFE escape had been the wrong thing to chase. The vendor's daemon sends exactly that escape to exactly that panel and it works, so the claim was wrong and it was withdrawn the same evening. It was caught because someone read the summary table further down this page, saw that it still said the old thing, and asked which of the two was wrong. A stale page is an instrument, and that is the second time in one day that the oldest text in the repository corrected the newest.

And the issue count was right by luck, which is worse. The same command caps at thirty, and there were thirty issues, so the broken method printed the true number. It would have printed thirty if there had been three hundred. A figure that cannot be wrong when the method is wrong is not evidence of anything, and the only reason this one is trusted now is that it was taken again with the limit set.

And the measurements themselves had a hole in them, which is the fifth rule below. Every datapath number on this page and in the measurement log was taken by sending: a generator on a dark front port, thousands of packets a second, hours of soak, 0% loss. All of it real, and not one of those runs could ever have filled a receive ring. The appliance had been unable to complete a single download the entire time - a 16.7 MB file returned 0 bytes in 25 s, three opnsense-update runs installed nothing - and the loss figures stayed at zero throughout, because the defect was in the half nobody was measuring. It was found by putting the coprocessor's count of what it handed to the host beside the host's count of what it took, two numbers that were both already being collected and had never been put side by side.

The repository is a week old. The problem is not: the blocker that nineteen of those negatives were aimed at had been open for eighteen months before the week that closed it.

The shape of it

flowchart TD
    A["A coprocessor with no documentation<br/>and a vendor driver for another OS"] --> B

    B["Read the vendor's own sources<br/>GPL drop, BSP, board file, disassembly"] --> C
    B --> D

    C["The control plane<br/>handshake, NetAgent, RPC"] --> C1["Works"]
    D["The datapath<br/>SDP rings in and out"] --> D1["Out: works"]
    D --> D2["Back in: nothing"]

    D2 --> E["Eighteen months on the last hop"]

    E --> X1["Is the ring misprogrammed?"]
    E --> X2["Is the host out of credit?"]
    E --> X3["Is an interrupt missing?"]
    E --> X4["Are the VFs missing?"]
    E --> X5["Is the metadata wrong?"]
    E --> X6["Is the LIF missing?"]
    E --> X7["...nineteen of these"]

    X1 --> N["No. Measured, written down,<br/>and never retried"]
    X2 --> N
    X3 --> N
    X4 --> N
    X5 --> N
    X6 --> N
    X7 --> N

    N --> F["Three of them WERE missing.<br/>Implemented. Changed nothing."]

    F --> G["The grant was off by a constant factor"]
    G --> H["The doorbell counts BYTES of the<br/>scatter list - sixteen per buffer<br/>(the grant was too small; the unit<br/>was not sixteen - a ring that fills)"]
    H --> I["The last hop opens"]

    I --> J["Front ports become interfaces"]
    J --> K["But only the two direct cages.<br/>Ten more are behind a switch<br/>nobody had programmed"]

    K --> L["Reach the 88E6193X over MDIO"]
    L --> M["Eight copper ports at a gigabit"]
    K --> O["Two SFP cages stay dark"]
    O --> P["One bit each, on the CPLD"]

    M --> Q["Frames arrive and die at the LIF"]
    Q --> R1["Try interface 14"]
    Q --> R2["Try interface 0"]
    R1 --> R3["No"]
    R2 --> R3
    R3 --> S["The index was never wrong.<br/>The TAG was."]
    S --> T["The switch uplink runs DSA.<br/>Every frame names its source port."]
    T --> U["Twelve front ports,<br/>twelve real addresses"]

    U --> V["It survives a reboot"]
    V --> W["And it is measured"]

    style E fill:#5a1a1a,color:#fff
    style N fill:#5a1a1a,color:#fff
    style F fill:#5a1a1a,color:#fff
    style R3 fill:#5a1a1a,color:#fff
    style H fill:#1a4a1a,color:#fff
    style S fill:#1a4a1a,color:#fff
    style I fill:#1a4a1a,color:#fff
    style W fill:#1a4a1a,color:#fff

What a frame crosses today

Eleven things have to be right at once for one packet, and each of them was a separate piece of work. That is the honest measure of the size of this, more than any commit count:

flowchart LR
    W(("wire")) --> P1["copper panel port<br/>PHY powered, page 0, 0x1140"]
    P1 --> SW["88E6193X<br/>forwarding, TCAM entry live<br/>because the host named the MAC"]
    SW --> DSA["uplink in DSA frame mode<br/>tags every frame with its source port"]
    DSA --> FP["the coprocessor's fast path<br/>tag to LIF index, MTU, forwarding mode"]
    FP --> RING["SDP output ring<br/>granted its size above<br/>the coprocessor's watermark"]
    RING --> PCI(("PCIe"))
    PCI --> DRV["octep<br/>82-byte prefix, tag to ifnet"]
    DRV --> MB["mbuf"]
    MB --> IF["oxpN<br/>its own MAC, its own link state"]
    IF --> PF["pf"]
    PF --> OS(("OPNsense"))

The last day, which was a quarter of the whole

2026-10-01 is in the figures above as one date among seven, and it carried eight merges. They are worth naming because three of them changed what the project is, rather than adding to it.

A frame posted for one front port could leave by another. Twelve interfaces shared one transmit buffer: a frame was copied in, an instruction pointing at it was posted, the lock was dropped, and the next frame overwrote it before the coprocessor had read it. At 1.9 microseconds a packet that is the normal case. Nothing was dropped - frames were replaced, and went out of a different port with another port's tag. Every counter agreed with every other one the whole time, because the count was always right and only the contents were wrong. What found it was loading one port and watching a different one, and one buffer per instruction slot took the loss from 29% to 0.0% at every rate.

IPv6 started working, because SIOCADDMULTI stopped being a no-op and the port is now asked to pass multicast. A ping to ff02::1 came back from more than twenty neighbours on a front port that could not have seen one of them the day before.

The crypto engine answered. A security association installs and reads back, and the two metadata bytes that ask for encryption were found by bisection. It stops at one named counter, and that counter is now an issue with the question written on it.

And the appliance was opened and photographed, which settled three things that reading could not - including what the front panel really is.

The dead ends, and what each one cost

Nineteen were recorded against the last hop alone. These are the ones whose lesson outlived them:

what was believed	what it cost	what was actually true
the ring is misprogrammed	weeks of register sweeps	it was programmed correctly the whole time
the host is out of credit	a reading of the doorbell that was worthless, because the grant it reported had been corrupt since the ring was first armed	the register counts bytes, not entries (it counts entries after all; the grant had been under the coprocessor's output watermark - a ring that fills)
the coprocessor is not deciding to forward	a search for a forwarding mechanism that does not exist	it was deciding correctly; the frames were asking to be encrypted
the cage LEDs show traffic	a published claim, later withdrawn	those cages have no LED key at all in the board file
the LIF index is wrong	two interface numbers tried and a third being searched for	the index was right; the tag was a value the far side never produces
the sensors are on the host's SMBus	three probes, two of them invalidated by a controller left stuck	the bus the controller serves has nothing on it, not even DIMM SPD
a PON stick is not an Ethernet module	a cage written off as unusable	an ONU stick presents 1000BASE-X like anything else; its laser was simply off

And one that is not a dead end but a method. 0xFE 0x58 in the panel's protocol is the escape followed by the letter X - and a row of X characters on the display was the clue that the escape was not being honoured. The byte that looked like garbage was the answer.

What this project does instead of guessing

Five rules, all of them learned by paying for the alternative:

1. Read the vendor's own sources before probing the hardware. The port tags, the DSA tag format, the LED scheme, the CPLD's pins, the panel's protocol and the sensor map were all in files already on disk. One of them - the port tag table - had been published in this repository's own reference page for months while the driver used values the far side never produces.
2. Write down what did not work, with the measurement. Nineteen negatives meant nineteen things nobody retried.
3. Withdraw a claim in the same place it was made. Eight on the OCTEON TX page alone, struck through rather than deleted, because a log that quietly edits itself is not a log.
4. A number that has not been taken is not a number nobody needs. Row five of the progress table read 0% for months. Measuring it took an afternoon and showed the driver already had six and a half times the headroom ten of the twelve ports need.
5. Ask what a measurement cannot see. Every datapath figure here was a transmit figure, taken against a port with no cable, and the receive path went unexercised for the whole week while it was broken badly enough that no download ever finished. A soak that loads one direction is a measurement of one direction, however long it runs.

The three parts this is all about

Photographed on the appliance, because every number in this repository was read from a file until somebody opened the box. The markings below are the parts' own, from the silicon.

	MARVELL OCTEON TX — CN8365-1500BG1676-SCP-G, at board position U11. This is the coprocessor that owns the twelve front ports, and everything in this repository is about reaching it. The platform database named it; this is the die cap saying the same thing.
	88X5113-BVM4 — 2244 A1F, at U42. The 10G PHY behind PortF1 and PortF2, which the board file reaches at mdio45:0:7. A claim about this part was withdrawn once in this project, over a register that was never the indicator it was taken for; the part itself was never in doubt and now it is on a photograph.
	AMD Ryzen YE1780C3T4MFB — the V1780B, the x86 host, and the other end of the PCIe link. Its unit codes and data matrix are blurred; the model number is the evidence and stays.

And a second set, taken later and with better light, is in octeon-tx-peripherals.md: the front panel's own micro-controller, the nine bypass relays, the service headers with what each one takes to use, the management NIC, and the carrier board carrying a name nobody expected. Three of those pictures settled questions that reading could not.

Inside, with the lid off: the two heatsinks, the SFP cages down the right edge, the relay bank beside them, and a single blue indicator that is the only light this board shows without software. Every printed label has been blurred - a photograph of somebody's hardware must not carry its serial numbers, and the repository's own private-data checker reads text rather than images, so that one is done by hand.

Two thin channels carried all of it

Neither of the paths that opened this appliance is fast, and the gap between what they carry and what they unlocked is the real measure of the work.

SPI reaches exactly one device: the CPLD. npu0.cpld.location=spi:0:1:3 puts it on the coprocessor's SPI rather than the host's - /dev/spidev0.1, mode 3, 3 MHz. A read is one byte of reg | 0x80 and then four big-endian bytes; a write is a single five-byte transfer; every register is 32 bits wide. What rides on that one bus is most of the board that is not a network port: the four SFP cages' pins in register 0x25, the fail-to-wire relay at 0x39, 0x3a and 0x3b, the SFF event register at 0x38, and the block id and version that let a read be checked before it is believed. Clearing one bit on that bus is what lit a cage that had been dark since the appliance was built.

And there are three UARTs, all on the host, doing three unrelated jobs:

port	address	what it is	speed
uart0	0x3f8	the host's own console	115200, and 38400 under the vendor's firmware
uart1	0x2f8	the front panel, through an EZIO bridge	0xFE then an HD44780 instruction, which is what the vendor's daemon sends; the rate is the open question
uart2	0x3e8	the coprocessor's console	115200 raw, and it needs a device hint to appear

Two of those three were not documentation but tools. uart2 is how mvsw got onto the coprocessor at all - sixty kilobytes base64'd through a serial line - and therefore how the switch behind the panel ports was first reached. uart1 is the panel whose protocol came out of the vendor's own daemon.

So: a 3 MHz bus and a 115200 line. That is what an undocumented coprocessor, a managed switch, twelve front ports and a CPLD were opened with.

The XGS 136: the same three parts, and the channels that opened it

The other appliance here is the one the ARMADA driver was written for, and it was opened on 2026-10-11. The markings below are the parts' own; every close-up, with what each was identified from, is on hardware/xgs136.md.

	Marvell CN9130-2200 — the coprocessor, under its lid, with four SK hynix DDR4 chips around it. The kernel on it reports a CN9131, which is a CN9130 with a second CP115, and the part marked 88F8215-A0 beside it is that second CP: by its marking, and by the port map, which puts labels 9 to 12 on COMPHY lanes a bare CN9130 does not have.
	Marvell 88E6193X-BXB2 — the LinkStreet switch behind labels 1 to 8 and the two SFP cages, ports 9 and 10 on the switch, with port 0 as the one uplink to the coprocessor. The same part as the XGS 3300's, reached there over the coprocessor's MDIO; here it is the part behind ten of the fourteen ports.
	AMD Ryzen Embedded — which FreeBSD reports as an R1606G, 4 CPUs and 8 GB: the x86 host, and the other end of the PCIe link, 11ab:7080. Its lot code and matrix are pixelated; the model is the evidence and stays.

Twelve of the fourteen were verified port by port with loopback cables, and the sending port's own counter never moved: the switch does not forward between front ports behind the host's back. F1 and F2 are untested, because a copper lead cannot loop a cage.

And the channels that opened it are different ones. This coprocessor is held in reset at power-up and has to be released, and what releases it is an MCP2210 USB-to-SPI bridge on the appliance's internal USB, three of whose GPIO pins reach the coprocessor; the release has to be a pulse - hold, a pause, release - because writing the release state alone leaves it dead. Its console is the host's third UART, and the whole of its boot is readable there once the echo is turned off. The control plane and the datapath are the PCIe endpoint's BARs and the five facilities the coprocessor publishes into them.

channel	where	what it carries
PCIe 11ab:7080	Gen3 x4; BAR0 1 MB, BAR2 16 MB, BAR4 16 MB; MSI-X 16	the five facilities - control, the management link, the network agent, RPC, the datapath - published at the top of BAR2 about fourteen seconds after a reset pulse
mvmgmt0	two rings in host memory	a virtual Ethernet to the coprocessor; both sides poll every 10 ms and nothing rings a doorbell
MCP2210, 04d8:00de	the internal USB, /dev/hidraw0	GP0, GP3 and GP5, mask 0x0029: the reset line; and the assembly number, in the bridge's own EEPROM
uart2	0x3e8, IRQ 3, 115200	the coprocessor's console; it needs a hint to appear, and the echo off before it is read
intsmb0	the AMD FCH SMBus	the I2C path to the SFP cages and the PoE controller; the vendor's platform tool reads the VPD EEPROM at 0x57 over it

So: a USB bridge with three pins, a 115200 line and one PCIe endpoint. The XGS 3300 was opened with a bus and a serial line that reach past the coprocessor; this one needed a reset pulse first, and after that nothing that is not on the PCIe link.

Discussion

The discussion of this page is held in the project's GitHub Discussions. Showing it here loads giscus from giscus.app and GitHub, and writing in it needs a GitHub account - what that means.

Show the discussion