Switch Port PRBS BER and Snake Test

Introduction

Switch Port Acceptance Overview

A 32-port 400G top-of-rack switch carries 256 SerDes lanes on its faceplate, eight per QSFP-DD cage, each running 53.125 Gb/s of PAM4 at 26.5625 GBd. The ASIC, a Tomahawk 3 class device, does not see clean bits on those lanes: it sees a pre-FEC bit error ratio that the RS(544,514) forward error correction of IEEE 802.3 Clause 119 turns into a frame loss ratio, provided the errors are few enough and random enough. A port that ships with one lane at the edge of that budget links up in the customer's rack, passes every ping, and drops one frame in a billion under load six months later when the optic ages. The production test that catches it is not a link-up test: it is a bit error ratio measured per lane, an eye margin read from the SerDes, and a snake that pushes line-rate frames through every port at once.

The parts this test exercises: the 256 SerDes transmitters and receivers, the PCB traces and QSFP-DD connectors between them and the cages, the FEC and MAC behind each port, the forwarding pipeline that moves frames port to port, the power stage feeding 32 modules, the fan trays that keep them cool, and the SONiC image and SAI package the whole thing boots on.

IEEE 802.3 gives the production test writer two numbers. The pre-FEC BER a 400GBASE-R link may run at is 2.4e-4, the point at which RS(544,514) still delivers a frame loss ratio below 1.7e-12 for 64-octet frames (Clause 119 and the PMD clauses that cite it). The electrical chip-to-module interface, 400GAUI-8, is budgeted at a symbol error ratio of 1e-5 (Annex 120E), two decades under the FEC limit, because the optical link is supposed to spend the rest. A 0 dB loopback plug in the cage adds nothing the host SerDes has to equalise, so a port tested through one should meet the AUI budget, not merely the FEC limit; Broadcom's own hardware design guideline for this ASIC family asks for a simulated pre-FEC BER better than 1e-6 as a guard band. Every limit in this template is one of those numbers, or says where it came from instead.

Test Purpose

The procedure records one acceptance fingerprint per switch:

• Serial, platform, ASIC, SONiC build and SAI package as one object against the release, and every one of the 32 cages populated with the station's loopback module
• PRBS31 on all 256 lanes for 60 s with linkscan off, each port judged on its worst lane and on its count of errored lanes, the raw per-lane shell dump attached
• The worst PAM4 eye per port, height and width, from the SerDes dump
• A line-rate snake through every port the PRBS phase cleared, the frames it lost, its receive rate, and the FEC corrected and uncorrected codewords per port afterwards
• Loopback module temperature and Vcc over CMIS at the end of the snake, with the modules dissipating what a 400G optic would
• PRBS off, linkscan on, the golden configuration reloaded and its checksum checked, every port back up, in teardown

The mock switch's worst lane on each port after 60 s of PRBS31: thirty-one ports between 2e-7 and 8e-7, and port 17 at 3.0e-6 on one lane, an order of magnitude above its neighbours and still three times under the production limit. That is the port the dashboard should show trending across the lot.

The framework mechanics on show are previous-results injection, with the snake phase receiving the PRBS phase's result object and leaving errored ports out of the chain; attach.file for the raw diag-shell dump next to the summarised measurement; then: {fail: retry} on the PRBS phase so a lane that lost lock once is re-run before the port is condemned; a plug at scope: station for the traffic generator next to a per-unit plug for the switch; run.metadata for what the station did and unit.metadata for what the unit is; and four multi-dimensional measurements indexed by port, each with two curves and their own aggregations.

Equipment & Setup

To run this acceptance on a production line, the following are required:

• 32 QSFP-DD loopback modules with programmable power dissipation and CMIS monitors, one per cage, left in place for the whole test
• A 400GE traffic generator with one QSFP-DD port on a direct-attach cable into port 1, sending at line rate and counting what returns
• A management connection to the switch: SSH into SONiC, and bcmcmd into the Broadcom diagnostic shell behind it
• The Device Under Test (DUT): a 32-port 400G QSFP-DD switch on a Tomahawk 3 class ASIC, booted on the release SONiC image, with a golden config_db.json on the station
• A TofuPilot Framework procedure to sequence the steps and validate the limits
• The TofuPilot Dashboard to keep the per-port evidence and watch the worst lane of each port across the lot

Hardware Components

Loopback Modules and Traffic Generator

The MultiLane ML4062-LB-112 is a QSFP-DD loopback that returns each lane's transmitter to its own receiver through a short trace, with five heaters that emulate any QSFP-DD power class up to 16 W, a temperature monitor and Vcc sense behind a CMIS memory map, and a cut-off that shuts the heaters at 85 °C. Thirty-two of them in the faceplate turn the PRBS test into a test of the host SerDes and the PCB alone, and the snake into a thermal test of the chassis under the load a fully populated switch sees. On port 1, a direct-attach cable to a Keysight AresONE 400GE port sends 1518 byte frames at 99.99 % of line rate and reports frames sent, received and the receive rate; the 0.01 % left on the table is the room a ±100 ppm clock offset between the generator and the switch needs so that no hop drops a frame on its own.

A single-fixture station: the switch with a loopback in every cage, one cable to the generator, and the production computer running the procedure over the management port.

The Switch and Its Diagnostic Shell

The Broadcom BCM56980 Tomahawk 3 carries 256 Blackhawk SerDes lanes at 50G PAM4 that form 32 ports of 400GbE, each port eight lanes at 53.125 Gb/s with RS(544,514) FEC. SONiC drives it through the SAI, and the vendor's diagnostic shell, reached with bcmcmd, exposes the PRBS generator and checker on every lane (phy diag cd0-cd31 prbs set p=3 for PRBS31, prbs get for the error counters), the SerDes dump with the eye margins (dsc), and linkscan, which must be turned off first: when the PRBS generator replaces data on the transmit lanes the PCS reports link-down, and a running linkscan re-initialises the SerDes under the checker. FEC counters come from show interfaces counters fec, the identity from show platform syseeprom and show version, and the CMIS monitors from the module EEPROM.

Why 60 s: a lane at 53.125 Gb/s accumulates 3.2e12 bits in a minute. With zero errors, that shows the lane is under 9.4e-13 with 95 % confidence, the grade of a post-FEC link, so the dwell measures every lane on the switch at its actual rate rather than merely checking it against the 1e-5 limit, which 6 µs would do. A port at 1e-6 collects three million errors in that time; the counters do not saturate.

Where the Limits Come From

Two rows in that table deserve a note. The BER is recorded as log10 rather than as the ratio, because a dashboard trending 3e-6 against 2e-7 across a lot needs a linear axis to show the drift; the aggregation limit is −5.0, which is 1e-5. And the FEC ceiling is a translation, not a new limit: at 425 Gb/s a port sees 2.3e10 codewords in five minutes, and a lane at the 1e-5 production BER would have the decoder correct about 1.3e9 of them. A port under the BER limit in PRBS and over this ceiling in the snake is a port whose errors are bursty, which PRBS averages away and the FEC does not.

Test Procedure

Overview

The procedure maps the acceptance onto the framework's three stages. Identity and presence live in setup: so no lane is exercised on a switch booted on the wrong image or with an empty cage. The four main phases share one management session and one generator, and chain with depends_on so the snake cannot start while PRBS still owns the lanes. The restore lives in teardown: so PRBS is switched off and linkscan back on even when a main phase fails.

1. Setup: serial against the label, identity object, 32 cages with loopback modules.
2. Main: linkscan off, PRBS31 on 256 lanes, 60 s, BER per lane, worst lane and errored lanes per port, raw dump attached.
3. Main: eye height and width of the worst lane per port from the SerDes dump.
4. Main: PRBS off, snake through the ports PRBS cleared at 99.99 % of line rate for 5 min, frames lost, throughput, FEC counters per port.
5. Main: loopback module temperature and Vcc over CMIS.
6. Teardown: PRBS off, linkscan on, golden configuration reloaded, checksum and ports up.

Every metric validates against limits declared in procedure.yaml, and results stream to TofuPilot as the per-switch evidence.

Why TofuPilot Framework?

TofuPilot Framework is a YAML + Python test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe what the test does in a procedure.yaml file, and how in small Python phase files. The framework handles:

• Automatic Python environment management (via uv)
• Operator UI (no frontend code needed)
• Measurement validation and live charts
• Process isolation between phases and equipment plugs

Project Structure

You can find the full source on GitHub. The SwitchUnderTest plug is a mock of the SONiC CLI and the diagnostic shell together, and TrafficGenerator a mock of the generator's API; between them they synthesize a healthy switch with 256 lanes between 1e-7 and 1e-6, one lane on port 17 at 3e-6, eyes that follow the BER, a FEC that corrects everything it sees and loopback modules warmest in the middle of the faceplate, so the procedure runs end-to-end without a switch or a generator connected. The 60 s dwell and the 5 min snake are time-scaled by 100 in the mock; the timeout on each phase is sized for the real durations.

For CI or bench automation, the same run executes headless:

The Procedure File

procedure.yaml declares the unit, the two plugs with the label serial and the management addresses as constructor arguments, and the three stages with every measurement and limit:

name: Switch Port PRBS BER and Snake Testversion: 0.1.0description: Port acceptance of a 32-port 400G QSFP-DD top-of-rack switch on a Tomahawk 3 class ASIC running SONiC. Identity and loopback presence, PRBS31 on all 256 lanes for 60 s against the 802.3 budget with the raw dump attached, the worst eye per port, a line-rate snake through every port and plug with the PRBS results deciding which ports enter it, FEC counters, loopback module temperature and Vcc over CMIS, and a teardown that puts the switch back in its shipping configuration.unit:  auto_identify: true  serial_number:    description: "Scan the serial label on the rear of the chassis"    placeholder: "SW32D-00000000"    pattern: "^SW32D-\\d{8}$"    default_value: "SW32D-26380417"  part_number:    default_value: "TOR32X400G-R0"  batch_number:    default_value: "LOT-2026-38"plugs:  - name: Switch    description: "The switch under test: SONiC CLI over SSH and the Broadcom diag shell for PRBS, dsc, FEC and CMIS (mock)"    python: plugs.switch_dut:SwitchUnderTest    key: switch    config:      serial_number: "SW32D-26380417"      host: "192.168.10.41"  - name: Traffic Generator    description: "400GE traffic generator on a DAC into port 1 (mock, station scope: the session is held across units)"    python: plugs.traffic_generator:TrafficGenerator    key: tgen    scope: station    config:      address: "192.168.10.9"      port: 1setup:  - name: Identify    key: identify    python: phases.identify    measurements:      - name: Reported Serial        key: reported_serial        validators:          - {operator: matches, expected_value: "^SW32D-\\d{8}$"}      - name: Serial Matches Label        key: serial_matches_label        description: The serial in the syseeprom TLVs is the one on the chassis label.        validators:          - {operator: "==", expected_value: true}      - name: Switch Identity        key: switch_identity        description: Platform, ASIC, SONiC build and SAI package as one object against the release.        validators:          - operator: "=="            expected_value:              platform: x86_64-tor32x400g-r0              asic: BCM56980              sonic: SONiC.202405.12              sai: 10.1.28.0      - name: Ports Present        key: ports_present        validators:          - {operator: "==", expected_value: 32}      - name: Loopback Plugs Detected        key: loopback_plugs_detected        description: Every cage reports a module, and every module's CMIS vendor PN is the station's loopback.        validators:          - {operator: "==", expected_value: true}main:  - name: PRBS BER    key: prbs_ber    python: phases.prbs_ber    timeout: 5m    retry:      limit: 1      delay: 2s    then:      fail: retry    measurements:      - name: Pre-FEC BER per Port        key: ber_per_port        title: Worst-lane pre-FEC BER and errored lanes per port, PRBS31 for 60 s        x_axis:          legend: Port        y_axis:          - legend: Worst lane BER (log10)            key: worst_lane_log10            unit: log10            aggregations:              - type: max_log10                unit: log10                validators:                  - {operator: "<=", expected_value: -5.0}          - legend: Errored lanes            key: errored_lanes            unit: lanes            aggregations:              - type: total                unit: lanes                validators:                  - {operator: "==", expected_value: 0}  - name: Eye Margin    key: eye_margin    python: phases.eye_margin    depends_on: [prbs_ber]    measurements:      - name: Worst Eye per Port        key: eye_per_port        title: Worst-lane eye height and width per port from the SerDes dump        x_axis:          legend: Port        y_axis:          - legend: Eye height            key: height_mv            unit: mV            aggregations:              - type: min_mv                unit: mV                validators:                  - {operator: ">=", expected_value: 40.0}          - legend: Eye width            key: width_ui            unit: UI            aggregations:              - type: min_ui                unit: UI                validators:                  - {operator: ">=", expected_value: 0.30}  - name: Snake Test    key: snake_test    python: phases.snake_test    depends_on: [eye_margin]    timeout: 10m    measurements:      - name: Ports in Snake        key: ports_in_snake        description: "Ports the PRBS phase cleared, chained port to port through their loopback plugs. All 32, or the unit is not shippable."        validators:          - {operator: "==", expected_value: 32}      - name: Frames Sent        key: frames_sent      - name: Frames Lost        key: frames_lost        description: Sent minus received at the generator after every frame crossed 32 ports and 32 plugs.        validators:          - {operator: "==", expected_value: 0}      - name: Throughput        key: throughput_pct        unit: "%"        description: "Receive rate at the generator as a fraction of 400G line rate. 99.9 % is this line's floor; the generator sends at 99.99 % to leave the +-100 ppm clock offset its room."        validators:          - {operator: ">=", expected_value: 99.9}      - name: FEC Codewords per Port        key: fec_per_port        title: RS(544,514) corrected and uncorrected codewords per port over the 5 min snake        x_axis:          legend: Port        y_axis:          - legend: Corrected codewords            key: corrected_cw            unit: codewords            aggregations:              - type: max_cw                unit: codewords                validators:                  - {operator: "<=", expected_value: 1300000000}          - legend: Uncorrected codewords            key: uncorrected_cw            unit: codewords            aggregations:              - type: total                unit: codewords                validators:                  - {operator: "==", expected_value: 0}  - name: Module Monitors    key: module_monitors    python: phases.module_monitors    depends_on: [snake_test]    measurements:      - name: Loopback Modules per Port        key: modules_per_port        title: Loopback module temperature and Vcc over CMIS at the end of the snake        x_axis:          legend: Port        y_axis:          - legend: Module temperature            key: temp_c            unit: °C            aggregations:              - type: max_c                unit: °C                validators:                  - {operator: "<=", expected_value: 70.0}          - legend: Module Vcc            key: vcc_v            unit: V            aggregations:              - type: min_v                unit: V                validators:                  - {operator: ">=", expected_value: 3.135}              - type: max_v                unit: V                validators:                  - {operator: "<=", expected_value: 3.465}teardown:  - name: Restore Config    key: restore_config    python: phases.restore_config    measurements:      - name: PRBS Disabled        key: prbs_disabled        validators:          - {operator: "==", expected_value: true}      - name: Linkscan Enabled        key: linkscan_enabled        validators:          - {operator: "==", expected_value: true}      - name: Config Checksum        key: config_checksum        description: SHA-256 of the running configuration after the golden config_db.json is reloaded.        validators:          - {operator: "==", expected_value: "sha256:9f3c1e7a4b2d0c8e5f6a7b8c9d0e1f2a3b4c5d6e7f8091a2b3c4d5e6f7a8b9c0"}      - name: Ports Up        key: ports_up        validators:          - {operator: "==", expected_value: 32}

Framework features to notice:

1. Previous-results injection. snake_test names prbs_ber as a parameter and receives the completed phase's measurements: it reads the errored_lanes curve, chains only the ports with none, and names the others in the log. depends_on guarantees the order; the snake reports on the ports that can carry traffic instead of failing on the one already known.
2. attach.file next to the measurement. The 256-line prbs get output is attached as prbs_per_lane.txt; the measurement keeps the worst lane and the errored count per port. The review gets both.
3. then: {fail: retry} with retry: {limit: 1, delay: 2s}. A PRBS checker that loses lock once at start reports a lane at 1e-3; the phase is re-run once before the port is condemned, and the second attempt is the one that counts.
4. Two plug scopes. The generator is scope: station, its session opened once and held across units; the switch plug is per unit, since every unit is a new SSH session.
5. run.metadata and unit.metadata. The PRBS polynomial, the dwell and the generator's identity describe what the station did on this run; the ASIC and the SONiC build describe what the unit is. Both stay queryable on the dashboard.
6. Four measurements indexed by port. ber_per_port, eye_per_port, fec_per_port and modules_per_port each carry two curves on one x-axis of 32 ports, with a different aggregation and limit on each curve.
7. teardown: always runs. PRBS off, linkscan on, golden configuration reloaded, whatever a main phase did.

Identify

The first SSH session. The serial in the syseeprom TLVs is compared with the scanned label; the platform, ASIC, SONiC build and SAI package are compared as one object against the release; and every cage must report a module whose CMIS vendor part number is the station's loopback. A switch booted on a lab SAI passes PRBS and forwards frames differently in the field.

from utils.recipe import LOOPBACK_MODULE, PORTSdef identify(measurements, switch, unit, log):    """Setup: the serial from the EEPROM against the label, platform, ASIC,    SONiC build and SAI as one object against the release, and every cage    populated with a loopback module before any lane is exercised. A switch    booted on the wrong SAI passes PRBS and drops frames in the field."""    eeprom = switch.syseeprom()    identity = switch.identity()    presence = switch.transceiver_presence()    plugs_ok = presence["present"] == PORTS and presence["loopback_modules"] == PORTS and presence["module_pn"] == LOOPBACK_MODULE    measurements.reported_serial = eeprom["serial"]    measurements.serial_matches_label = eeprom["serial"] == unit.serial_number    measurements.switch_identity = identity    measurements.ports_present = presence["present"]    measurements.loopback_plugs_detected = plugs_ok    unit.metadata["sonic"] = identity["sonic"]    unit.metadata["asic"] = identity["asic"]    log.info(f"{eeprom['part']} {eeprom['serial']} (label {unit.serial_number}): {identity['asic']} on {identity['sonic']}, {presence['present']} cages, {presence['loopback_modules']} x {presence['module_pn']}")

PRBS BER

Linkscan off, PRBS31 started on all 256 lanes at once, 60 s of accumulation, then the error counter and lock status of every lane. The phase computes each lane's BER as errors over bits, keeps the worst lane of each port as a log10 and counts the lanes that lost lock or crossed the budget, records both curves on one x-axis of 32 ports, stamps the polynomial and the dwell onto the run metadata, and attaches the raw shell dump.

import mathimport numpy as npfrom utils.recipe import BER_PRODUCTION_LIMIT, LANES_PER_PORT, PORT_NUMBERS, PRBS_DWELL_S, PRBS_POLYNOMIALdef prbs_ber(measurements, attach, run, switch, log):    """PRBS31 on all 256 lanes at once through the 0 dB loopback plugs,    linkscan off, 60 s of accumulation, then BER = errors / bits per lane.    Each port is judged on its worst lane against the 802.3 AUI budget of    1e-5, two decades under the 2.4e-4 the FEC is built for; a lane that    lost lock or crossed the budget counts as errored. The raw shell dump    is attached for the review."""    switch.set_linkscan(False)    switch.prbs_start(PRBS_POLYNOMIAL)    log.info(f"{PRBS_POLYNOMIAL} on {len(PORT_NUMBERS)} x {LANES_PER_PORT} lanes, {PRBS_DWELL_S:.0f} s dwell (mock: time-scaled)")    errors, locked, bits = switch.prbs_accumulate(PRBS_DWELL_S)    attach.file(switch.prbs_dump(), "prbs_per_lane.txt")    run.metadata["prbs_polynomial"] = PRBS_POLYNOMIAL    run.metadata["prbs_dwell_s"] = PRBS_DWELL_S    e = np.asarray(errors, dtype=float)    lock = np.asarray(locked, dtype=bool)    ber = np.maximum(e, 1.0) / bits  # a lane with zero errors reports the resolution floor, not zero    errored = (~lock) | (ber > BER_PRODUCTION_LIMIT)    worst_log10 = np.log10(ber.max(axis=1))    errored_lanes = errored.sum(axis=1)    worst_port = int(np.argmax(worst_log10)) + 1    for p, w, n in zip(PORT_NUMBERS, worst_log10, errored_lanes):        log.info(f"Port {p:2d}: worst lane BER 1e{w:.2f}, {n} errored lanes")    m = measurements.ber_per_port    m.x_axis = PORT_NUMBERS    m.y_axis.worst_lane_log10 = worst_log10.round(3).tolist()    m.y_axis.worst_lane_log10.aggregations.max_log10 = round(float(worst_log10.max()), 3)    m.y_axis.errored_lanes = errored_lanes.tolist()    m.y_axis.errored_lanes.aggregations.total = int(errored_lanes.sum())    log.info(f"Worst lane on the unit: port {worst_port} at BER {10 ** worst_log10.max():.1e} (limit {BER_PRODUCTION_LIMIT:.0e}), {int(errored_lanes.sum())} errored lanes, {int(e.sum())} errors over {math.log10(bits):.2f} decades of bits per lane")

A lane with zero errors is recorded at the resolution floor of the dwell, 3.1e-13, rather than at zero, so the log10 stays finite and the dashboard's trend shows what the measurement could resolve.

Eye Margin

With PRBS still running, the SerDes dump on every lane: the smallest of the three PAM4 eyes in mV and its width in UI, as the SerDes firmware reports them. The worst lane of each port is kept. BER says which lane errs; the eye says how far the others are from erring, and a port whose eye closes over the lot while its BER holds is the early warning the BER alone does not give.

import numpy as npfrom utils.recipe import PORT_NUMBERSdef eye_margin(measurements, switch, log):    """The SerDes dump with PRBS still running: per lane, the smallest of the    three PAM4 eyes and its width, the worst lane kept per port. BER says a    lane errs; the eye says how close the others are to erring."""    height, width = switch.eye_scan()    h = np.asarray(height).min(axis=1)    w = np.asarray(width).min(axis=1)    tight = int(np.argmin(h)) + 1    m = measurements.eye_per_port    m.x_axis = PORT_NUMBERS    m.y_axis.height_mv = h.round(1).tolist()    m.y_axis.height_mv.aggregations.min_mv = round(float(h.min()), 1)    m.y_axis.width_ui = w.round(3).tolist()    m.y_axis.width_ui.aggregations.min_ui = round(float(w.min()), 3)    log.info(f"Smallest eye on port {tight}: {h.min():.1f} mV; narrowest {w.min():.3f} UI on port {int(np.argmin(w)) + 1}; median {np.median(h):.1f} mV x {np.median(w):.3f} UI")

The mock switch's worst eyes: a cluster around 65 mV by 0.40 UI, and port 17 at 58.1 mV by 0.357 UI, the lane at 3e-6 showing where the BER comes from. The floors are this line's; a switch whose cluster migrates toward them across a lot has a PCB lamination or a connector problem before any port fails.

Snake Test

The PRBS phase's result object is injected. Ports with an errored lane are left out of the chain and named; the rest are configured as a snake, each port's ingress forwarded to the next port's egress, the last back to the first, so a frame from the generator crosses 32 ports and 32 loopback plugs before it returns. Five minutes at 99.99 % of line rate, then the generator's counters and the FEC counters of every port.

import numpy as npfrom utils.recipe import PORT_NUMBERS, SNAKE_DURATION_S, SNAKE_FRAME_BYTES, SNAKE_RATE_PCTdef snake_test(measurements, run, switch, tgen, prbs_ber, log):    """Line-rate traffic from one generator port, forwarded port to port    through every loopback plug and back. The PRBS phase's results are    injected: a port with an errored lane is left out of the chain and    named, so the snake reports on the ports that can carry traffic rather    than failing on the one already known. Frame loss must be zero and the    FEC must have corrected everything it saw."""    errored = prbs_ber.measurements.ber_per_port.y_axis.errored_lanes    ports = [p for p, n in zip(PORT_NUMBERS, errored) if n == 0]    skipped = [p for p, n in zip(PORT_NUMBERS, errored) if n]    if skipped:        log.warning(f"Ports left out of the snake after PRBS: {skipped}")    switch.prbs_stop()    switch.set_linkscan(True)    hops = switch.snake_configure(ports)    switch.clear_fec_counters()    tgen.configure(SNAKE_FRAME_BYTES, SNAKE_RATE_PCT)    run.metadata["traffic_generator"] = tgen.identity()    log.info(f"{hops} hops, {SNAKE_FRAME_BYTES} B frames at {SNAKE_RATE_PCT} % of 400G for {SNAKE_DURATION_S / 60:.0f} min (mock: time-scaled)")    stats = tgen.run(SNAKE_DURATION_S)    corrected, uncorrected = switch.fec_counters(SNAKE_DURATION_S)    c = np.asarray(corrected)    u = np.asarray(uncorrected)    measurements.ports_in_snake = hops    measurements.frames_sent = stats["sent"]    measurements.frames_lost = stats["lost"]    measurements.throughput_pct = stats["rx_rate_pct"]    m = measurements.fec_per_port    m.x_axis = PORT_NUMBERS    m.y_axis.corrected_cw = c.tolist()    m.y_axis.corrected_cw.aggregations.max_cw = int(c.max())    m.y_axis.uncorrected_cw = u.tolist()    m.y_axis.uncorrected_cw.aggregations.total = int(u.sum())    log.info(f"{stats['sent']} frames sent, {stats['lost']} lost, {stats['rx_rate_pct']:.3f} % of line rate; FEC corrected up to {c.max():.2e} codewords on port {int(np.argmax(c)) + 1}, {int(u.sum())} uncorrected")

The FEC counters after the mock snake: 9.75 billion frames sent and none lost, every port between 1e7 and 4e7 corrected codewords, port 17 at 1.55e8, the same port and the same order of magnitude as its PRBS result, and zero uncorrected everywhere. The snake is where a forwarding, buffer or power-stage problem shows that PRBS cannot see, because PRBS never moves a frame.

Module Monitors

At the end of the snake, with the 32 loopbacks having dissipated their emulated 12 W for five minutes, the CMIS temperature and Vcc of every module. The x-axis is the port, so the curve is a picture of the faceplate: a fan tray that has lost a rotor, or a power stage that sags under 32 modules, appears as a shape before it appears as an alarm.

import numpy as npfrom utils.recipe import LOOPBACK_POWER_W, PORT_NUMBERSdef module_monitors(measurements, switch, log):    """CMIS temperature and Vcc from every loopback module at the end of the    snake, with the plugs dissipating what a 400G optic would. A cage that    runs hot here, or a Vcc rail that sags under 32 modules, is a fan tray    or a power stage problem the optics will report as an alarm in the rack."""    temp, vcc = switch.module_monitors()    t = np.asarray(temp)    v = np.asarray(vcc)    m = measurements.modules_per_port    m.x_axis = PORT_NUMBERS    m.y_axis.temp_c = t.tolist()    m.y_axis.temp_c.aggregations.max_c = round(float(t.max()), 1)    m.y_axis.vcc_v = v.tolist()    m.y_axis.vcc_v.aggregations.min_v = round(float(v.min()), 3)    m.y_axis.vcc_v.aggregations.max_v = round(float(v.max()), 3)    log.info(f"Modules at {LOOPBACK_POWER_W:.0f} W: warmest {t.max():.1f} C on port {int(np.argmax(t)) + 1}, coolest {t.min():.1f} C; Vcc {v.min():.3f} to {v.max():.3f} V")

Restore Config

The teardown phase switches PRBS off on every lane, turns linkscan back on, reloads the golden config_db.json and records the checksum of the running configuration and the number of ports that came back up through their plugs. No switch leaves the station with a SerDes in a diagnostic mode.

def restore_config(measurements, switch, log):    """Teardown: PRBS off, linkscan back on, the golden configuration    reloaded and its checksum compared, every port back up through its    plug. Runs whatever a main phase did, so no unit leaves the station    with its SerDes in a diagnostic mode."""    prbs_off = switch.prbs_stop()    linkscan = switch.set_linkscan(True)    reloaded = switch.config_reload()    measurements.prbs_disabled = prbs_off    measurements.linkscan_enabled = linkscan    measurements.config_checksum = reloaded["checksum"]    measurements.ports_up = reloaded["ports_up"]    log.info(f"PRBS off, linkscan on, config {reloaded['checksum'][:19]}..., {reloaded['ports_up']} ports up")

Mock Plugs

SwitchUnderTest stands in for the SONiC CLI and the diagnostic shell because PRBS, the eye dump, the FEC counters and the CMIS reads all describe the same 256 lanes. The unit's physical property is one pre-FEC BER per lane, drawn once at construction: a log-normal around 1.6e-7 clipped at 8e-7 on thirty-one ports, and on port 17 a spread around 1e-6 with lane 3 fixed at 3e-6. The PRBS errors are a Poisson draw on that BER over the 3.2e12 bits of the dwell, the eye height and width follow the BER with a little noise, the corrected codewords in the snake follow the port's mean BER over the 2.3e10 codewords of five minutes, and the module temperatures follow a bump centred on the middle of the faceplate. TrafficGenerator computes frames sent from the rate and the frame size and returns them all received. Both sleep the real dwell divided by 100. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from measurements.<key> returns a proxy, so the phases keep locals for their log lines.

On a real bench, keep one SSH session per unit in SwitchUnderTest and run the SONiC commands and bcmcmd through it, parsing prbs get and dsc into the same lists; read the CMIS bytes through the platform's sfputil or the EEPROM page; and drive the generator through its REST API with one stream, frames sent and received per port. Keep the 60 s dwell and the 5 min snake at their real values and drop the time scale. The phases, measurements and limits stay the same.