NCP-AIN Hub

Lossless Congestion Pipeline

Step through PFC, ECN, and DCQCN as a coordinated chain on a leaf-spine RoCEv2 fabric. Each stage lights up with the exact mechanism, real default values, and Wireshark-style frame capture. Pillar 11 (obj 2.3, 30% domain).

Ready -- press Step or Play
Speed 1.0x

Stage 0 of 5 -- Idle

Press Step to walk one stage at a time, or Play to auto-advance. The canvas shows a 2-tier leaf-spine topology with sender (GPU server) and receiver.

RoCEv2 frame capture -- leaf01 swp3 (uplink to Sender NIC)
No.TimeSourceDestinationProtoLenInfo
Packets: 0 Display filter: roce || pfc || cnp

PFC and ECN/DCQCN address the same problem (congestion) at different layers and with different trade-offs. The system works best when ECN/DCQCN absorbs most congestion so PFC rarely fires.

PFC -- Priority Flow Control (802.1Qbb)

  • Acts at the switch: sends a PAUSE frame per priority class upstream
  • Blunt: freezes the entire priority queue, not just the congested flow
  • Immediate effect -- no NIC involvement required for the stop
  • Fires when buffer crosses the PFC threshold (separate from ECN thresholds)
  • Default priority: SP3 (RoCE traffic class)
  • Risk: head-of-line (HOL) blocking -- unrelated flows on the same priority also pause
  • Risk: PFC storm / deadlock if pause frames loop back (watchdog: 100ms polling, robustness 3)
  • Propagates pause hop-by-hop upstream until sender NIC stops transmitting
Blunt last resort -- tune ECN/DCQCN so PFC rarely fires

ECN + DCQCN -- Graceful Rate Control

  • ECN acts at the switch ASIC: marks CE bit on packets (no drop, no pause)
  • Marks start at Kmin (146.48 KB) and reach 100% probability at Kmax (1.43 MB) [know]
  • Receiver NIC sees CE-marked packet and generates a CNP back to sender (TC6, strict priority)
  • DCQCN runs at the sender NIC: multiplicative decrease, then additive increase recovery
  • Fine-grained: per-flow rate control, other flows on the same priority are unaffected
  • DCQCN rate curve shape: illustrative [inferred] -- exact algorithm is vendor implementation-specific
  • ECN enabled on TC0 and TC3 (RoCE data) by default
  • CNP scheduling on TC6 ensures feedback reaches sender without delay
Preferred mechanism -- proactive, per-flow, lossless, low latency impact

Operational order of events

  1. Traffic builds. Switch buffer fills toward Kmin (146.48 KB on TC0/3).
  2. Buffer crosses Kmin: switch marks ECN CE bit. Probability scales linearly to 100% at Kmax.
  3. Receiver NIC generates CNP (UDP/4791, TC6, strict priority) to sender.
  4. Sender NIC runs DCQCN: multiplicative rate decrease (alpha-based), then additive recovery.
  5. If ECN/DCQCN is insufficient and buffer keeps growing past the PFC threshold: switch sends 802.1Qbb PAUSE frame on SP3. Upstream sender halts that priority.
  6. Buffer drains. PFC XOFF resolves, XON frame restores flow. DCQCN continues additive increase.
Exam shortcut One command configures all three layers: nv set qos roce auto-configures PFC on SP3, ECN on TC0+TC3 with the exact thresholds, CNP on TC6 strict priority, DSCP trust, and buffer pools. No per-parameter tuning needed for baseline lossless RoCEv2.

These are the values shown in nv show qos roce on a Spectrum switch with the default lossless config. Numbers tagged [know] are from the pillars data source (pillar n10/n11); numbers tagged [inferred] are illustrative.

Parameter Value Mechanism Source
ECN min-threshold (Kmin) 146.48 KB Buffer depth at which ECN marking begins (prob = 0 below this) [know] pillars n10/n11, nv show qos roce
ECN max-threshold (Kmax) 1.43 MB Buffer depth at which ECN marking probability reaches 100% [know] pillars n10/n11, nv show qos roce
ECN probability 100% Marks every eligible packet at or above Kmax [know] nv show qos roce
ECN enabled-tc TC0, TC3 Traffic classes where ECN marking is active [know] pillars n11
PFC priority (switch-priority) SP3 The priority class that gets PAUSE frames (802.1Qbb) [know] pillars n11
RoCE data traffic class TC3 DSCP 26 maps to SP3 maps to TC3; DWRR 50% [know] pillar n10
CNP traffic class TC6 (strict priority) CNP always delivered before data; DSCP 48 maps to SP6 [know] pillars n11
RoCEv2 UDP port 4791 BTH carried in UDP/IP; L3-routable; LLDP App TLV protocol-id [know] pillar n10
DCQCN rate curve Illustrative only Multiplicative decrease / additive increase; exact curve is NIC firmware-specific [inferred] shape documented in RFC; exact params = vendor
PFC watchdog polling 100 ms, robustness 3 Detects DEADLOCK state per TC; auto-drops pause frames to break storm [know] pillar n19
Spectrum-4 SN5600 packet buffer 128 MB shared On-chip buffer; lossless mode carves a dedicated pool for SP3 [know] pillar n9
cumulus@switch:mgmt:~$ nv show qos roce operational applied ------------------ ----------- -------- mode lossless lossless congestion-control congestion-mode ECN enabled-tc 0,3 min-threshold 146.48 KB max-threshold 1.43 MB probability 100 pfc pfc-priority 3 rx-enabled enabled tx-enabled enabled trust trust-mode pcp,dscp lldp-app-tlv priority 3 protocol-id 4791 selector UDP

Recall drill on PFC / ECN / DCQCN. Click an answer, then read the explanation before moving on.