Does PCIe 3.0 x8 provide enough bandwidth for a dual QSFP 40Gbit NIC?

I'm researching dual QSFP 40GBit network cards for a work project and have a few questions regarding PCIe 3.0's theoretical max bandwidth.

I'm currently looking at a dual QSFP PCIe 3.0 x8 card on CDW (Mellanox MCX314A-BCBT), but I don't think PCIe x8 would provide enough bandwidth for both 40Gbit links at 100% utilization.

Wikipedia states that PCIe 3.0 has a theoretical max bandwidth of 985MB/s per lane. Thus, by my calculations, PCIe 3.0 x8 would yield a max bandwidth of 7880MB/s. If this is true, the card would saturate the PCIe connection if both links are at 100% utilization.

Here are my specific questions:

• What is the max bandwidth that a dual QSFP 40Gbit network card can output?

• What is the max bandwidth that a PCIe 8x can handle before saturation?

• Is there an easy way to calculate this?

2 Answers

Doing a little dimensional analysis

Converting 7880 Mbytes per second to Gbits per second we get 63.04 Gbits/sec

(63.04 gigabits per second)/(40 (gigabits per second)) = 1.576

If you have a layout like this:

Unit (1) PCIe 3.0 slot, x8 or larger -> one QSFP card providing 1 x 40 Gbps connected to 8 lanes

Unit (2) PCIe 3.0 slot, x8 or larger -> one QSFP card providing 1 x 40 Gbps connected to 8 lanes

... then it will work fine, even accounting for the fact that there will be some overhead making it difficult to achieve theoretical throughput, because each PCIe slot will get its own lanes.

If however you have a layout like this:

Unit (1) PCIe 3.0 slot, x8 or larger -> one QSFP card providing (2 x 40 Gbps) connected to 8 lanes

... then it won't work fine, because now instead of having 1.576 times more bandwidth you need, you actually only have 0.788x as much bandwidth as you need.

I guess the manufacturer figures that protocol overhead at the ethernet layer will slow it down enough that this limitation won't matter, or otherwise some bottleneck further down in the I/O subsystem, or app turns (round trips). It seems odd that they would design the card so that the maximum theoretical throughput of the two ports on the board exceeds the theoretical throughput of the 8 lanes, but if you are really expecting to utilize >78% of 80 Gbps of throughput, you might want to just buy two cards (ideally, with one port apiece, if you can find a different model with that), and put them in separate slots, with each slot being at least 8 lanes wide.

What is the max bandwidth that a PCIe 8x can handle before saturation?

The max bandwidth of a single PCIe v3 lane is 985 MB/sec. (8.0 Gbit/second).

x8 means that up to 8 PCIe lanes can be used, which give a theoretical max of 64 Gbit/sec.
This is less then two 40Gbit links.

So you can not run both links at full speed. It might be enough in practise though. Especially if the traffic is bursty. Just as long as both channels do not burst at the same time.