Though this may not be directly related to the increase of errors on mx_stream, we noticed irqbalance doesn't work well with Hyper Threading on Gentoo servers.
It occurs on k1dc0, k1nds0, k1nds1, and k1fw0 and it's better to fix on the next maintenance day.
----
On k1dc0 and other affected servers, all loads are concentrated to CPU0 as follows.
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11
0: 55234 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge timer
8: 140 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge rtc0
9: 3 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
18: 958961 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, ehci_hcd:usb2
72: 13196807 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge ahci
73: 44 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge ahci
74: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0
75: 2545764089 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-TxRx-0
76: 2749653483 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-TxRx-1
77: 2422420500 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-TxRx-2
78: 2478695662 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-TxRx-3
...
On the other hand, k1fw1 and other unaffected servers seem to be able to use all CPU cores as follows.
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
0: 168 5450 0 0 0 0 IO-APIC-edge timer
8: 0 154 0 0 0 0 IO-APIC-edge rtc0
9: 0 3 0 0 0 0 IO-APIC-fasteoi acpi
18: 0 0 183 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, ehci_hcd:usb2
72: 0 0 0 0 0 539268 PCI-MSI-edge ahci
73: 0 0 0 0 0 41 PCI-MSI-edge ahci
74: 0 0 0 0 0 0 PCI-MSI-edge eth0
75: 0 0 0 0 0 15925231 PCI-MSI-edge eth0-TxRx-0
76: 5239188 0 0 0 0 0 PCI-MSI-edge eth0-TxRx-1
77: 32705758 0 0 0 0 0 PCI-MSI-edge eth0-TxRx-2
78: 0 67699476 0 0 0 0 PCI-MSI-edge eth0-TxRx-3
79: 0 14661393 0 0 0 0 PCI-MSI-edge eth0-TxRx-4
80: 0 0 9937904 0 0 0 PCI-MSI-edge eth0-TxRx-5
88: 0 0 0 0 0 962942 PCI-MSI-edge eth3-TxRx-0
89: 0 0 0 0 0 952423 PCI-MSI-edge eth3-TxRx-1
90: 0 0 0 0 0 736032612 PCI-MSI-edge eth3-TxRx-2
91: 0 0 0 0 0 926493 PCI-MSI-edge eth3-TxRx-3
92: 1143161 0 0 0 0 0 PCI-MSI-edge eth3-TxRx-4
93: 924492 0 0 0 0 0 PCI-MSI-edge eth3-TxRx-5
...
This issue seem to occur on too old version of Linux. On k1nds2 which is the Debian system, loads are dispersed to all CPU cores even if HT is enabled.