Mellanox Infiniband卡切換IB/Ethernet/VPI工作模式

Mellanox Infiniband 卡 如何在 IB/Ethernet/VPI 3種工作模式之間切換

查看網卡的當前工作狀態

執行ibstat, 返回的Link layer信息, 就是 網卡的工作模式

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
knightli@epyc:~$ sudo ibstat
CA 'mlx4_0'
        CA type: MT4103
        Number of ports: 2
        Firmware version: 2.42.5700
        Hardware version: 0
        Node GUID: 0x98f2b3ffffd5c3c0
        System image GUID: 0x98f2b3ffffd5c3c3
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x9af2b3fffed5c3c1
                Link layer: Ethernet
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x9af2b3fffed5c3c2
                Link layer: Ethernet

切換工作模式

安裝mst軟件

1
root@epyc:/home/knightli# systemctl start mst

查詢設備名稱

MST devices下就是返回的設備名,注意2個端口也對應一個設備名

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
root@epyc-7551p:/home/knightli# mst status
MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4103_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:28:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00
/dev/mst/mt4103_pci_cr0          - PCI direct access.
                                   domain:bus:dev.fn=0000:28:00.0 bar=0xeb700000 size=0x100000
                                   Chip revision is: 00

切換工作模式

mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P1=1

  • -d 設備名稱
  • LINK_TYPE
    LINK_TYPE_P1 : 端口1
    LINK_TYPE_P2 : 端口2
    1 : IB
    2 : Ethernet
    3 : VPI
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
root@epyc:/home/knightli# mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P1=1

Device #1:
----------

Device type:    ConnectX3Pro    
Device:         /dev/mst/mt4103_pciconf0

Configurations:                              Next Boot       New
         LINK_TYPE_P1                        IB(1)           IB(1)           

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

root@epyc:/home/knightli# mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P2=1

Device #1:
----------

Device type:    ConnectX3Pro    
Device:         /dev/mst/mt4103_pciconf0

Configurations:                              Next Boot       New
         LINK_TYPE_P2                        IB(1)           IB(1)           

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

重啟主機

1
root@epyc-7551p:/home/knightli# sudo reboot
记录并分享
Built with Hugo
Theme Stack designed by Jimmy