Ubuntu 20.04下如何为Mellanox Infiniband网卡设置IP地址

检查网卡的状态

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


knightli@epyc:~$ ibstat
CA 'mlx4_0'
        CA type: MT4103
        Number of ports: 2
        Firmware version: 2.42.5700
        Hardware version: 0
        Node GUID: 0x98f2b3ffffd5c3c0
        System image GUID: 0x98f2b3ffffd5c3c3
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x9af2b3fffed5c3c1
                Link layer: Ethernet
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x9af2b3fffed5c3c2
                Link layer: Ethernet

需要能看到网口的状态

查看网卡的设备名称(logical name)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


knightli@epyc:~$ sudo lshw -c network
[sudo] password for knightli: 
  *-network DISABLED
       description: Ethernet interface
       product: MT27520 Family [ConnectX-3 Pro]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:28:00.0
       logical name: enp40s0
       logical name: /dev/fb0
       version: 00
       serial: 98:f2:b3:d5:c3:c1
       capacity: 56Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd 40000bt-fd 56000bt-fd autonegotiation fb
       configuration: autonegotiation=on broadcast=yes depth=32 driver=mlx4_en driverversion=4.9-4.1.7 duplex=full firmware=2.42.5700 latency=0 link=no mode=1024x768 multicast=yes visual=truecolor xres=1024 yres=768
       resources: iomemory:1130-112f irq:133 memory:eb700000-eb7fffff memory:11390000000-11391ffffff memory:eb600000-eb6fffff memory:11392000000-113b1ffffff
  *-network DISABLED
       description: Ethernet interface
       physical id: 3
       bus info: pci@0000:28:00.0
       logical name: enp40s0d1
       serial: 98:f2:b3:d5:c3:c2
       capacity: 56Gbit/s
       capabilities: ethernet physical fibre 1000bt-fd 10000bt-fd 40000bt-fd 56000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx4_en driverversion=4.9-4.1.7 firmware=2.42.5700 link=no multicast=yes

多个端口的会有多个设备名称, 如上有2个端口，分别有2个logical name, enp40s0和enp40s0d1

修改netplan配置文件

为每个端口设置对应的IP

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


root@epyc:/home/knightli# vi /etc/netplan/00-installer-config.yaml 


# This is the network config written by 'subiquity'
network:
  ethernets:
    eno1:
      dhcp4: true
    enp40s0:
      addresses: [172.18.8.1/24]
    enp40s0d1:
      addresses: [172.18.9.1/24]
  version: 2

修改完成后执行 netplan apply

1

root@epyc:/home/knightli# netplan apply

检查结果

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


root@epyc:/home/knightli# ifconfig
enp40s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.8.1  netmask 255.255.255.0  broadcast 172.18.8.255
        inet6 fe80::9af2:b3ff:fed5:c3c1  prefixlen 64  scopeid 0x20<link>
        ether 98:f2:b3:d5:c3:c1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40  bytes 5671 (5.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp40s0d1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.9.1  netmask 255.255.255.0  broadcast 172.18.9.255
        inet6 fe80::9af2:b3ff:fed5:c3c2  prefixlen 64  scopeid 0x20<link>
        ether 98:f2:b3:d5:c3:c2  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40  bytes 5671 (5.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 130  bytes 11184 (11.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 130  bytes 11184 (11.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@epyc:/home/knightli# ibstat
CA 'mlx4_0'
        CA type: MT4103
        Number of ports: 2
        Firmware version: 2.42.5700
        Hardware version: 0
        Node GUID: 0x98f2b3ffffd5c3c0
        System image GUID: 0x98f2b3ffffd5c3c3
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x9af2b3fffed5c3c1
                Link layer: Ethernet
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x9af2b3fffed5c3c2
                Link layer: Ethernet