author：headsen chen
date： 2019-01-18 10:22:20
notice：created by headsen chen himself and not allowed to copy, or you will count law question!
版本环境：centos6.8 ，64位，内核：2.6.32
1，配置网卡：
在新卡装上机器，接收光纤，两根线都有接，而且是反接的方式接，接通后，光纤灯会亮
2，安装软件RDMA的方式安装，编译内核和用户态，重启进入新内核4.7
3，安装驱动：
正常的kernel安装方法(2.6的内核)
# /mnt/mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64/install
这里必须采用这种，因为是新内核4.7
#tar fx mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64.tgz#cd mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64#./install --add-kernel-support --skip-repo
Logs dir:/tmp/mlnx-en.28728.logs
General log file:/tmp/mlnx-en.28728.logs/general.log
Verifying KMP rpms compatibility with target kernel...
This program will install the mlnx-en package on your machine.
Note that all other Mellanox, OEM, OFED, RDMAorDistribution IB packages will be removed.
Those packages are removed due to conflicts with mlnx-en, do notreinstall them.
Do you want tocontinue?[y/N]:y
rpm--nosignature -e --allmatches --nodeps rdma rdma
Starting mlnx-en-4.4-2.0.7.0installation ...
Installing mlnx-en-utils 4.4RPM
Preparing...##################################################
mlnx-en-utils ##################################################
Installing kmod-mlnx-en 4.4RPM
Preparing...##################################################
kmod-mlnx-en ##################################################
Installing mlnx-en-sources 4.4RPM
Preparing...##################################################
mlnx-en-sources ##################################################
Installing mlnx-en-doc 4.4RPM
Preparing...##################################################
mlnx-en-doc ##################################################
Installing user level RPMs:
Preparing...##################################################
ofed-scripts ##################################################
Preparing... ##################################################
mstflint ##################################################
Device (83:00.0):83:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4Lx]
Link Width: x8
PCI Link Speed: 8GT/s
Device (83:00.1):83:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4Lx]
Link Width: x8
PCI Link Speed: 8GT/s
Installation finished successfully.
Preparing...########################################### [100%]
1:mlnx-fw-updater ########################################### [100%]
Updated /usr/share/hwdata/pci.ids
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...
Device#1:
----------Device Type: ConnectX4LX
Part Number: MCX4121A-XCA_Ax
Description: ConnectX-4 Lx EN network interface card; 10GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
PSID: MT_2420110004
PCI Device Name:83:00.0Base MAC: ec0d9ad2fd68
Versions: Current Available
FW14.20.1010 14.23.1020PXE3.5.0210 3.5.0504UEFI N/A 14.16.0017Status: Update required---------Found1device(s) requiring firmware update...
Device#1: Updating FW ... Done
Restart neededforupdates to take effect.
Log File:/tmp/mlnx-en.28728.logs/fw_update.log
Configuring/etc/security/limits.conf.
To load the new driver, run:/etc/init.d/mlnx-en.d restart
4，重启服务：
/etc/init.d/mlnx-en.d restart
5，安装MLNX_OFED_LINUX-4.4
这里不用像软件RDMA 那样的启动rxe_cfg了。
yum -y install libmml tcl tk libmnl
tar fx MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64.tgz
cd MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64
./mlnxofedinstall --add-kernel-support --skip-repo/etc/init.d/openibd restart #这个命令最好在管理卡上执行，xshell上执行有可能导致网卡掉IP，
/etc/init.d/network restart
chkconfig openibd on
ibv_devices
出现一下结果代表成功：#ibv_devices
device node GUID------ ----------------mlx5_1 ec0d9a0300d2fc99
mlx5_0 ec0d9a0300d2fc98
如果这一步不成功(有时候rxe_cfg不启动也可以)：#rxe_cfg start (并绑定eth4网卡)
#ibv_devices
device node GUID------ ----------------rxe0 ee0d9afffed2fd68
#ibv_devinfo rxe0
hca_id: rxe0
transport: InfiniBand (0)
fw_ver:0.0.0
node_guid: ee0d:9aff:fed2:fd68
sys_image_guid:0000:0000:0000:0000vendor_id:0x0000vendor_part_id: 0
hw_ver:0x0phys_port_cnt:1port:1state: PORT_ACTIVE (4)
max_mtu:4096 (5)
active_mtu:1024 (3)
sm_lid: 0
port_lid: 0
port_lmc:0x00link_layer: Ethernet
6，利用rping 命令测试：
生成server端：
[[email protected]:/root]#rping -s -a 10.20.15.23 -v -C 10
生成client端：
client端的安装和服务端一样，生成命令是：
#rping -c -a 10.20.15.23 -v -C 10
此时就会出现一下界面，证明安装成功：
#rping -s -a 10.20.15.23 -v -C 10
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
server DISCONNECT EVENT...
waitfor RDMA_READ_ADV state 10
#rping -c -a 10.20.15.23 -v -C 10
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
client DISCONNECT EVENT...
------------------------------------------------------
利用udaddy来测试，出现一下结果代表成功：
x86_64]#udaddy
udaddy: starting server
receiving data transfers
sending replies
data transfers complete
test completereturn status 0
root]#udaddy -s 10.20.15.23
udaddy: starting client
udaddy: connecting
initiating data transfers
receiving data transfers
data transfers complete
test completereturn status 0
原文：https://www.cnblogs.com/kaishirenshi/p/10286270.html