firewall-cmd --permanent --add-service=nfs
firewall-cmd --permanent --add-service=mountd
firewall-cmd --permanent --add-service=rpc-bind
firewall-cmd --reloadSaturday, February 29, 2020
Centos NFS Firewalld rules
Sunday, February 16, 2020
Thursday, February 13, 2020
Iozone & dd on the Gigabyte AORUS NVMe Gen4 1TB SSD
I am going to run a few tests on, there aren't really big performance differences between using ext4 with or without journal. Also using kernel 4.18.0x vs 5.3.3x doesn't make much of a difference.
This runs on an Asus motherboard: https://www.asus.com/us/Motherboards/TUF-GAMING-X570-PLUS/ with the AMD Ryzen 5 3600: https://www.amd.com/en/products/cpu/amd-ryzen-5-3600 and 64 GB of RAM from G-Skill (F4-3200C16-32GTZN)
The test drive is from Gigabyte and 1TB large: https://www.gigabyte.com/us/Solid-State-Drive/AORUS-NVMe-Gen4-SSD-1TB
Iozone: Performance Test of File I/O
Version $Revision: 3.489 $
Compiled for 64 bit mode.
Build: linux-AMD64
Most importantly, I am re-mounting the drive for each test, that clears the buffer cache. Running watch -n 0.2 free -h I can observe the wanted effect.
The exact command used to run the tests is:
./iozone -e -r 4 -r 8 -r 16 -r 32 -r 64 -r 128 -r 512 -r 1024 -r 2048 -r 4096 -r 8192 -r 16384 -s 4g -i 0 -i 1 -i 2 -i 8 -f /apps/tfile -U /apps
CentOS Linux release 8.1.1911 (Core)
4.18.0-147.5.1.el8_1.x86_64
mkfs.ext4 /dev/nvme0n1
mkfs.ext4 /dev/nvme0n1
mkfs.ext4 /dev/nvme0n1
This runs on an Asus motherboard: https://www.asus.com/us/Motherboards/TUF-GAMING-X570-PLUS/ with the AMD Ryzen 5 3600: https://www.amd.com/en/products/cpu/amd-ryzen-5-3600 and 64 GB of RAM from G-Skill (F4-3200C16-32GTZN)
The test drive is from Gigabyte and 1TB large: https://www.gigabyte.com/us/Solid-State-Drive/AORUS-NVMe-Gen4-SSD-1TB
Iozone: Performance Test of File I/O
Version $Revision: 3.489 $
Compiled for 64 bit mode.
Build: linux-AMD64
Most importantly, I am re-mounting the drive for each test, that clears the buffer cache. Running watch -n 0.2 free -h I can observe the wanted effect.
The exact command used to run the tests is:
./iozone -e -r 4 -r 8 -r 16 -r 32 -r 64 -r 128 -r 512 -r 1024 -r 2048 -r 4096 -r 8192 -r 16384 -s 4g -i 0 -i 1 -i 2 -i 8 -f /apps/tfile -U /apps
CentOS Linux release 8.1.1911 (Core)
4.18.0-147.5.1.el8_1.x86_64
mkfs.ext4 /dev/nvme0n1
| random | random | ||||||
| testfile MB | reclen kB | write | rewrite | read | reread | read | write |
| 4096 | 4 | 1609 | 1892 | 2391 | 2291 | 98 | 1672 |
| 4096 | 8 | 1669 | 1972 | 2386 | 2289 | 167 | 1831 |
| 4096 | 16 | 1700 | 2049 | 2398 | 2299 | 256 | 1947 |
| 4096 | 32 | 1720 | 2075 | 2395 | 2302 | 314 | 1102 |
| 4096 | 64 | 1733 | 2079 | 2301 | 2303 | 503 | 2048 |
| 4096 | 128 | 1723 | 2061 | 2303 | 2298 | 728 | 2055 |
| 4096 | 512 | 1659 | 1953 | 2406 | 2309 | 1747 | 1969 |
| 4096 | 1024 | 1657 | 1975 | 3458 | 3405 | 2176 | 1968 |
| 4096 | 2048 | 1661 | 1972 | 2304 | 2307 | 2796 | 1961 |
| 4096 | 4096 | 1669 | 1964 | 3879 | 3876 | 3255 | 1955 |
| 4096 | 8192 | 1668 | 1952 | 4143 | 3866 | 3732 | 1948 |
| 4096 | 16384 | 1605 | 1898 | 4082 | 4079 | 3824 | 1901 |
mkfs.ext4 /dev/nvme0n1
tune2fs -O ^has_journal /dev/nvme0n1
| random | random | ||||||
| testfile MB | reclen kB | write | rewrite | read | reread | read | write |
| 4096 | 4 | 1674 | 1999 | 2300 | 2298 | 97 | 1734 |
| 4096 | 8 | 1746 | 2086 | 2302 | 2302 | 167 | 1911 |
| 4096 | 16 | 1787 | 2124 | 2302 | 2302 | 254 | 2018 |
| 4096 | 32 | 1808 | 2157 | 2302 | 2305 | 314 | 2105 |
| 4096 | 64 | 1812 | 2182 | 2304 | 2298 | 502 | 2144 |
| 4096 | 128 | 1801 | 2153 | 2305 | 2299 | 808 | 2122 |
| 4096 | 512 | 1735 | 2037 | 2315 | 2378 | 1819 | 2028 |
| 4096 | 1024 | 1735 | 2046 | 3500 | 3456 | 2157 | 2034 |
| 4096 | 2048 | 1748 | 2036 | 2310 | 2307 | 2778 | 2042 |
| 4096 | 4096 | 1745 | 2031 | 3885 | 3903 | 3249 | 2030 |
| 4096 | 8192 | 1731 | 2016 | 4008 | 3909 | 3741 | 2028 |
| 4096 | 16384 | 1674 | 1977 | 4072 | 4059 | 3997 | 1979 |
mkfs.ext4 /dev/nvme0n1
tune2fs -O ^has_journal /dev/nvme0n1
tune2fs -o ^discard /dev/nvme0n1
| random | random | ||||||
| testfile MB | reclen kB | write | rewrite | read | reread | read | write |
| 4096 | 4 | 1659 | 1980 | 2394 | 2261 | 96 | 1735 |
| 4096 | 8 | 1737 | 2082 | 2342 | 2291 | 117 | 1922 |
| 4096 | 16 | 1724 | 2133 | 2398 | 2393 | 252 | 2028 |
| 4096 | 32 | 1805 | 2157 | 2398 | 2298 | 310 | 2100 |
| 4096 | 64 | 1815 | 2175 | 2302 | 2297 | 493 | 2133 |
| 4096 | 128 | 1807 | 2087 | 2388 | 2295 | 838 | 2142 |
| 4096 | 512 | 1736 | 2048 | 2439 | 2313 | 1756 | 2038 |
| 4096 | 1024 | 1736 | 2035 | 3471 | 3325 | 2115 | 2047 |
| 4096 | 2048 | 1732 | 2043 | 2403 | 2404 | 2680 | 2050 |
| 4096 | 4096 | 1745 | 2046 | 3879 | 3896 | 3260 | 2045 |
| 4096 | 8192 | 1739 | 2027 | 3975 | 3884 | 3727 | 2032 |
| 4096 | 16384 | 1675 | 1978 | 4071 | 4055 | 3983 | 1985 |
CentOS Linux release 8.1.1911 (Core)
5.5.3-1.el8.elrepo.x86_64
mkfs.ext4 /dev/nvme0n1
| random | random | ||||||
| testfile MB | reclen kB | write | rewrite | read | reread | read | write |
| 4096 | 4 | 1583 | 1857 | 2397 | 2297 | 97 | 1647 |
| 4096 | 8 | 1639 | 1957 | 2399 | 2258 | 167 | 1819 |
| 4096 | 16 | 1687 | 1981 | 2386 | 2385 | 248 | 1920 |
| 4096 | 32 | 1697 | 2021 | 2393 | 2384 | 316 | 2000 |
| 4096 | 64 | 1722 | 2048 | 2395 | 2397 | 509 | 2031 |
| 4096 | 128 | 1711 | 2010 | 2397 | 2298 | 841 | 2016 |
| 4096 | 512 | 1644 | 1889 | 2713 | 2585 | 1934 | 1939 |
| 4096 | 1024 | 1659 | 1919 | 3490 | 3468 | 2258 | 1942 |
| 4096 | 2048 | 1649 | 1936 | 2402 | 2404 | 2766 | 1930 |
| 4096 | 4096 | 1649 | 1880 | 3920 | 3932 | 3320 | 1935 |
| 4096 | 8192 | 1642 | 1867 | 4035 | 3986 | 3727 | 1914 |
| 4096 | 16384 | 1598 | 1871 | 4124 | 4118 | 4064 | 1870 |
mkfs.ext4 /dev/nvme0n1
tune2fs -O ^has_journal /dev/nvme0n1
| random | random | ||||||
| testfile MB | reclen kB | write | rewrite | read | reread | read | write |
| 4096 | 4 | 1660 | 1938 | 2395 | 2392 | 99 | 1716 |
| 4096 | 8 | 1723 | 2025 | 2397 | 2389 | 166 | 1887 |
| 4096 | 16 | 1770 | 2028 | 2305 | 2393 | 255 | 1997 |
| 4096 | 32 | 1781 | 2067 | 2302 | 2390 | 313 | 2074 |
| 4096 | 64 | 1800 | 2143 | 2396 | 2395 | 518 | 2122 |
| 4096 | 128 | 1792 | 2123 | 2398 | 2393 | 800 | 2122 |
| 4096 | 512 | 1724 | 1974 | 2423 | 2399 | 1821 | 2009 |
| 4096 | 1024 | 1725 | 2019 | 3483 | 3465 | 2210 | 2012 |
| 4096 | 2048 | 1727 | 1964 | 2401 | 2404 | 2818 | 2006 |
| 4096 | 4096 | 1729 | 2023 | 3909 | 3931 | 3307 | 1997 |
| 4096 | 8192 | 1718 | 2005 | 4038 | 4040 | 3743 | 1971 |
| 4096 | 16384 | 1660 | 1912 | 4136 | 4111 | 4071 | 1938 |
The exact command used to run the tests is:
./iozone -r 4 -r 8 -r 16 -r 32 -r 64 -r 128 -r 512 -r 1024 -r 2048 -r 4096 -r 8192 -r 16384 -s 4g -i 0 -i 1 -i 2 -i 8 -f /apps/tfile -U /apps
The biggest difference here is, not specifying the -e option which then excludes flush (fsync,fflush) in the timing calculations.
./iozone -r 4 -r 8 -r 16 -r 32 -r 64 -r 128 -r 512 -r 1024 -r 2048 -r 4096 -r 8192 -r 16384 -s 4g -i 0 -i 1 -i 2 -i 8 -f /apps/tfile -U /apps
The biggest difference here is, not specifying the -e option which then excludes flush (fsync,fflush) in the timing calculations.
mkfs.ext4 /dev/nvme0n1
tune2fs -O ^has_journal /dev/nvme0n1
dd if=/dev/zero of=/apps/file1.txt count=50240 bs=1M conv=fsync
50240+0 records in
50240+0 records out
52680458240 bytes (53 GB, 49 GiB) copied, 26.8013 s, 2.0 GB/s
dd if=/dev/zero of=/apps/file2.txt count=50240 bs=1M
50240+0 records in
50240+0 records out
52680458240 bytes (53 GB, 49 GiB) copied, 26.3247 s, 2.0 GB/s
| random | random | ||||||
| testfile MB | reclen kB | write | rewrite | read | reread | read | write |
| 4096 | 4 | 2782 | 3831 | 2397 | 2389 | 99 | 2939 |
| 4096 | 8 | 2997 | 4164 | 2397 | 2390 | 168 | 3507 |
| 4096 | 16 | 3124 | 4427 | 2398 | 2395 | 259 | 3974 |
| 4096 | 32 | 3211 | 4523 | 2400 | 2393 | 309 | 4314 |
| 4096 | 64 | 3296 | 4589 | 2398 | 2395 | 507 | 4467 |
| 4096 | 128 | 3217 | 4574 | 2398 | 2392 | 799 | 4537 |
| 4096 | 512 | 3037 | 4079 | 2487 | 2402 | 1808 | 4032 |
| 4096 | 1024 | 3022 | 4093 | 3499 | 3464 | 2292 | 4088 |
| 4096 | 2048 | 3025 | 4063 | 2406 | 2404 | 2857 | 4068 |
| 4096 | 4096 | 3023 | 4065 | 3917 | 3934 | 3282 | 4067 |
| 4096 | 8192 | 3003 | 4032 | 4029 | 4024 | 3708 | 4061 |
| 4096 | 16384 | 2836 | 3804 | 4120 | 4110 | 3885 | 3817 |
dd if=/dev/zero of=/apps/file1.txt count=50240 bs=1M conv=fsync
50240+0 records in
50240+0 records out
52680458240 bytes (53 GB, 49 GiB) copied, 26.8013 s, 2.0 GB/s
dd if=/dev/zero of=/apps/file2.txt count=50240 bs=1M
50240+0 records in
50240+0 records out
52680458240 bytes (53 GB, 49 GiB) copied, 26.3247 s, 2.0 GB/s
Monday, February 10, 2020
Centos 8 & Bonding 2 x 10GbE ports
uuidgen bond0, that will create the UUID needed
more /etc/sysconfig/network-scripts/ifcfg-bond0
BONDING_OPTS="downdelay=0 miimon=100 mode=balance-alb updelay=0"
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_DISABLED=yes
IPV6INIT=no
NAME=bond0
UUID=1a59e993-a523-40b2-92f3-a59aef54d932
DEVICE=bond0
ONBOOT=yes
MTU=9000
MACADDR=00:51:82:11:22:02
more /etc/sysconfig/network-scripts/ifcfg-enp4s0f0
TYPE=Ethernet
NAME=enp4s0f0
UUID=e4d0c588-1938-4c5a-b8ec-f4c2af13991c
DEVICE=enp4s0f0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
more /etc/sysconfig/network-scripts/ifcfg-enp4s0f1
TYPE=Ethernet
NAME=enp4s0f1
UUID=a1422007-0895-4817-97b5-2b07d960c7c1
DEVICE=enp4s0f1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
Then activate the connections:
ifup bond0
ifup enp4s0f0
ifup enp4s0f1
Configured a LAG group in the switch for the ports in scope
Run next to check the status of the of the connection
> nmcli
[root@node-2 ~]# nmcli
bond0: connected to bond0
"bond0"
bond, 00:51:82:11:22:02, sw, mtu 9000
ip4 default
inet4 192.168.1.110/16
route4 0.0.0.0/0
route4 192.168.0.0/16
enp4s0f0: connected to enp4s0f0
"Intel X550T"
ethernet (ixgbe), 00:51:82:11:22:02, hw, mtu 9000
master bond0
enp4s0f1: connected to enp4s0f1
"Intel X550T"
ethernet (ixgbe), A0:36:9F:27:B5:52, hw, mtu 9000
master bond0
enp6s0: disconnected
"Realtek RTL8111/8168/8411"
1 connection available
ethernet (r8169), A8:5E:45:E2:F1:04, hw, mtu 1500
more /etc/sysconfig/network-scripts/ifcfg-bond0
BONDING_OPTS="downdelay=0 miimon=100 mode=balance-alb updelay=0"
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_DISABLED=yes
IPV6INIT=no
NAME=bond0
UUID=1a59e993-a523-40b2-92f3-a59aef54d932
DEVICE=bond0
ONBOOT=yes
MTU=9000
MACADDR=00:51:82:11:22:02
more /etc/sysconfig/network-scripts/ifcfg-enp4s0f0
TYPE=Ethernet
NAME=enp4s0f0
UUID=e4d0c588-1938-4c5a-b8ec-f4c2af13991c
DEVICE=enp4s0f0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
more /etc/sysconfig/network-scripts/ifcfg-enp4s0f1
TYPE=Ethernet
NAME=enp4s0f1
UUID=a1422007-0895-4817-97b5-2b07d960c7c1
DEVICE=enp4s0f1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
Then activate the connections:
ifup bond0
ifup enp4s0f0
ifup enp4s0f1
Configured a LAG group in the switch for the ports in scope
Run next to check the status of the of the connection
> nmcli
[root@node-2 ~]# nmcli
bond0: connected to bond0
"bond0"
bond, 00:51:82:11:22:02, sw, mtu 9000
ip4 default
inet4 192.168.1.110/16
route4 0.0.0.0/0
route4 192.168.0.0/16
enp4s0f0: connected to enp4s0f0
"Intel X550T"
ethernet (ixgbe), 00:51:82:11:22:02, hw, mtu 9000
master bond0
enp4s0f1: connected to enp4s0f1
"Intel X550T"
ethernet (ixgbe), A0:36:9F:27:B5:52, hw, mtu 9000
master bond0
enp6s0: disconnected
"Realtek RTL8111/8168/8411"
1 connection available
ethernet (r8169), A8:5E:45:E2:F1:04, hw, mtu 1500
Tune TCP network for data transfer in 10GbE or higher network
Some background information: https://community.rti.com/static/documentation/perftest/3.0/tuning_os.html
10GbE, Centos 7: /etc/sysctl.d/99-sysctl.conf
net.core.rmem_default = 524287
net.core.rmem_max = 16777216
net.core.wmem_default = 524287
net.core.wmem_max = 16777216
https://community.rti.com/kb/how-can-i-improve-my-throughput-performance-linux
echo "8388608" > /proc/sys/net/ipv4/ipfrag_high_threshold
Some more background: https://mapr.com/docs/51/AdministratorGuide/Configure-NFS-Write-Perfo-Thekerneltunablevalu-d3e72.html
echo "options sunrpc tcp_slot_table_entries=128" >> /etc/modprobe.d/sunrpc.conf
echo 128 > /proc/sys/sunrpc/tcp_slot_table_entries
Some tuning ideas for Windows: https://www.drastic.tv/support-59/supporttipstechnical/320-optimizing-windows-networking
https://www.cyberciti.biz/faq/linux-tcp-tuning/
https://www.ibm.com/support/knowledgecenter/linuxonibm/liaag/wkvm/wkvm_c_tune_tcpip.htm
/etc/sysctl.d/10-network.conf
#https://www.cyberciti.biz/faq/linux-tcp-tuning/
net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.ipv4.tcp_rmem=2097152 16777216 33554432
net.ipv4.tcp_wmem=2097152 16777216 33554432
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_no_metrics_save=1
net.core.netdev_max_backlog=5000
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_low_latency=1
Thursday, February 6, 2020
Install mpi4py
yum install -y openmpi-devel
export CC=/usr/lib64/openmpi/bin/mpicc
pip install mpi4py
export CC=/usr/lib64/openmpi/bin/mpicc
pip install mpi4py
Subscribe to:
Posts (Atom)