OpenBSD NFS Performance Tuning - Part 2
·
11min
·
Dan F.
Table of Contents
Part two NFS performance tuning. I have some new data to publish. I reran the full NFS test, this time with the script transferring three files, with one sync and unmount in between the three transfers. I saw more interesting behavior that is worth noting, as well as publishing the script used to compile these logs.
First off, here is the first table of 20 entries, sorted by the smallest chunk transferred:
iothreads protocol mount readahead buffer time(256/512/1024M) transfer speed(256/512/1024M) sync time umount time
8 tcp soft 1 32768 0:00:04 0:00:10 0:00:23 58.79MB/s 49.09MB/s 43.51MB/s 10.67 84.07
9 tcp hard 4 32768 0:00:04 0:00:11 0:00:22 58.31MB/s 46.02MB/s 44.68MB/s 1.20 41.46
14 tcp hard 3 32768 0:00:04 0:00:11 0:00:24 58.15MB/s 44.69MB/s 41.43MB/s 11.35 52.50
20 tcp hard 3 32768 0:00:04 0:00:11 0:00:26 58.07MB/s 43.04MB/s 38.03MB/s 8.64 59.78
11 tcp hard 2 32768 0:00:04 0:00:10 0:00:23 56.59MB/s 48.88MB/s 44.43MB/s 0.80 45.29
10 tcp hard 2 32768 0:00:04 0:00:10 0:00:23 56.17MB/s 49.19MB/s 43.63MB/s 2.04 53.81
10 tcp hard 3 32768 0:00:04 0:00:11 0:00:23 55.94MB/s 43.11MB/s 42.95MB/s 7.81 54.39
12 tcp hard 3 32768 0:00:04 0:00:11 0:00:25 55.69MB/s 45.27MB/s 40.19MB/s 6.79 56.81
19 tcp hard 3 32768 0:00:04 0:00:11 0:00:25 55.43MB/s 46.25MB/s 40.02MB/s 7.34 45.69
14 tcp soft 2 32768 0:00:04 0:00:11 0:00:25 55.06MB/s 44.01MB/s 39.78MB/s 12.21 51.23
10 tcp soft 3 32768 0:00:04 0:00:10 0:00:23 54.89MB/s 48.13MB/s 43.27MB/s 6.27 52.66
20 tcp hard 2 32768 0:00:04 0:00:12 0:00:27 54.58MB/s 40.78MB/s 37.39MB/s 1.91 57.69
14 tcp hard 2 32768 0:00:04 0:00:12 0:00:25 54.54MB/s 41.13MB/s 40.66MB/s 4.92 50.05
11 tcp soft 1 32768 0:00:04 0:00:10 0:00:23 54.43MB/s 46.87MB/s 43.46MB/s 20.84 46.20
18 tcp hard 1 32768 0:00:04 0:00:12 0:00:24 54.30MB/s 42.14MB/s 41.23MB/s 7.14 53.33
14 tcp soft 3 32768 0:00:04 0:00:11 0:00:24 54.29MB/s 43.11MB/s 41.10MB/s 4.48 50.62
18 tcp hard 2 32768 0:00:04 0:00:12 0:00:26 54.25MB/s 42.56MB/s 39.09MB/s 0.96 50.61
15 tcp hard 3 32768 0:00:04 0:00:12 0:00:24 54.24MB/s 40.13MB/s 41.00MB/s 10.90 45.38
17 tcp hard 2 32768 0:00:04 0:00:12 0:00:26 54.23MB/s 42.00MB/s 39.04MB/s 13.67 43.90
12 tcp soft 2 32768 0:00:04 0:00:11 0:00:24 54.06MB/s 45.37MB/s 41.60MB/s 2.68 51.55
Next up, here are the top 20 fastest transfers of 512M, from the same test:
iothreads protocol mount readahead buffer time(256/512/1024M) transfer speed(256/512/1024M) sync time umount time
6 tcp hard 3 32768 0:00:05 0:00:09 0:00:22 48.60MB/s 52.03MB/s 44.89MB/s 2.04 51.39
9 tcp hard 1 32768 0:00:05 0:00:09 0:00:22 49.75MB/s 51.93MB/s 45.53MB/s 18.70 44.10
10 tcp soft 2 32768 0:00:04 0:00:09 0:00:24 51.52MB/s 51.66MB/s 42.59MB/s 11.11 53.44
7 tcp hard 3 32768 0:00:05 0:00:10 0:00:22 48.32MB/s 50.88MB/s 46.30MB/s 28.05 73.16
6 tcp soft 1 32768 0:00:05 0:00:10 0:00:21 48.72MB/s 50.56MB/s 47.11MB/s 2.08 51.73
4 tcp soft 3 32768 0:00:05 0:00:10 0:00:21 49.95MB/s 50.54MB/s 48.69MB/s 0.29 66.03
7 tcp soft 1 32768 0:00:05 0:00:10 0:00:22 47.29MB/s 50.51MB/s 45.89MB/s 21.27 67.94
9 tcp hard 2 32768 0:00:05 0:00:10 0:00:23 49.25MB/s 50.31MB/s 43.46MB/s 7.92 42.74
5 tcp soft 3 32768 0:00:04 0:00:10 0:00:21 52.20MB/s 50.21MB/s 48.21MB/s 13.79 43.81
8 tcp soft 2 32768 0:00:05 0:00:10 0:00:25 47.25MB/s 49.64MB/s 39.93MB/s 4.88 83.95
6 tcp hard 2 32768 0:00:05 0:00:10 0:00:22 48.94MB/s 49.61MB/s 46.43MB/s 0.56 42.96
9 tcp soft 2 32768 0:00:05 0:00:10 0:00:23 47.40MB/s 49.56MB/s 44.33MB/s 3.28 44.51
5 tcp hard 1 32768 0:00:05 0:00:10 0:00:22 50.17MB/s 49.47MB/s 45.62MB/s 20.57 45.36
7 tcp hard 4 32768 0:00:05 0:00:10 0:00:22 47.17MB/s 49.20MB/s 45.14MB/s 31.66 73.40
10 tcp hard 2 32768 0:00:04 0:00:10 0:00:23 56.17MB/s 49.19MB/s 43.63MB/s 2.04 53.81
8 tcp soft 1 32768 0:00:04 0:00:10 0:00:23 58.79MB/s 49.09MB/s 43.51MB/s 10.67 84.07
8 tcp soft 3 32768 0:00:05 0:00:10 0:00:24 48.62MB/s 49.00MB/s 42.31MB/s 2.06 51.68
6 tcp hard 1 32768 0:00:05 0:00:10 0:00:21 51.16MB/s 48.90MB/s 46.81MB/s 12.09 52.83
11 tcp hard 2 32768 0:00:04 0:00:10 0:00:23 56.59MB/s 48.88MB/s 44.43MB/s 0.80 45.29
13 tcp soft 2 32768 0:00:04 0:00:10 0:00:23 51.92MB/s 48.87MB/s 43.36MB/s 2.30 43.94
And finally, here are the top 20 transfers for the largest 1G file, from the same test:
iothreads protocol mount readahead buffer time(256/512/1024M) transfer speed(256/512/1024M) sync time umount time
6 tcp hard 4 32768 0:00:05 0:00:10 0:00:20 49.81MB/s 47.70MB/s 48.85MB/s 0.80 53.48
5 tcp hard 2 32768 0:00:05 0:00:11 0:00:20 49.90MB/s 45.07MB/s 48.77MB/s 14.55 45.18
4 tcp soft 3 32768 0:00:05 0:00:10 0:00:21 49.95MB/s 50.54MB/s 48.69MB/s 0.29 66.03
4 tcp hard 4 32768 0:00:04 0:00:11 0:00:21 52.42MB/s 45.46MB/s 48.50MB/s 0.11 58.33
5 tcp soft 3 32768 0:00:04 0:00:10 0:00:21 52.20MB/s 50.21MB/s 48.21MB/s 13.79 43.81
5 tcp soft 2 32768 0:00:04 0:00:10 0:00:21 51.73MB/s 48.63MB/s 48.13MB/s 6.14 41.85
5 tcp soft 1 32768 0:00:04 0:00:10 0:00:21 52.08MB/s 48.08MB/s 47.72MB/s 1.56 42.19
4 tcp soft 4 32768 0:00:05 0:00:10 0:00:21 50.74MB/s 48.37MB/s 47.70MB/s 0.23 67.55
5 tcp hard 3 32768 0:00:05 0:00:10 0:00:21 49.96MB/s 48.34MB/s 47.50MB/s 11.49 43.22
6 tcp soft 2 32768 0:00:05 0:00:10 0:00:21 45.99MB/s 47.30MB/s 47.44MB/s 4.03 52.66
5 tcp hard 4 32768 0:00:05 0:00:11 0:00:21 50.65MB/s 45.52MB/s 47.31MB/s 3.68 44.94
6 tcp soft 3 32768 0:00:10 0:00:10 0:00:21 23.45MB/s 48.69MB/s 47.16MB/s 12.89 51.94
4 tcp hard 1 32768 0:00:05 0:00:11 0:00:21 50.33MB/s 44.79MB/s 47.12MB/s 0.37 72.02
6 tcp soft 1 32768 0:00:05 0:00:10 0:00:21 48.72MB/s 50.56MB/s 47.11MB/s 2.08 51.73
6 tcp hard 1 32768 0:00:05 0:00:10 0:00:21 51.16MB/s 48.90MB/s 46.81MB/s 12.09 52.83
5 tcp soft 4 32768 0:00:05 0:00:10 0:00:21 50.32MB/s 47.55MB/s 46.76MB/s 16.86 45.05
7 tcp soft 3 32768 0:00:05 0:00:11 0:00:22 48.92MB/s 44.58MB/s 46.54MB/s 22.80 69.44
6 tcp hard 2 32768 0:00:05 0:00:10 0:00:22 48.94MB/s 49.61MB/s 46.43MB/s 0.56 42.96
4 tcp soft 2 32768 0:00:04 0:00:10 0:00:22 51.88MB/s 46.78MB/s 46.33MB/s 0.25 63.40
7 tcp hard 3 32768 0:00:05 0:00:10 0:00:22 48.32MB/s 50.88MB/s 46.30MB/s 28.05 73.16
What immediately stuck out to me is that while iothreads seemed to have no affect on transfer speeds in my first posting on NFS performance, I can definitely see something here when comparing different sized transfers. I looks like the iothreads has a major impact when transferring smaller sized files. So perhaps, depending on your own environment, increase the iothread count when transferring multiple small files, but consider keeping the iothread count low, when moving fewer, but larger data files.
The script
Here is the script that was used to generate these logs:
#!/usr/local/bin/bash
# This script tests the following parameters and how they affect nfs data transfer:
# iothreads 4-20
# TCP vs UDP
# soft vs hard
# readahead 1-4
# buffer size 16K-65K
# Log location
log=/tmp/nfstest_triple.log
# NFS share to mount
share=<your share>
# Mount location
mnt_dir=/data
# Create data array
data_time=()
data_trans=()
data_sync=()
data_umount=()
# Data file to be created
data[0]=/tmp/nfstest.data1
data[1]=/tmp/nfstest.data2
data[2]=/tmp/nfstest.data3
# Create data lump
echo "$(date) - Creating data files"
dd if=/dev/random of=${data[0]} bs=16384 count=16384 >/dev/null 2>&1 # 256M
dd if=/dev/random of=${data[1]} bs=16384 count=32768 >/dev/null 2>&1 # 512M
dd if=/dev/random of=${data[2]} bs=16384 count=65536 >/dev/null 2>&1 # 1G
# Declare starting point in log
echo "Starting test..." >> $log
echo "iothreads,protocol,hardness,readahead,buffer,transfer time,transfer speed,sync time,umount time" >> $log
# Start first with iothreads
for iothreads in {4..20}
do
echo "$(date) - Using $iothreads iothreads"
# Set iothreads
doas sysctl vfs.nfs.iothreads=$iothreads >/dev/null 2>&1
for proto in udp tcp
do
echo "$(date) - Using $proto"
if [[ $proto == "tcp" ]]; then
protocol='-T'
else
protocol=""
fi
for hard in hard soft
do
echo "$(date) - Using $hard"
if [[ $hard == "soft" ]]; then
hardness='-s'
else
hardness=""
fi
for readahead in {1..4}
do
echo "$(date) - Using $readahead readahead"
for size in 16384 32768 65536
do
echo "$(date) - Using a $size r/w buffer"
# Unmount share
doas umount $mnt_dir >/dev/null 2>&1
echo "$(date) - Remounting share"
doas mount_nfs $protocol $hardness -a $readahead -r $size -w $size $share $mnt_dir
for e in 0 1 2
do
echo "$(date) - Using data lump $e"
# Transfer file
echo -n "$(date) - Transferring data: "
action=$(rsync --progress ${data[$e]} $mnt_dir)
# echo transfer info
action_thr=$(echo "$action" | grep xfr | awk '{print $4}')
action_time=$(echo "$action" | grep xfr | awk '{print $5}')
echo "$action_thr"
# Delete file
echo "$(date) - Remove transferred data ${e}"
rm -f $mnt_dir/$data
data_time[$e]=$action_time
data_trans[$e]=$action_thr
done
# Sync
echo "$(date) - Syncing"
sync_time=$({ time -p sync;} 2>&1 | grep real | awk '{print $2}')
# Unmount share and get duration
echo "$(date) - Unmounting share"
umount_time=$({ time -p doas umount $mnt_dir;} 2>&1 | grep real | awk '{print $2}')
data_time[$e]=$action_time
data_trans[$e]=$action_thr
data_sync[$e]=$sync_time
data_umount[$e]=$umount_time
# Save transfer info to log
# echo "$iothreads,$proto,$hard,$readahead,$size,$action_time,$action_thr,$sync_time,$umount_time" >> $log
echo "$iothreads,$proto,$hard,$readahead,$size,\
${data_time[0]},${data_time[1]},${data_time[2]},\
${data_trans[0]},${data_trans[1]},${data_trans[2]},\
$sync_time,$umount_time" >> $log
done
done
done
done
done
Has been tested on OpenBSD 6.5-beta