OpenBSD NFS Performance Tuning - Part 2

· 11min · Dan F.
Table of Contents

Part two NFS performance tuning. I have some new data to publish. I reran the full NFS test, this time with the script transferring three files, with one sync and unmount in between the three transfers. I saw more interesting behavior that is worth noting, as well as publishing the script used to compile these logs.

First off, here is the first table of 20 entries, sorted by the smallest chunk transferred:

iothreads   protocol    mount readahead buffer time(256/512/1024M)          transfer speed(256/512/1024M)       sync time   umount time
8           tcp         soft  1         32768  0:00:04  0:00:10  0:00:23    58.79MB/s  49.09MB/s  43.51MB/s     10.67       84.07
9           tcp         hard  4         32768  0:00:04  0:00:11  0:00:22    58.31MB/s  46.02MB/s  44.68MB/s     1.20        41.46
14          tcp         hard  3         32768  0:00:04  0:00:11  0:00:24    58.15MB/s  44.69MB/s  41.43MB/s     11.35       52.50
20          tcp         hard  3         32768  0:00:04  0:00:11  0:00:26    58.07MB/s  43.04MB/s  38.03MB/s     8.64        59.78
11          tcp         hard  2         32768  0:00:04  0:00:10  0:00:23    56.59MB/s  48.88MB/s  44.43MB/s     0.80        45.29
10          tcp         hard  2         32768  0:00:04  0:00:10  0:00:23    56.17MB/s  49.19MB/s  43.63MB/s     2.04        53.81
10          tcp         hard  3         32768  0:00:04  0:00:11  0:00:23    55.94MB/s  43.11MB/s  42.95MB/s     7.81        54.39
12          tcp         hard  3         32768  0:00:04  0:00:11  0:00:25    55.69MB/s  45.27MB/s  40.19MB/s     6.79        56.81
19          tcp         hard  3         32768  0:00:04  0:00:11  0:00:25    55.43MB/s  46.25MB/s  40.02MB/s     7.34        45.69
14          tcp         soft  2         32768  0:00:04  0:00:11  0:00:25    55.06MB/s  44.01MB/s  39.78MB/s     12.21       51.23
10          tcp         soft  3         32768  0:00:04  0:00:10  0:00:23    54.89MB/s  48.13MB/s  43.27MB/s     6.27        52.66
20          tcp         hard  2         32768  0:00:04  0:00:12  0:00:27    54.58MB/s  40.78MB/s  37.39MB/s     1.91        57.69
14          tcp         hard  2         32768  0:00:04  0:00:12  0:00:25    54.54MB/s  41.13MB/s  40.66MB/s     4.92        50.05
11          tcp         soft  1         32768  0:00:04  0:00:10  0:00:23    54.43MB/s  46.87MB/s  43.46MB/s     20.84       46.20
18          tcp         hard  1         32768  0:00:04  0:00:12  0:00:24    54.30MB/s  42.14MB/s  41.23MB/s     7.14        53.33
14          tcp         soft  3         32768  0:00:04  0:00:11  0:00:24    54.29MB/s  43.11MB/s  41.10MB/s     4.48        50.62
18          tcp         hard  2         32768  0:00:04  0:00:12  0:00:26    54.25MB/s  42.56MB/s  39.09MB/s     0.96        50.61
15          tcp         hard  3         32768  0:00:04  0:00:12  0:00:24    54.24MB/s  40.13MB/s  41.00MB/s     10.90       45.38
17          tcp         hard  2         32768  0:00:04  0:00:12  0:00:26    54.23MB/s  42.00MB/s  39.04MB/s     13.67       43.90
12          tcp         soft  2         32768  0:00:04  0:00:11  0:00:24    54.06MB/s  45.37MB/s  41.60MB/s     2.68        51.55

Next up, here are the top 20 fastest transfers of 512M, from the same test:

iothreads   protocol    mount readahead buffer time(256/512/1024M)          transfer speed(256/512/1024M)       sync time   umount time
6           tcp         hard  3         32768  0:00:05  0:00:09  0:00:22    48.60MB/s  52.03MB/s  44.89MB/s     2.04        51.39
9           tcp         hard  1         32768  0:00:05  0:00:09  0:00:22    49.75MB/s  51.93MB/s  45.53MB/s     18.70       44.10
10          tcp         soft  2         32768  0:00:04  0:00:09  0:00:24    51.52MB/s  51.66MB/s  42.59MB/s     11.11       53.44
7           tcp         hard  3         32768  0:00:05  0:00:10  0:00:22    48.32MB/s  50.88MB/s  46.30MB/s     28.05       73.16
6           tcp         soft  1         32768  0:00:05  0:00:10  0:00:21    48.72MB/s  50.56MB/s  47.11MB/s     2.08        51.73
4           tcp         soft  3         32768  0:00:05  0:00:10  0:00:21    49.95MB/s  50.54MB/s  48.69MB/s     0.29        66.03
7           tcp         soft  1         32768  0:00:05  0:00:10  0:00:22    47.29MB/s  50.51MB/s  45.89MB/s     21.27       67.94
9           tcp         hard  2         32768  0:00:05  0:00:10  0:00:23    49.25MB/s  50.31MB/s  43.46MB/s     7.92        42.74
5           tcp         soft  3         32768  0:00:04  0:00:10  0:00:21    52.20MB/s  50.21MB/s  48.21MB/s     13.79       43.81
8           tcp         soft  2         32768  0:00:05  0:00:10  0:00:25    47.25MB/s  49.64MB/s  39.93MB/s     4.88        83.95
6           tcp         hard  2         32768  0:00:05  0:00:10  0:00:22    48.94MB/s  49.61MB/s  46.43MB/s     0.56        42.96
9           tcp         soft  2         32768  0:00:05  0:00:10  0:00:23    47.40MB/s  49.56MB/s  44.33MB/s     3.28        44.51
5           tcp         hard  1         32768  0:00:05  0:00:10  0:00:22    50.17MB/s  49.47MB/s  45.62MB/s     20.57       45.36
7           tcp         hard  4         32768  0:00:05  0:00:10  0:00:22    47.17MB/s  49.20MB/s  45.14MB/s     31.66       73.40
10          tcp         hard  2         32768  0:00:04  0:00:10  0:00:23    56.17MB/s  49.19MB/s  43.63MB/s     2.04        53.81
8           tcp         soft  1         32768  0:00:04  0:00:10  0:00:23    58.79MB/s  49.09MB/s  43.51MB/s     10.67       84.07
8           tcp         soft  3         32768  0:00:05  0:00:10  0:00:24    48.62MB/s  49.00MB/s  42.31MB/s     2.06        51.68
6           tcp         hard  1         32768  0:00:05  0:00:10  0:00:21    51.16MB/s  48.90MB/s  46.81MB/s     12.09       52.83
11          tcp         hard  2         32768  0:00:04  0:00:10  0:00:23    56.59MB/s  48.88MB/s  44.43MB/s     0.80        45.29
13          tcp         soft  2         32768  0:00:04  0:00:10  0:00:23    51.92MB/s  48.87MB/s  43.36MB/s     2.30        43.94

And finally, here are the top 20 transfers for the largest 1G file, from the same test:

iothreads   protocol    mount readahead buffer time(256/512/1024M)          transfer speed(256/512/1024M)       sync time   umount time
6           tcp         hard  4         32768  0:00:05  0:00:10  0:00:20    49.81MB/s  47.70MB/s  48.85MB/s     0.80        53.48
5           tcp         hard  2         32768  0:00:05  0:00:11  0:00:20    49.90MB/s  45.07MB/s  48.77MB/s     14.55       45.18
4           tcp         soft  3         32768  0:00:05  0:00:10  0:00:21    49.95MB/s  50.54MB/s  48.69MB/s     0.29        66.03
4           tcp         hard  4         32768  0:00:04  0:00:11  0:00:21    52.42MB/s  45.46MB/s  48.50MB/s     0.11        58.33
5           tcp         soft  3         32768  0:00:04  0:00:10  0:00:21    52.20MB/s  50.21MB/s  48.21MB/s     13.79       43.81
5           tcp         soft  2         32768  0:00:04  0:00:10  0:00:21    51.73MB/s  48.63MB/s  48.13MB/s     6.14        41.85
5           tcp         soft  1         32768  0:00:04  0:00:10  0:00:21    52.08MB/s  48.08MB/s  47.72MB/s     1.56        42.19
4           tcp         soft  4         32768  0:00:05  0:00:10  0:00:21    50.74MB/s  48.37MB/s  47.70MB/s     0.23        67.55
5           tcp         hard  3         32768  0:00:05  0:00:10  0:00:21    49.96MB/s  48.34MB/s  47.50MB/s     11.49       43.22
6           tcp         soft  2         32768  0:00:05  0:00:10  0:00:21    45.99MB/s  47.30MB/s  47.44MB/s     4.03        52.66
5           tcp         hard  4         32768  0:00:05  0:00:11  0:00:21    50.65MB/s  45.52MB/s  47.31MB/s     3.68        44.94
6           tcp         soft  3         32768  0:00:10  0:00:10  0:00:21    23.45MB/s  48.69MB/s  47.16MB/s     12.89       51.94
4           tcp         hard  1         32768  0:00:05  0:00:11  0:00:21    50.33MB/s  44.79MB/s  47.12MB/s     0.37        72.02
6           tcp         soft  1         32768  0:00:05  0:00:10  0:00:21    48.72MB/s  50.56MB/s  47.11MB/s     2.08        51.73
6           tcp         hard  1         32768  0:00:05  0:00:10  0:00:21    51.16MB/s  48.90MB/s  46.81MB/s     12.09       52.83
5           tcp         soft  4         32768  0:00:05  0:00:10  0:00:21    50.32MB/s  47.55MB/s  46.76MB/s     16.86       45.05
7           tcp         soft  3         32768  0:00:05  0:00:11  0:00:22    48.92MB/s  44.58MB/s  46.54MB/s     22.80       69.44
6           tcp         hard  2         32768  0:00:05  0:00:10  0:00:22    48.94MB/s  49.61MB/s  46.43MB/s     0.56        42.96
4           tcp         soft  2         32768  0:00:04  0:00:10  0:00:22    51.88MB/s  46.78MB/s  46.33MB/s     0.25        63.40
7           tcp         hard  3         32768  0:00:05  0:00:10  0:00:22    48.32MB/s  50.88MB/s  46.30MB/s     28.05       73.16

What immediately stuck out to me is that while iothreads seemed to have no affect on transfer speeds in my first posting on NFS performance, I can definitely see something here when comparing different sized transfers. I looks like the iothreads has a major impact when transferring smaller sized files. So perhaps, depending on your own environment, increase the iothread count when transferring multiple small files, but consider keeping the iothread count low, when moving fewer, but larger data files.

The script

Here is the script that was used to generate these logs:

#!/usr/local/bin/bash

# This script tests the following parameters and how they affect nfs data transfer:
# iothreads 4-20
# TCP vs UDP
# soft vs hard
# readahead 1-4
# buffer size 16K-65K

# Log location
log=/tmp/nfstest_triple.log

# NFS share to mount
share=<your share>

# Mount location
mnt_dir=/data

# Create data array
data_time=()
data_trans=()
data_sync=()
data_umount=()

# Data file to be created
data[0]=/tmp/nfstest.data1
data[1]=/tmp/nfstest.data2
data[2]=/tmp/nfstest.data3

# Create data lump
echo "$(date) - Creating data files"
dd if=/dev/random of=${data[0]} bs=16384 count=16384 >/dev/null 2>&1        # 256M
dd if=/dev/random of=${data[1]} bs=16384 count=32768 >/dev/null 2>&1        # 512M
dd if=/dev/random of=${data[2]} bs=16384 count=65536 >/dev/null 2>&1        # 1G

# Declare starting point in log
echo "Starting test..." >> $log
echo "iothreads,protocol,hardness,readahead,buffer,transfer time,transfer speed,sync time,umount time" >> $log

# Start first with iothreads
for iothreads in {4..20}
do
    echo "$(date) - Using $iothreads iothreads"

    # Set iothreads
    doas sysctl vfs.nfs.iothreads=$iothreads >/dev/null 2>&1

    for proto in udp tcp 
    do
        echo "$(date) - Using $proto"

        if [[ $proto == "tcp" ]]; then
            protocol='-T'
        else
            protocol=""
        fi
        for hard in hard soft
        do
            echo "$(date) - Using $hard"

            if [[ $hard == "soft" ]]; then
                hardness='-s'
            else
                hardness=""
            fi
            for readahead in {1..4}
            do
                echo "$(date) - Using $readahead readahead"

                for size in 16384 32768 65536 
                do
                    echo "$(date) - Using a $size r/w buffer"

                    # Unmount share
                    doas umount $mnt_dir >/dev/null 2>&1
                    echo "$(date) - Remounting share"
                    doas mount_nfs $protocol $hardness -a $readahead -r $size -w $size $share $mnt_dir

                    for e in 0 1 2
                    do
                        echo "$(date) - Using data lump $e"

                        # Transfer file
                        echo -n "$(date) - Transferring data: "
                        action=$(rsync --progress ${data[$e]} $mnt_dir)

                        # echo transfer info
                        action_thr=$(echo "$action" | grep xfr | awk '{print $4}')
                        action_time=$(echo "$action" | grep xfr | awk '{print $5}')
                        echo "$action_thr"

                        # Delete file
                        echo "$(date) - Remove transferred data ${e}"
                        rm -f $mnt_dir/$data

                        data_time[$e]=$action_time                        
                        data_trans[$e]=$action_thr
                    done

                    # Sync
                    echo "$(date) - Syncing"
                    sync_time=$({ time -p sync;} 2>&1 | grep real | awk '{print $2}')

                    # Unmount share and get duration
                    echo "$(date) - Unmounting share"
                    umount_time=$({ time -p doas umount $mnt_dir;} 2>&1 | grep real | awk '{print $2}')

                    data_time[$e]=$action_time                        
                    data_trans[$e]=$action_thr
                    data_sync[$e]=$sync_time
                    data_umount[$e]=$umount_time

                    # Save transfer info to log
                    # echo "$iothreads,$proto,$hard,$readahead,$size,$action_time,$action_thr,$sync_time,$umount_time" >> $log
                    echo "$iothreads,$proto,$hard,$readahead,$size,\
${data_time[0]},${data_time[1]},${data_time[2]},\
${data_trans[0]},${data_trans[1]},${data_trans[2]},\
$sync_time,$umount_time" >> $log

                done
            done
        done
    done
done
                 

Has been tested on OpenBSD 6.5-beta