With dd. And normally when I do this I drop the caches with "echo 3 > /proc/sys/vm/drop_caches" but this doesn't work with ZFSOnLinux as it has a separate ARC cache that doesn't get cleared by that command (and I don't believe it's possible to clear it, other than by exporting and reimporting the zpool.)
So I read a 10GB chunk of a file that I know is not cached:
$ dd if=/some/large/file of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 11.3378 s, 947 MB/s
The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.
So I read a 10GB chunk of a file that I know is not cached:
The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.