-------Sequential Output-------- ---Sequential Input-- --Random-- Physical Raid Disk Attachment
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Device Path HBA Level Array type
Machine GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU /sec %CPU
jasket 36 100.3 99.5 877.4 97.4 403.5 35.1 101.5 98.8 471.7 27.4 604 2.2 LVM over /dev/sda2 /data1 LSI MegaRAID 2108 R6 16x WD "Red" SATA 4TB
mainstay 20 82.0 99.6 548.1 71.6 214.6 33.0 82.2 96.6 511.7 40.8 3000 5.2 Dell PERC6/i
mainstay 20 79.8 97.1 530.1 69.9 213.2 33.1 81.9 96.5 521.5 41.6 2973 5.5 Dell PERC6/i
catz_3 6 61.6 97.8 308.7 82.3 89.9 17.4 64.6 93.7 359.8 23.8 442 1.0 /dev/sde1 /3 DELL Perc5 R5 Dell MD1000 SAS
catz_1 6 59.8 95.7 228.4 61.8 94.3 18.7 62.8 91.4 340.6 21.9 488 1.0 /dev/sdc1 /1 DELL Perc5 R5 Dell MD1000 SAS
goapp20 12 66.5 98.9 99.7 19.2 131.3 24.2 61.6 91.8 472.2 33.5 1096 2.2 /dev/sdb1 /data1 3ware 9650SE R6 internal SATA
anchor2014 40 58.3 96.3 273.9 68.9 108.3 25.1 63.0 88.8 347.8 26.5 1394 2.2 LVM over /dev/sdb /users DELL/LSI Perc5 R5 new WD 6x 1TB enterprise SATA
beacon 20 58.8 94.5 253.3 32.2 109.6 18.5 48.0 73.1 263.0 19.0 2631 4.2 LVM over /dev/sda /srv DELL/LSI Perc6 R5 8x 2.5" 146GB 10krpm SAS
black 18 75.7 99.1 99.0 14.5 43.6 7.8 63.7 87.8 116.7 10.8 1749 3.6 LVM over hardware mirror half-stroked 15krpm U320 SCSI
anchor_d 20 70.8 92.5 129.3 33.6 42.1 8.2 50.9 58.3 78.4 6.3 527 1.1 /dev/sdd1 /data QLA SP202 R5 Sun 3511 FCAL
anchor 20 55.2 71.4 122.6 31.2 67.2 14.4 50.7 58.1 177.0 13.5 285 0.6 /dev/sdb1 /users DELL/LSI Perc5 R5 5x Seagate 750GB enterprise SATA (pre-2014)
caboodle 16 56.1 84.2 60.3 9.2 39.1 7.0 57.1 82.9 345.7 27.7 2175 4.2 LVM over /dev/sda2 /half-terabyte Perc5i R5 refurb Seagate/HP 146GB SAS
caboodle 16 51.0 75.9 53.1 8.1 37.4 6.8 53.0 77.0 308.9 25.2 2102 4.3 (new run Sept.29 2014)
levimd20 5 41.2 98.9 68.6 25.5 44.8 20.6 14.1 99.8 355.0 51.6 854 4.5 /dev/md20 /var/time_machine 3ware9550SE R6 SATA II
trammel 6 45.8 94.2 63.0 40.0 38.9 30.9 49.5 94.1 193.1 52.8 254 2.9 /dev/md5 /data/1 internal/sw R6 SATA/IDE
trammel 6 43.4 94.8 61.9 45.0 37.6 31.4 48.9 92.7 193.8 52.1 223 2.7 second run for comparison
jade_br 12 29.5 95.7 90.0 64.1 47.5 70.3 35.2 99.1 152.0 79.8 129 14.7 /dev/dsk/c3t40d1s0 /data/po/ruddick QLA2340 R5 Sun 3511(#2) FCAL
jade_ct 12 29.4 96.2 89.9 64.3 45.9 70.2 35.2 99.0 147.1 77.9 136 14.2 /dev/dsk/c3t40d0s0 /data/po/taggart QLA2340 R5 Sun 3511(#2) FCAL
jade_hr 12 29.4 96.3 70.0 49.6 9.5 11.2 35.5 99.5 142.3 73.9 136 12.8 /dev/dsk/c4t40d2s6 /data/po/ritchie1 QLA2340 R5 Sun 3511(#1) FCAL
jade_hr 12 29.9 97.0 71.5 49.9 8.6 10.2 35.3 98.9 146.5 75.3 132 12.3 /dev/dsk/c4t40d2s6 /data/po/ritchie1 QLA2340 R5 Sun 3511(#1) FCAL
jade_js1 12 28.4 92.4 47.3 33.5 8.5 10.1 35.4 99.3 123.6 64.8 161 10.9 /dev/dsk/c4t40d0s6 ..jade1_archive QLA2340 R5 Sun 3511(#1) FCAL
hold 6 35.3 85.8 33.9 10.4 24.1 9.0 44.8 98.6 516.1 92.0 947 5.6 LVM on /dev/md20 3Ware9550S-12/sw internal SATA
leviatha 5 32.7 85.9 35.8 17.7 27.0 14.5 30.0 69.2 107.2 33.0 326 2.7 LVM on /dev/md10 /test/one 3Ware 9500S-12/sw internal SATA -- tiny boost by using mke2fs -E stride=16
leviatha 5 32.4 85.1 35.0 16.8 26.5 14.2 28.2 65.2 101.4 31.0 312 2.7 LVM on /dev/md10 /backups/web_full_0626 3Ware 9500S-12/sw internal SATA
leviatha 5 36.5 96.9 45.0 22.2 24.1 11.3 28.2 64.8 71.7 18.7 321 2.5 Using 128kB + mke2fs -E stride=32
leviatha 5 36.8 96.6 45.0 22.2 24.1 11.3 28.4 65.8 72.0 19.1 333 3.0 Using 128kB chunk size
leviatha 5 31.4 84.2 29.9 14.7 7.8 5.4 34.5 86.9 61.9 17.7 330 2.9 Using 1024kB chunk size
leviatha 5 7.5 20.8 7.4 4.0 6.6 5.2 34.5 82.7 57.1 18.5 331 3.1 Using 4096kB chunk size
rhel-4.5 4.5 24.6 91.8 53.6 66.8 27.5 42.8 26.8 96.2 72.2 54.2 297 5.2 /dev/sdc1 /array/data1 AIC-7892A R5 AC&NC/Jetstor U160 SCSI
dqone 36 51.3 68.5 51.9 11.6 13.9 2.4 24.5 29.0 32.0 2.7 1420 2.9 /dev/sdc1 ..new-dqone-array LSI-53c1030 R5 IFT-7200 U320 SCSI
costello 8 disk failure during test -- aborted /dev/md6 /data QLA SP212/sw R6 Sun 3511 FCAL
catz-loc 12 44.7 72.7 44.6 20.6 19.0 8.2 28.6 57.4 46.8 9.9 204 0.8 /dev/md5 /state/partition1 internal/sw R1 SATA
taggart3 5 19.2 87.7 29.5 85.4 19.9 35.5 28.9 91.4 59.4 41.5 107 2.2 /dev/md4 /data/samba-share internal/sw R1 SATA
anchor_n 6 22.4 42.3 35.9 7.3 22.6 24.8 45.9 98.6 101.3 93.3 2630 46.8 remote NFS from thinclient
hold_nfs 6 22.3 42.1 35.2 8.1 23.0 30.1 45.3 99.1 103.0 93.7 3351 52.0 remote NFS from thinclient
acoustic 10 10.5 25.4 11.0 10.4 5.3 1.1 11.6 26.8 11.1 0.7 631 1.9 /dev/md4 /data/acoustic/kr-experiments internal/sw R5 SATA
acoustic 10 9.9 21.8 10.3 11.9 4.9 1.0 10.2 24.4 10.6 1.0 413 1.3
hartling 8 10.0 23.6 50.0 40.3 8.4 3.9 8.0 24.5 11.3 2.1 112 0.6 /dev/sde1 /data/ext..lun1 Emulex-LP9000 R? IFT-7250 FCAL
kt-nfs-64 0 9.8 28.0 9.8 7.7 9.5 6.1 44.7 99.9 304.7 100 2010 18.6 remote nfs /data/acoustic 100Mbit-hme0 R5 NFS->SATA/IDE
kt-nfs-512 0 10.0 25.7 10.1 7.3 9.8 6.0 46.5 100 291.0 100 1071 37.5 remote nfs /data/acoustic 100Mbit-hme0 R5 NFS->SATA/IDE
kt-nfs-2GB 2 10.2 25.2 10.1 7.3 3.4 2.5 46.8 100 291.2 100 557 51.8 remote nfs /data/model 100Mbit-hme0 R5 NFS->SATA/IDE
innukshu 5 11.1 55.3 18.7 24.5 1.2 2.6 18.9 92.3 49.6 50.9 47 8.0 /dev/dsk/c1t1d0s1 /data/po/acoustic QLA2100 R5 Sun T3 FCAL
kt-4800 28 6.9 19.8 7.0 6.8 1.1 1.6 34.9 85.7 60.8 37.6 106 4.9 /dev/md/dsk/d54 /data/acoustic QLA2300? R5 Sun 6120(#1) FCAL
Most 'per char' results are CPU bound! 99% CPU usage indicates this. Look at the 'per block' results instead.
Discrepancy on poor rewrite performance between subsequent Jade runs suggests retesting. It seems the array on controller#4 -- 3511(#1) may have a problem compared to the array on controller#3 -- 3511(#2). Or perhaps it was simply busy during the test (sharing NFS to dqone-four for example).
Tests were intended to be peformed while systems were idle, but unexpected remote NFS access may influence the results. In particular 'dqone' is always busy! (load avg. 6 during test). Size of test file (in GB, second column) is always 4 GB greater than total RAM on each test system. This is to ensure "real" disk I/O performance is tested, rather than merely access to RAM buffers.
Catz & Anchor performance is outstanding! Anchor: Dell PERC5/i controller with internal backplane, and Qlogic FCAL controller to Sun StorEdge 3511 array. Catz: Dell PERC5/e controllers with Dell MD1000 arrays.
Note:
'sw' in the HBA column indicated software raid; others have hardware raid controllers
- R0 = RAID 0 -- striping across disks (no redundancy) -- size = numdisks * size of disks
- R1 = RAID 1 -- mirroring -- size = (numdisks * size of disks) / 2
- R5 = RAID 5 -- striping across disks plus parity calculation -- size = (numdisks - 1) * size of disks
- R6 = RAID 6 -- striping across disks plus 2x parity calcuation (double redundancy) -- size = (numdisks - 2) * size of disks
- SAS = Serial attached SCSI. Topology: star with daisy chaining
- SATA/IDE = internal "generic PC style" SATA or IDE ribbon cables. Topology: star
- FCAL = fibre channel arbitrated loop -- SCSI over optical fibre network. Topology: mesh