The RAIDzilla project
History
When I started the media collection project, I was using a NETGEAR ND520 drive,
but it turned out to be incredibly slow and rather unreliable. I was faced with
a "buy-or-build" decision for a more reliable server with higher capacity. The
lack of a good 1 RU (1.75") OEM chassis that could hold 4 drives turned me
toward pre-built solutions. I selected the Snap Server 4100 (Snap has gone
through many incarnations, first as Meridian Data, then a part of Quantum,
then an independent company again - maybe Quantum was upset that they wouldn't
use Quantum-branded drives in their products, preferring IBM drives, and now
they are owned by Adaptec). The 4100 is a 1 RU chassis which came with 4 drives
of varying sizes - the smallest model I've seen with "factory" drives had 30GB
drives, and the largest was 120GB drives. They can be used in a variety of
RAID/JBOD configurations. In a RAID 5 configuration, usable capacity is about
3/4 of the total drive capacity, minus about 1GB for overhead.
Since then, I've upgraded the 4100's to larger and larger drives, with the most
recent incarnation holding 4 120GB drives, for about 350GB of usable storage.
These units are quite common on the used market, particularly with smaller
capacity drives installed. The Dell 705N is an OEM version of the 4100 which
can also be upgraded.
Unfortunately, Snap decided to not support 48-bit addressing on the 4100, which
means that drives larger than approximately 137GB can't be used at their full
capacity. Despite a number of negotiations to have this feature implemented on
the 4100, Snap finally decided not to do so. I can't really blame them - the
4100 is a nearly-discontinued product, and adding a feature to let customers
put larger drives in "on the cheap" really isn't in their best interest.
This meant that I would either need to add more 4100's to my already large
collection (I had 13 of the 4100 units now) or I would need to change to newer
hardware which would support larger drives.
I decided to build my own servers this time, for a number of reasons:
- More Snap 4100 units would just take up more space, and I'd need to manually
balance usage between the servers.
- Snap 4100 performance isn't that great - about 35Mbit/sec when reading
natively (Unix NFS) and about 12Mbit/sec when reading using Windows file
sharing.
- Good OEM cases were now available in a variety of sizes.
- By building the system myself, I'd be aware of any hardware or software
limitations and could address them myself.
- I could save money by building my own.
RAIDzilla
When I started planning for this project, the Western Digital WD2500 was the
largest single drive available, at 250GB per drive. Western Digital called
these drives
"Drivezilla", so calling a chassis with 24 of them "RAIDzilla" was an
obvious choice.
However, when I actually started pricing parts for the system, Seagate had just
announced a
400GB drive with a 5-year warranty. I changed my plans and decided
to build a server with 16 of the Seagate 400GB drives instead of 24 of the
Western Digital 250GB ones.
I also planned on using FreeBSD as the operating system for the server, but I
needed features which were only available in the 5.x release family, which was
in testing at the time. Between these delays, the complete system didn't get
integrated (that's computer geek for "put together") until early January, 2005.
Fortunately, pieces of it were up and running for 6 months or so by then, so I
had a lot of experience with what would work and what wouldn't by the time it
was time to build the first production server.
Hardware
Once I made the decision to use a 16-bay chassis, I selected the Ci Design
SR316 as
the base of the system. It holds up to 16 hot-swap Serial ATA (SATA) drives,
a floppy drive, CD-ROM, a triple-redundant power supply, and a variety of
motherboards.
For the motherboard, I selected the Tyan
Thunder i7501
Pro with the optional SCSI controller. I had used Tyan motherboards in the
past and was mostly satisfied with them (anyone remember the Tomcat COAST
fiasco?). It supports 2 Intel
Xeon processors. I selected the 3.06GHz / 533MHz ones as the "sweet spot"
on the price/performance curve. I used 2 processors so I could feel free to
run local programs on the server without impacting its file-serving
performance. I often do CPU-intensive tasks such as converting CD images to
MP3s, checksumming files, etc. The motherboard has 3 on-board Ethernet ports,
one which supports 10/100 Mbit and two that support 10/100/1000 Mbit. I felt
that Gigabit Ethernet performance was necessary for this project, and this
board had two of them already installed, which was ideal.
For memory, I use Kingston products exclusively. I selected two of the
KVR266X72RC25L/2G modules as they were on Tyan's approved memory list, and
4GB would give me lots of memory for disk buffers. While the motherboard
supports up to 12GB of memory (6 of these 2GB modules), operating system
support for more than 4GB is not common at this time.
One of the most important items for a file server is the disk controller. I
picked the 3Ware/AMCC
9500S-8MI
controllers. Each controller supports 8 drives and connects to the chassis
backplane via a Multilane Interconnect (MI) cable. This cable takes the place
of 4 normal SATA cables, reducing the total disk cable count from 16 to 4.
Finally, I added internal SCSI cables and external terminators from
CS Electronics. The folks at
CS are wonderful, building and shipping whatever custom cables I've needed in
only a day or two.
The 16 drives in the system are split into 2 independent groups of 8, with
one group on one controller and the other group on the second controller.
Within the groups of 8, 7 drives are used as a RAID 5 set with the 8th drive
being a "hot spare" in case any of the other 7 drives fail.
Software
As I mentioned above, I'm using FreeBSD 6.x as the operating system. I also
use Samba as my CIFS server software, and rdiff-backup for synchronizing my
offsite backup RAIDzillas to the local units. There is a large variety of
other software installed as well - one of the nice things about the RAIDzilla
as opposed to vendor's packaged NAS box is that I can run a huge amount of
software on them locally without needing to build a customized NAS OS (if
the sources are even available) and cross-compiling the applications.
What I'd do differently this time (2 years later)
If I were building these from scratch now, the major change I'd make would be
to use a single 16-channel RAID card, so I'd have one parity disk and one spare
drive in the chassis, instead of the two of each I have now.
I'd use more modern hardware, such as a PCI Express motherboard and RAID card,
and go with larger-capacity drives (750GB or 1TB).
Since support for systems with > 4GB of memory is now common, I might also
experiment with a huge memory size - it should be possible to easily run 16GB
or even 32GB on server-class motherboards today.
Enough talk, let's see the pictures!
Each of the images is clickable to display a higher-resolution version.
![[Front view of the server]](6F5S4390-s.jpg)
This is the front view of the server, which is the only part you'll see
when it is installed in a rack. Most of the front is taken up by the 16
hot-swap drive bays, with a little space left on the top for the floppy,
control panel, and CD-ROM.
![[A case of 400GB drives]](6F5S4383-s.jpg)
Here is a freshly-opened carton of 20 400GB Seagate disk drives. It is
important that all drives be identical, down to the revision of the drive and
firmware, for the best RAID performance. The best way to ensure that is to
purchase a case lot from the manufacturer.
![[Drive cooling fans]](6F5S4389-s.jpg)
In this view of the top of the chassis (with the cover removed) you can see the
three large cooling fans that pull air in across the disk drives and then
exhaust it out the back of the case.
![[The motherboard area]](6F5S4388-s.jpg)
This is the back part of the case, where you can see the motherboard with two
CPUs mounted (under the black plastic covers), and the triple redundant power
supply on the right.
![[Close-up of expansion card area]](6F5S4391-s.jpg)
Here you can see a close-up picture of the expansion card area. The two 3Ware
controllers are prominent in the center of the picture, with the disk cables
exiting in a coil to the right. The wavy flat red and gray cables on the
controllers connect the controllers to the 16 individual drive activity lights
visible from the front. At the bottom of the picture are the custom orange and
white Ultra-320 SCSI cables which connect bulkheads on the back of the chassis
to the two internal SCSI channels. These are used to attach devices such as DLT
tape drives.
![[Memory close-up]](6F5S4392-s.jpg)
This is a close-up shot of the 2 2GB memory SIMMs. For someone who has been in
the computer industry as long as I have, this type of density is amazing. The
first memory board I purchased new (in 1978) was 16 inches square and a half
inch thick, held a whopping 8Kbytes of memory, and cost $22,000. And that was
considered a major accomplishment.
For the computer geeks out there, this is a "dmesg" output of the system
booting up, listing the installed hardware:
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.2-STABLE #0: Thu Jun 11 20:17:49 EDT 2009
terry@rz1.glaver.org:/usr/obj/usr/src/sys/RAIDZILLA
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3065.81-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4400<CNXT-ID,xTPR>
Logical CPUs per core: 2
real memory = 4160684032 (3967 MB)
avail memory = 4073725952 (3885 MB)
ACPI APIC Table: <A M I OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP/HT): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP/HT): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
ioapic3 <Version 2.0> irqs 72-95 on motherboard
ioapic4 <Version 2.0> irqs 96-119 on motherboard
kbd1 at kbdmux0
acpi0: <A M I OEMRSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, f7f00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci5: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci5
pci7: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xd880-0xd8bf mem 0xfe9c0000-0xfe9dffff irq 48 at device 1.0 on pci7
em0: [FILTER]
em0: Ethernet address: 00:e0:81:28:94:d6
em1: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xdc00-0xdc3f mem 0xfe9e0000-0xfe9fffff irq 49 at device 1.1 on pci7
em1: [FILTER]
em1: Ethernet address: 00:e0:81:28:94:d7
pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci5
pci6: <ACPI PCI bus> on pcib3
3ware device driver for 9000 series storage controllers, version: 3.70.05.001
twa0: <3ware 9000 series Storage Controller> port 0xc800-0xc8ff mem 0xfe8ffc00-0xfe8ffcff,0xfb800000-0xfbffffff irq 24 at device 3.0 on pci6
twa0: [ITHREAD]
twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue:
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9500S-8MI, 8 ports, Firmware FE9X 2.08.00.009, BIOS BE9X 2.03.01.052
pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci2
pci4: <ACPI PCI bus> on pcib5
ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0xb000-0xb0ff,0xa800-0xa8ff mem 0xfe6fc000-0xfe6fdfff irq 98 at device 2.0 on pci4
ahd0: [ITHREAD]
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0xb800-0xb8ff,0xb400-0xb4ff mem 0xfe6fe000-0xfe6fffff irq 99 at device 2.1 on pci4
ahd1: [ITHREAD]
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci2
pci3: <ACPI PCI bus> on pcib6
twa1: <3ware 9000 series Storage Controller> port 0x9800-0x98ff mem 0xfe3ffc00-0xfe3ffcff,0xfa000000-0xfa7fffff irq 72 at device 3.0 on pci3
twa1: [ITHREAD]
twa1: INFO: (0x04: 0x0053): Battery capacity test is overdue:
twa1: INFO: (0x15: 0x1300): Controller details:: Model 9500S-8MI, 8 ports, Firmware FE9X 2.08.00.009, BIOS BE9X 2.03.01.052
uhci0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> port 0xec00-0xec1f irq 16 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib7
fxp0: <Intel 82551 Pro/100 Ethernet> port 0x8c00-0x8c3f mem 0xfe2fe000-0xfe2fefff,0xfe2a0000-0xfe2bffff irq 19 at device 1.0 on pci1
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:e0:81:28:93:fb
fxp0: [ITHREAD]
vgapci0: <VGA-compatible display> port 0x8800-0x88ff mem 0xfd000000-0xfdffffff,0xfe2ff000-0xfe2fffff irq 18 at device 2.0 on pci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
ichsmb0: <Intel 82801CA (ICH3) SMBus controller> port 0x540-0x55f at device 31.3 on pci0
ichsmb0: [GIANT-LOCKED]
ichsmb0: [ITHREAD]
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse Explorer, device ID 4
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff,0xcb000-0xcb7ff,0xcb800-0xccfff,0xcd000-0xce7ff pnpid ORM0000 on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
Waiting 5 seconds for SCSI devices to settle
acd0: CDROM <CD-224E/1.9A> at ata1-master UDMA33
sa0 at ahd0 bus 0 target 5 lun 0
sa0: <QUANTUM SDLT600 3535> Removable Sequential Access SCSI-4 device
sa0: 160.000MB/s transfers (80.000MHz DT, offset 120, 16bit)
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9500S-8MI DISK 2.08> Fixed Direct Access SCSI-3 device
da0: 2288754MB (4687368192 512 byte sectors: 255H 63S/T 291775C)
da1 at twa1 bus 0 target 0 lun 0
da1: <AMCC 9500S-8MI DISK 2.08> Fixed Direct Access SCSI-3 device
da1: 2288754MB (4687368192 512 byte sectors: 255H 63S/T 291775C)
ch0 at ahd0 bus 0 target 5 lun 1
ch0: <QUANTUM UHDL 0061> Removable Changer SCSI-2 device
ch0: 160.000MB/s transfers (80.000MHz DT, offset 120, 16bit)
ch0: 16 slots, 1 drive, 1 picker, 0 portals
Trying to mount root from ufs:/dev/da0s1a
This shows the active filesystems. The /backups* filesystems are a mirrored
copy of the primary filesystems to be used in the event one of the RAID arrays
is unavailable, allowing the system to be booted for recovery purposes.
The actual filesystems provided to the network are two 2TB filesystems, named
/data0 and /data1. Additionally, 2 120GB filesystems named /work0 and /work1
are provided for scratch space for network users.
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/da0s1a 507630 51894 415126 11% /
devfs 1 1 0 100% /dev
/dev/da0s1d 16244334 37408 14907380 0% /var
/dev/da0s1e 16244334 6 14944782 0% /var/crash
/dev/da0s1f 16244334 1607096 13337692 11% /usr
/dev/da0s1g 8122126 62 7472294 0% /tmp
/dev/da0s1h 8121252 26 7471526 0% /sysprog
/dev/da0s2h 120462010 4 110825046 0% /work0
/dev/da0s3h 2079915614 859342082 1054180284 45% /data0
/dev/da1s1a 507630 249998 217022 54% /backups_root
/dev/da1s1d 16244334 38762 14906026 0% /backups_var
/dev/da1s1e 16244334 62 14944726 0% /backups_var_crash
/dev/da1s1f 16244334 1686090 13258698 11% /backups_usr
/dev/da1s1g 8122126 126 7472230 0% /backups_tmp
/dev/da1s1h 8121252 86 7471466 0% /backups_sysprog
/dev/da1s2h 120462010 4 110825046 0% /work1
/dev/da1s3h 2079915614 1474939512 438582854 77% /data1
![[Windows view of one of the filesystems]](rz1.gif)
Here is a windows view of one of the server's filesystems showing its 1.93TB
capacity. Please note that this system is firewalled from the Internet and
don't bother trying to come visit. It will only bother me and cause me to make
unhappy noises at your ISP.
Here are some quick benchmarks I ran. Note that this is only testing one of the
two RAID channels in the system - total throughput would be approximately
double this (subject to any CPU limitations). I ran 2 benchmarks, one with 2GB
of data and one with 16GB of data. The read performance on the 2GB test is
dominated by cache effects, as 2GB fits comfortably in the filesystem cache.
I did some Ethernet performance testing between a pair of RAIDzillas and was
able to achieve > 650Mbit/sec using FTP (get /dev/zero /dev/null). Performance
of actual file serving is limited by the speed of the disk subsystem, of
course.
I haven't done a lot of performance tuning yet (mostly because this level of
throughput is more than adequate for my applications).
Iozone www.smallnetbuilder.com test (local):
Iozone: Performance Test of File I/O
Version $Revision: 3.267 $
Compiled for 32 bit mode.
Build: freebsd
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg,
Erik Habbinga, Kris Strecker, Walter Wong.
Run began: Mon Apr 16 23:45:42 2007
Auto Mode
CPU utilization Resolution = 0.000 seconds.
CPU utilization Excel chart enabled
Using Maximum Record Size 64 KB
Using minimum file size of 4 kilobytes.
Using maximum file size of 1048576 kilobytes.
Cross over of record size disabled.
Command line used: iozone -Rab raidzilla-local.wks -i 0 -i 1 -+u -f /work2/temp.tmp -q 64k -n 4k -g 1G -z
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
KB reclen write rewrite read reread
4 4 97652 401314 582427 806376
8 4 154056 467294 660446 887549
8 8 222501 623622 992567 1612752
16 4 168697 550950 640316 1004538
16 8 262343 764288 1068515 1605256
16 16 348064 1068515 1347413 2329708
32 4 213448 627755 799840 1033989
32 8 320477 914254 1139314 1681806
32 16 463130 1233554 1528576 2641785
32 32 463130 1331449 1601534 2930153
64 4 217743 645579 799377 1049372
64 8 374488 969761 1143223 1828508
64 16 503814 1363961 1452528 2662899
64 32 503814 1336792 1599680 3363612
64 64 557144 1460430 1780008 3541098
128 4 224556 539674 814915 1040839
128 8 325611 735635 1142750 1749872
128 16 413039 947186 1504659 2660335
128 32 460514 983630 1526043 2969325
128 64 463696 942199 1827299 3036502
256 4 201422 490560 812329 955451
256 8 262284 621448 1080434 1523457
256 16 367241 782721 1570244 2131261
256 32 349206 808050 1729594 2325094
256 64 348300 850956 1817419 2210228
512 4 238376 432019 800163 826016
512 8 301324 566946 1096322 929322
512 16 292222 680918 1454140 1625889
512 32 357985 729244 1594501 1809465
512 64 369694 730236 1488408 1924603
1024 4 273428 405384 815943 821876
1024 8 350214 515366 1192117 1227903
1024 16 271681 607680 1544504 1609925
1024 32 421579 616490 1654579 1786736
1024 64 416106 618354 1659694 1906496
2048 4 29067 397207 816937 829480
2048 8 244450 494817 1197736 1221235
2048 16 404654 223191 1570714 1622945
2048 32 406685 270040 1728076 1796754
2048 64 420029 355120 1773019 1911931
4096 4 53850 189663 824338 825962
4096 8 63517 139064 1216893 1220178
4096 16 59556 158600 1123101 1477608
4096 32 46457 116067 1735549 1799729
4096 64 49468 101476 1751831 1907643
8192 4 74959 98131 827650 833634
8192 8 68894 116001 1216885 1218871
8192 16 65117 120429 1040264 1561607
8192 32 58750 93341 1591922 1787074
8192 64 68920 109116 1861428 1912294
16384 4 83068 77595 829366 826553
16384 8 87731 87551 1224622 1224601
16384 16 55223 101425 1610230 1612118
16384 32 80081 69923 1783170 1784003
16384 64 100940 66056 1887078 1890400
32768 4 91932 88726 829635 829340
32768 8 89866 82737 1227026 1220098
32768 16 106548 79654 1201874 1607852
32768 32 92437 78224 1783374 1784555
32768 64 89668 93109 1896290 1890811
65536 4 95044 93261 829675 825194
65536 8 95452 102276 1228390 1217212
65536 16 94824 98805 1613466 1597655
65536 32 93651 101317 1791683 1771819
65536 64 101596 96864 1903958 1867024
131072 4 96234 98743 541091 541042
131072 8 94085 95320 685534 686549
131072 16 95667 99172 793086 790381
131072 32 99015 95541 835438 835694
131072 64 98118 97895 854172 856680
262144 4 98248 97737 539225 538871
262144 8 96171 95733 682240 683040
262144 16 95478 97660 788808 787876
262144 32 96520 98128 841807 833786
262144 64 98757 98122 854741 855835
524288 4 98150 99017 540235 539905
524288 8 97546 98534 682410 684669
524288 16 97620 98365 787614 786192
524288 32 98228 98370 835702 839428
524288 64 96632 98569 856034 854879
1048576 4 98640 98607 540553 540760
1048576 8 96916 98245 689887 684856
1048576 16 97068 98471 786585 792558
1048576 32 97068 98276 834014 833814
1048576 64 96968 98436 860658 856918
iozone test complete.
Iozone www.smallnetbuilder.com test (client PC via CIFS):
Iozone: Performance Test of File I/O
Version $Revision: 3.281 $
Compiled for 32 bit mode.
Build: Windows
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong.
Run began: Tue Apr 17 03:07:04 2007
Auto Mode
CPU utilization Resolution = -0.000 seconds.
CPU utilization Excel chart enabled
Using Maximum Record Size 64 KB
Using minimum file size of 4 kilobytes.
Using maximum file size of 1048576 kilobytes.
Cross over of record size disabled.
Command line used: iozone -Rab raidzilla.wks -i 0 -i 1 -+u -f u:\temp.tmp -q 64k -n 4k -g 1G -z
Output is in Kbytes/sec
Time Resolution = 0.000003 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
KB reclen write rewrite read reread
4 4 5419 90621 234724 397727
8 4 15588 135717 16854 371900
8 8 16410 172508 17439 636363
16 4 27429 192837 14625 430621
16 8 28679 253419 17106 690032
16 16 21994 316423 21442 1060605
32 4 54623 234244 9433 433884
32 8 58771 357954 11907 707070
32 16 42065 450966 20732 1013676
32 32 44622 540308 32839 1613315
64 4 69232 294083 14440 444836
64 8 102869 450966 18513 743801
64 16 80410 610909 20123 1151210
64 32 84008 758579 29288 1613315
64 64 92078 884520 37979 2101751
128 4 147515 319068 24064 446570
128 8 169383 518305 27375 753588
128 16 189487 736626 28838 1159953
128 32 170517 956538 32967 1619016
128 64 225040 1117516 36896 2101751
256 4 191148 327272 4406 440983
256 8 244233 561153 30751 741394
256 16 313072 839160 3195 1078074
256 32 351097 1114797 37258 1642228
256 64 389114 1343641 2323 1829068
512 4 220650 341480 31165 431229
512 8 358935 574162 16232 721262
512 16 456583 864493 10749 1076175
512 32 539195 1172570 8228 1456857
512 64 572369 1331923 6604 1748785
1024 4 284938 341831 12369 420929
1024 8 424930 559953 12582 688218
1024 16 593788 869208 21272 992272
1024 32 728863 1148325 10271 1379027
1024 64 844380 1441389 9415 1678321
2048 4 18526 325890 35202 406009
2048 8 41606 544078 33715 591822
2048 16 60872 813190 35307 946045
2048 32 115389 1078391 38746 1244636
2048 64 218617 1275606 38163 1475031
4096 4 14141 314267 35381 389083
4096 8 24067 509497 35853 621579
4096 16 44330 754946 36014 960800
4096 32 41193 999442 38705 1267555
4096 64 160770 1179076 38233 1461213
8192 4 11191 315915 30445 402537
8192 8 14811 509550 35863 471964
8192 16 29871 729171 35787 973592
8192 32 71658 967840 38891 1296645
8192 64 129337 1143444 33556 1564177
16384 4 10154 282599 35631 401152
16384 8 17069 506313 32458 651853
16384 16 30949 682937 32907 967553
16384 32 61081 951973 35165 1239611
16384 64 134512 1103553 37985 1546523
32768 4 7701 110972 35718 400138
32768 8 14488 126219 34374 644004
32768 16 14499 104241 34058 958437
32768 32 7426 22660 36891 1275703
32768 64 23131 41592 38038 481228
65536 4 8579 55692 34082 394228
65536 8 11998 53168 34091 34996
65536 16 14099 62417 34139 960926
65536 32 3833 11193 37036 1279027
65536 64 7412 13894 37190 1542506
131072 4 8826 44509 32602 342445
131072 8 11719 43346 34101 648674
131072 16 16058 46844 34443 935007
131072 32 8639 12777 32417 1243162
131072 64 4873 9351 35683 979137
262144 4 8420 41827 34288 394543
262144 8 12436 35846 34192 643853
262144 16 16197 44707 34241 952680
262144 32 4320 7991 36417 1239832
262144 64 5295 10355 35667 1507513
524288 4 9247 34684 34199 375231
524288 8 12886 32376 33806 599873
524288 16 16414 39454 31048 945440
524288 32 5126 9423 36399 1121913
524288 64 4852 11769 35386 1524367
1048576 4 10991 25622 33738 379241
1048576 8 12654 33492 33896 614082
1048576 16 13314 24679 34186 901599
1048576 32 5725 10757 36275 1194654
1048576 64 5252 13363 35132 1421791
iozone test complete.
bonnie++ 2GB test (local):
Script started on Thu Apr 19 01:38:53 2007
rz2:/tmp# time bonnie++ -d /work2 -s 2g -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
rz2.glaver.org 2G 201 99 93173 34 87933 38 337 99 550513 99 1955 82
Latency 41631us 150ms 224ms 204ms 1497us 182ms
Version 1.93c ------Sequential Create------ --------Random Create--------
rz2.glaver.org -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 4655 20 +++++ +++ +++++ +++ 4885 21 +++++ +++ +++++ +++
Latency 803ms 293us 372us 786ms 122us 15702us
1.93c,1.93c,rz2.glaver.org,1,1176993704,2G,,201,99,93173,34,87933,38,337,99,550513,99,1955,82,16,,,,,4655,20,+++++,+++,+++++,+++,4885,21,+++++,+++,+++++,+++,41631us,150ms,224ms,204ms,1497us,182ms,803ms,293us,372us,786ms,122us,15702us
3.976u 44.471s 1:30.25 53.6% 47+318k 5+35421io 2pf+0w
bonnie++ 16GB test (local):
rz2:/tmp# time bonnie++ -d /work2 -s 16g -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
rz2.glaver.org 16G 201 99 93708 37 16021 7 343 99 68782 17 305.0 17
Latency 41769us 564ms 747ms 36859us 175ms 82297us
Version 1.93c ------Sequential Create------ --------Random Create--------
rz2.glaver.org -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 4538 20 +++++ +++ +++++ +++ 5622 25 +++++ +++ 25383 51
Latency 970ms 74us 65us 669ms 58us 310ms
1.93c,1.93c,rz2.glaver.org,1,1176993337,16G,,201,99,93708,37,16021,7,343,99,68782,17,305.0,17,16,,,,,4538,20,+++++,+++,+++++,+++,5622,25,+++++,+++,25383,51,41769us,564ms,747ms,36859us,175ms,82297us,970ms,74us,65us,669ms,58us,310ms
11.679u 202.225s 25:32.56 13.9% 47+290k 271672+149224io 1pf+0w
![[Front view of the server]](6F5S4390-l.jpg)
![[A case of 400GB drives]](6F5S4383-l.jpg)
![[Drive cooling fans]](6F5S4389-l.jpg)
![[The motherboard area]](6F5S4388-l.jpg)
![[Close-up of expansion card area]](6F5S4391-l.jpg)
![[Memory close-up]](6F5S4392-l.jpg)
