環境さんぷる

構成や設定とかは前の投稿を見てもらうとして、
シングルパスアクセスでだいたいこれぐらい。
物理メモリ16GB搭載なので何もしなくても全部キャッシュに載ります。

ZFS(raidz2)とiSCSIとGbEのオーバーヘッドの参考に。
4KBの性能についてはひどいので、近いうちに修正版を出すかも？

この性能を出しているときのCPU使用率は1コアだけ85%ぐらい、
残りの5コアは1～5%ぐらい。（2.9GHz/6コア）
ここから察するに、3.2GHzの8コア以上あれば10Gbpsが処理できそう。

Athlon II X2 250e の性能は悪くないけどLANポート3個以上を
対応するには無理がある。

↓デュアルコアだと2Gbpsぐらいしかカバーできない。（使用率ほぼ１００％）

aria# dd if=/dev/zero of=/tmp/test20G.dat bs=1m count=20480
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 93.255702 secs (230279072 bytes/sec)

↓ヘキサコアだと4Gbpsぐらいカバーできそう。（使用率６０～８０％ぐらい）
しかもHDDの台数のせいで頭打ちな感じが…。＠２～３台増やしたほうが（ｒｙ

[root@aria /tmp]# dd if=/dev/zero of=./test3.dat bs=1m count=40960
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 99.046374 secs (433631956 bytes/sec)

設定はあまり調整していないので仮設定です。
で、このマシンを収容するのはPowerConnect 5324になってる。
LACPにすると頭打ちになるのはなんでかなぁ。
クラスC足りなくなったので移行中でめんどい。
そもそも足りない理由がVMの創成だからVMセグメントを分ければ（ｒｙ

～現時点の設定～
/boot/loader.conf:

autoboot_delay="3"
ahci_load="YES"
zfs_load="YES"
kern.maxfiles="32768"
kern.cam.boot_delay=8000
vfs.root.mountfrom="zfs:tank/root-82"
# 16GB/hexa-core/raidz2
vfs.zfs.arc_min="13G"
vfs.zfs.txg.synctime=5
vfs.zfs.txg.timeout=20
vfs.zfs.vdev.max_pending=8
vfs.zfs.zfetch.block_cap=512
vfs.zfs.zfetch.max_streams=16

/etc/sysctl.conf:

kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_inc=524288
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sendspace=524288
net.inet.tcp.recvspace=524288
net.inet.tcp.delayed_ack=0
net.inet.tcp.path_mtu_discovery=0
net.inet.tcp.delacktime=10
net.inet.tcp.inflight.enable=0
net.inet.tcp.hostcache.expire=1

/etc/rc.conf:

defaultrouter="172.18.0.1"
hostname="aria.ad.peach.ne.jp"
# IPv6
ipv6_enable="YES"
ipv6_gateway_enable="NO"
# List of cloned network interfaces
cloned_interfaces="lagg0"
# re0 for SMB
ipv4_addrs_re0="172.18.0.26/16"
#ipv6_prefix_re0="2001:03e0:06cf:18"
# lagg0 for iSCSI/primary
ifconfig_igb0="up"
ifconfig_igb1="up"
ifconfig_lagg0_name="is0"
#ifconfig_is0="laggproto lacp laggport igb0 laggport igb1"
ifconfig_is0="laggproto loadbalance laggport igb0 laggport igb1"
ipv4_addrs_is0="172.16.0.26/16"
ipv6_prefix_is0="2001:03e0:06cf:16"
# em0 for iSCSI/secondary
ifconfig_em0_name="is1"
ipv4_addrs_is1="172.17.0.26/16"
ipv6_prefix_is1="2001:03e0:06cf:17"
# em1 for NFS/Backup
ipv4_addrs_em1="172.19.0.26/16"
ipv6_prefix_em1="2001:03e0:06cf:19"
#ipv4_addrs_em2="172.19.0.27/16"
#ipv6_prefix_em2="2001:03e0:06cf:19"

sshd_enable="YES"
zfs_enable="YES"
powerd_enable="YES"
samba_enable="YES"
istgt_enable="YES"

※12/26 IPアドレス修正、powerd有効化とか
起動情報はこれ。dmesg.boot.gz
※1/1 パラメタ調整、旧IPv4削除、IPv6追加。

保管用スペースもそんなにあるわけではないので、新規に作成する事にしました。
現在 500GB x6 (raidz2/2TB), 2TB x2 (mirror/2TB)で運用中の所に 2TB x7 (raidz2/10TB)を新規で導入します。
今回は FreeBSD 8.2で運用開始して、後に8.3にアップグレードする予定です。
なお、現時点で8.2はまだリリースされていませんのでBETA1からインストールしてcsupで最新にします。

利用したパーツは以下の通り、スペアパーツとかからも選択したので若干バランスが良くないですがそのうち変えるかも。
ZFSで利用するの前提なのでメモリはマザーの最大搭載である16GBまで載せました。
※12/17 更新：パーツかえました。
※12/25 更新：CPU使用率が100%に張り付くのでCPUかえました。

ケース JMAX JX-FM500B (MicroATX仕様/リムーバブル7個)
マザー MSI 880GMA-E45 (SATA3.0×6/ATA133x1/USB3.0×2/USB2.0×12/LANx1/DDR3x4)
CPU AMD Athlon II X2 250e (3.0GHz/Dual-core/TDP45W)
CPU AMD Phenom II X6 1065T(2.9GHz/Hexa-core/TDP95W)
メモリ SanMax SMD-8G68HP-13H-D (DDR3-1333/4GB*2枚組) x2
電源 SilverStone Strider plus SST-ST75F-P (750W/80PLUS Silver)
SATAカード 玄人志向 SATA3I2-PCIe (SATA3.0×2)
SATAカード ASRock SATA3 Card (SATA3.0×2/eSATA共有x1)
LANカード Intel Gigabit ET Dual Port Server Adapter (82576)
LANカード Intel Gigabit CT Desktop Adapter (82574L)
LANカード PLANEX GN-1200TW2 (Realtek RTL8169SC)
LANカード Intel PRO/1000 MT Dual Port Server Adapter (82546)
USBメモリ Silicon Power Ultima II I-Series 4GB
HDD Seagate Barracuda LP ST32000542AS (2TB/32MB) x7

OS FreeBSD 8.2 (USBメモリ起動 + ZFS root)

起動時の消費電力は 205W ぐらい、OSが起動して安定すると105Wぐらい。
HDDなしでは69WぐらいなのでHDD一個あたり5Wぐらい。
powerdをいれると10Wぐらい消費電力落ちて95Wぐらいです。
※デュアルポートを刺したら+2Wぐらいになりました。
※X6のアイドル時は115Wぐらい。

HDD単体性能はこれぐらい。

aria# dd if=/dev/ada0 of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 9.077099 secs (115518845 bytes/sec)

ZFS raidz2(RAID6相当)で20GBのゼロファイルを読書きするとこれぐらい。

aria# dd if=/dev/zero of=/tmp/test20G.dat bs=1m count=20480
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 93.255702 secs (230279072 bytes/sec)

aria# dd if=/tmp/test20G.dat of=/dev/null bs=1m count=20480
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 40.978181 secs (524055385 bytes/sec)

見ての通り524MB/sの読み込みと230MB/sの書き込みができる。
先代の装置がR:260MB/sとW:110MB/sだったので約2倍の性能がある事に。
消費電力も30W以上も下がってるし、いい時代になったものだ。

6コアを使うと素敵な速度に。CPU使用率は60~80%ぐらい。
R:530MB/sとW:430MB/sぐらいで書き込み性能比二倍ぐらい。
価格三倍ぐらい。

[root@aria /tmp]# dd if=./test2.dat of=/dev/null bs=1m
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 81.029247 secs (530051488 bytes/sec)
[root@aria /tmp]# dd if=/dev/zero of=./test3.dat bs=1m count=40960
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 99.046374 secs (433631956 bytes/sec)

ポート割り当てはこんな感じになる予定です。
とりあえず、設定終わるまでオンボードとET dualの１ポートのみをVLANにいれといた。

オンボード(RTL8111DL) => SMB/リモート接続用
ET dual => iSCSI専用
CT => iSCSI or 予備
TW2 => NFS/バックアップタスク用

参考用のdmesgはこんな感じです。（横に長いからスクロールかソースでも見てください。）

[root@aria ~]# cat /var/run/dmesg.boot
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-PRERELEASE #0: Mon Dec 13 04:29:46 JST 2010
    aoyama@aria.ad.peach.ne.jp:/usr/obj/usr/src/sys/GENERIC amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) II X2 250e Processor (3000.09-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x100f63  Family = 10  Model = 6  Stepping = 3
  Features=0x178bfbff
  Features2=0x802009
  AMD Features=0xee500800
  AMD Features2=0x37ff
  TSC: P-state invariant
real memory  = 17179869184 (16384 MB)
avail memory = 16270336000 (15516 MB)
ACPI APIC Table: <7623MS A7623100>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0  irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <7623MS A7623100> on motherboard
acpi0: [ITHREAD]
ACPI Error: [PCI0] Namespace lookup failure, AE_NOT_FOUND (20101013/dswload-772)
ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20101013/psloop-326)
ACPI Error: Method parse/execution failed [\\] (Node 0xffffffff80b2e160), AE_NOT_FOUND (20101013/psparse-633)
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of ffb80000, 80000 (3) failed
acpi0: reservation of fec10000, 20 (3) failed
acpi0: reservation of fed80000, 1000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, cff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0:  on acpi0
cpu1:  on acpi0
acpi_hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
pcib1:  at device 1.0 on pci0
pci1:  on pcib1
vgapci0:  port 0x3000-0x30ff mem 0xd0000000-0xdfffffff,0xfd7f0000-0xfd7fffff,0xfd600000-0xfd6fffff irq 18 at device 5.0 on pci1
pci1:  at device 5.1 (no driver attached)
pcib2:  irq 18 at device 2.0 on pci0
pci2:  on pcib2
igb0:  port 0x5800-0x581f mem 0xfe4e0000-0xfe4fffff,0xfe000000-0xfe3fffff,0xfe4dc000-0xfe4dffff irq 18 at device 0.0 on pci2
igb0: Using MSIX interrupts with 3 vectors
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: Ethernet address: 00:1b:21:49:ec:60
igb1:  port 0x5400-0x541f mem 0xfe4a0000-0xfe4bffff,0xfd800000-0xfdbfffff,0xfe4d8000-0xfe4dbfff irq 19 at device 0.1 on pci2
igb1: Using MSIX interrupts with 3 vectors
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: Ethernet address: 00:1b:21:49:ec:61
pcib3:  irq 16 at device 4.0 on pci0
pci3:  on pcib3
em0:  port 0x6800-0x681f mem 0xfe5e0000-0xfe5fffff,0xfe500000-0xfe57ffff,0xfe5dc000-0xfe5dffff irq 16 at device 0.0 on pci3
em0: Using MSIX interrupts with 3 vectors
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
em0: Ethernet address: 00:1b:21:7d:2c:a3
pcib4:  irq 17 at device 5.0 on pci0
pci4:  on pcib4
re0:  port 0x7800-0x78ff mem 0xfcfff000-0xfcffffff,0xfcff8000-0xfcffbfff irq 17 at device 0.0 on pci4
re0: Using 1 MSI messages
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00000000
miibus0:  on re0
rgephy0:  PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 6c:62:6d:4e:8c:a5
re0: [FILTER]
pcib5:  irq 18 at device 6.0 on pci0
pci5:  on pcib5
ahci0:  port 0xb800-0xb807,0xb400-0xb403,0xb000-0xb007,0xa800-0xa803,0xa400-0xa40f mem 0xfe7ff800-0xfe7fffff irq 18 at device 0.0 on pci5
ahci0: [ITHREAD]
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0:  at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1:  at channel 1 on ahci0
ahcich1: [ITHREAD]
ahci1:  port 0xa000-0xa007,0x9800-0x9803,0x9400-0x9407,0x9000-0x9003,0x8800-0x880f mem 0xfe7ff400-0xfe7ff40f irq 19 at device 0.1 on pci5
ahci1: AHCI controller reset failure
device_attach: ahci1 attach returned 6
pcib6:  irq 19 at device 7.0 on pci0
pci6:  on pcib6
pci6:  at device 0.0 (no driver attached)
pcib7:  irq 17 at device 9.0 on pci0
pci7:  on pcib7
atapci0:  port 0xd800-0xd807,0xd400-0xd403,0xd000-0xd007,0xc800-0xc803,0xc400-0xc40f irq 17 at device 0.0 on pci7
atapci0: [ITHREAD]
ata2:  on atapci0
ata2: [ITHREAD]
ahci2:  mem 0xfec01000-0xfec013ff at device 17.0 on pci0
ahci2: [ITHREAD]
ahci2: AHCI v1.20 with 6 6Gbps ports, Port Multiplier supported
ahcich2:  at channel 0 on ahci2
ahcich2: [ITHREAD]
ahcich3:  at channel 1 on ahci2
ahcich3: [ITHREAD]
ahcich4:  at channel 2 on ahci2
ahcich4: [ITHREAD]
ahcich5:  at channel 3 on ahci2
ahcich5: [ITHREAD]
ahcich6:  at channel 4 on ahci2
ahcich6: [ITHREAD]
ahcich7:  at channel 5 on ahci2
ahcich7: [ITHREAD]
ohci0:  mem 0xfd5ff000-0xfd5fffff irq 18 at device 18.0 on pci0
ohci0: [ITHREAD]
usbus0:  on ohci0
ehci0:  mem 0xfd5fec00-0xfd5fecff irq 17 at device 18.2 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1:  on ehci0
ohci1:  mem 0xfd5fd000-0xfd5fdfff irq 18 at device 19.0 on pci0
ohci1: [ITHREAD]
usbus2:  on ohci1
ehci1:  mem 0xfd5fe800-0xfd5fe8ff irq 17 at device 19.2 on pci0
ehci1: [ITHREAD]
usbus3: EHCI version 1.0
usbus3:  on ehci1
pci0:  at device 20.0 (no driver attached)
pci0:  at device 20.2 (no driver attached)
isab0:
 at device 20.3 on pci0
isa0:  on isab0
pcib8:  at device 20.4 on pci0
pci8:  on pcib8
em1:  port 0xe800-0xe83f mem 0xfe9e0000-0xfe9fffff irq 20 at device 2.0 on pci8
em1: [FILTER]
em1: Ethernet address: 00:04:23:a8:d4:5c
em2:  port 0xe400-0xe43f mem 0xfe9c0000-0xfe9dffff irq 21 at device 2.1 on pci8
em2: [FILTER]
em2: Ethernet address: 00:04:23:a8:d4:5d
ohci2:  mem 0xfd5fc000-0xfd5fcfff irq 18 at device 20.5 on pci0
ohci2: [ITHREAD]
usbus4:  on ohci2
ohci3:  mem 0xfd5f7000-0xfd5f7fff irq 18 at device 22.0 on pci0
ohci3: [ITHREAD]
usbus5:  on ohci3
ehci2:  mem 0xfd5fe400-0xfd5fe4ff irq 17 at device 22.2 on pci0
ehci2: [ITHREAD]
usbus6: EHCI version 1.0
usbus6:  on ehci2
acpi_button0:
 on acpi0
atrtc0:  port 0x70-0x71 irq 8 on acpi0
atkbdc0:  port 0x60,0x64 irq 1 on acpi0
atkbd0:  irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0:
 irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
ppc0:
 port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: 
 on ppc0
plip0:
 on ppbus0
plip0: [ITHREAD]
lpt0:
 on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0:
 on ppbus0
acpi_hpet1:  iomem 0xfed00000-0xfed003ff on acpi0
device_attach: acpi_hpet1 attach returned 12
orm0:  at iomem 0xc0000-0xcefff,0xd2800-0xd37ff on isa0
sc0:  at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0:  at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0:  on cpu0
hwpstate0:  on cpu0
ZFS filesystem version 4
ZFS storage pool version 15
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 480Mbps High Speed USB v2.0
ugen0.1:  at usbus0
uhub0:  on usbus0
ugen1.1:  at usbus1
uhub1:  on usbus1
ugen2.1:  at usbus2
uhub2:  on usbus2
ugen3.1:  at usbus3
uhub3:  on usbus3
ugen4.1:  at usbus4
uhub4:  on usbus4
ugen5.1:  at usbus5
uhub5:  on usbus5
ugen6.1:  at usbus6
uhub6:  on usbus6
uhub4: 2 ports with 2 removable, self powered
uhub5: 4 ports with 4 removable, self powered
uhub0: 5 ports with 5 removable, self powered
uhub2: 5 ports with 5 removable, self powered
uhub6: 4 ports with 4 removable, self powered
uhub1: 5 ports with 5 removable, self powered
uhub3: 5 ports with 5 removable, self powered
ugen3.2:  at usbus3
umass0:  on usbus3
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:8:0:-1: Attached to scbus8
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0:  ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
ada1:  ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich3 bus 0 scbus3 target 0 lun 0da0 at umass-sim0 bus 0 scbus8 target 0 lun 0
da0:  Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 3822MB (7827456 512 byte sectors: 255H 63S/T 487C)

ada2:  ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich4 bus 0 scbus4 target 0 lun 0
ada3:  ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich5 bus 0 scbus5 target 0 lun 0
ada4:  ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ahcich6 bus 0 scbus6 target 0 lun 0
ada5:  ATA-8 SATA 2.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada6 at ahcich7 bus 0 scbus7 target 0 lun 0
ada6:  ATA-8 SATA 2.x device
ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
Trying to mount root from zfs:tank/root-82