Madgenius.com

We are buying a lot of Dell PE860's with the SAS Raid card for dedicated hosting. Here is some info for working with that card.

SAS 5/iR adapter is using chip SAS1068 from LSI. The adapter is supported by mptsas driver in Linux 2.6.18 kernel.

Monitoring of RAID status is possible by mpt-status program. We have the RedHat rpm here

Make sure mptctl is loaded and the node is created.

mknod /dev/mptctl c 10 220
modprobe mptctl

Here is an output:

[root@bolt40 root]# mpt-status
ioc0 vol_id 0 type IM, 2 phy, 148 GB, state OPTIMAL, flags ENABLED
ioc0 phy 1 scsi_id 32 ATA WDC WD1600JS-75N 2E04, 149 GB, state ONLINE, flags NONE
ioc0 phy 0 scsi_id 1 ATA WDC WD1600JS-75N 2E04, 149 GB, state ONLINE, flags NONE

By default SAS 5/iR RAID controller came with cache set to write through. That really horribly degrades performance

[root@bolt40 lsiutil]# time (dd if=/dev/zero of=bigfile count=10240 bs=1M; sync)

10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 257.313 seconds, 41.7 MB/s

real 6m32.576s
user 0m0.012s
sys 0m25.194s

The transfer rate is 10737418240 / (6*60 + 32.576) / 1024 / 1024 = 26.08 MiB/sec. YUCK!
after googling I found some answers.

The tool you need is LSIUtil, it is inside Dell SAS 5/iR Adapter Driver. Cache tuning is not accessible via RAID controller BIOS. If you prefer you can take pre-compiled version of lsiutil-1.56 from our server.

Usage of lsiutil is simple:

[root@bolt40 lsiutil]# ./lsiutil

LSI Logic MPT Configuration Utility, Version 1.56, March 19, 2008

1 MPT Port found

Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC
1. /proc/mpt/ioc0 LSI Logic SAS1068 B0 105 000a3100 0

Select a device: [1-1 or 0 to quit] 1

1. Identify firmware, BIOS, and/or FCode
2. Download firmware (update the FLASH)
4. Download/erase BIOS and/or FCode (update the FLASH)
8. Scan for devices
10. Change IOC settings (interrupt coalescing)
13. Change SAS IO Unit settings
16. Display attached devices
20. Diagnostics
21. RAID actions
22. Reset bus
23. Reset target
30. Beacon on
31. Beacon off
97. Reset SAS phy
98. Reset SAS link
99. Reset port

Main menu, select an option: [1-99 or e for expert or 0 to quit] 21

1. Show volumes
2. Show physical disks
3. Get volume state
23. Replace physical disk
30. Create volume
31. Delete volume
32. Change volume settings

RAID actions menu, select an option: [1-99 or e for expert or 0 to quit] 32

Volume: [0-1 or RETURN to quit]

Volume 0 Settings: write caching disabled, auto configure, priority resync
Volume 0 draws from Hot Spare Pools:

Enable write caching: [Yes or No, default is No] yes
Offline on SMART data: [Yes or No, default is No]
Auto configuration: [Yes or No, default is Yes]
Priority resync: [Yes or No, default is Yes]
Hot Spare Pools (bitmask of pool numbers): [00 to FF, default is 01]

RAID actions menu, select an option: [1-99 or e for expert or 0 to quit]

Main menu, select an option: [1-99 or e for expert or 0 to quit]

Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev
1. /proc/mpt/ioc0 LSI Logic SAS1068 B0 105 000a310

Select a device: [1-1 or 0 to quit]

Setting seams to be persistent, it survives system reboot.

[root@bolt40 lsiutil]# time (dd if=/dev/zero of=bigfile count=10240 bs=1M; sync)
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 158.017 seconds, 68.0 MB/s

real 3m59.514s
user 0m0.017s
sys 0m27.018s

Write rate is now a lot bigger: 10737418240 / (3 * 60 + 59.514) / 1024 / 1024 = 42.75 MiB/sec.

Possible reason why LSI do not provide this feature inside adapter BIOS might be because this card doesn't have it's own battery backed up memory, adapter does rely on cache of HDDs. I'm afraid that in case of power failure during massive I/O, system could end with badly corrupted file system.

The base Xen installation is configured such that all domains share a bridge to the first available ethernet interface. Although this makes testing VMs quick and painless, however our VPS hosting plans needed to offer multiple ip address blocks to a single Dom0 with only 2 network cards so I started working with Vlans.

Now, you obviously need to have a Xen dom0 set up and running properly. Once you are at that point, adding VLAN bridging is fairly straightforward, CentOS 5.4 has vlaning already installed.

create ifcfg-eth0.2 (vlan2)

DEVICE=eth0.2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
IPADDR=74.119.217.xxx
NETMASK=255.255.255.0
VLAN=yes
GATEWAY=74.119.217.x

create ifcfg-eth0.3 (vlan3)

DEVICE=eth0.3
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

ifconfig eth0
eth0 Link encap:Ethernet HWaddr A4:BA:DB:16:26:77
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:468338734 errors:0 dropped:0 overruns:0 frame:0
TX packets:498911873 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:156460216228 (145.7 GiB) TX bytes:114246923487 (106.4 GiB)
Interrupt:24 Memory:da000000-da012800

We only bring up the eth0 interface automatically but leave it unconfigured. It will not have an IP or any of the other usual information.

In this example the “vlan2″ interface is the replacement for your old “eth0″ interface. Its the interface you’ll be able to reach the dom0 on from the internal VLAN. The “vlan3″ interface is defined but is left unconfigured since there will be no publicly routeable path to the dom0.

Now we create a new network script for xen located at /etc/xen/scripts/network-bridge-wrapper

#!/bin/sh
/etc/xen/scripts/network-bridge netdev=eth0.2 start
/etc/xen/scripts/network-bridge netdev=eth0.3 start

That script basically just calls the standard “network-bridge” script supplied with Xen two times – once for each VLAN we are setting up.

Go ahead and add the call to that script in the place of network-bridge in /etc/xen/xend-config.sxp.

Now we need to do one final thing to another Xen script. By default, the xen-network-common.sh script calls “ifdown” directly to take an interface down. Unfortunately, this has the undesirable side effect of destroying any VLANs associated with that interface. This causes all kinds of problems when we call our “network-multi” script so we need to patch /etc/xen/scripts/xen-network-common.sh with the following section:

# do not call ifdown directly
ifdown() {
ip addr flush $1
ip link set $1 down
true
}

After youve done all that, reboot your dom0. You should now be up and running on the vlan interfaces. The output of “brctl show” should show this:

brctl show
bridge name bridge id STP enabled interfaces
eth0.100 8000.a4badb162677 no vifvm123.0
peth0.100
eth0.2 8000.a4badb162677 no vifvm138.0
vifvm137.0
vifvm127.0
vifvm126.0
vifvm125.0
peth0.2
eth0.3 8000.a4badb162677 no vifvm136.0
vifvm135.0
vifvm134.0
vifvm133.0
vifvm132.0
peth0.3

If so, you should be all ready to go. Now edit the vif line in your config file for a given domU to link its ethernet interface to a bridge.

vif = ['type=ioemu, ip=74.119.217.247 74.119.217.248, vifname=vifvm137.0, mac=00:16:3e:f3:28:f8, bridge=eth0.2']