DataDirect Networks: SFA10000 is back in our lab

SFA10000 is back in our lab 16 FC8 connected to the hosts (front-end bandwidth is 12,8GB/s!!!) 40 SAS 4X links to the back-end enclosures (5 enclosure with 60 disks each) For cache mirroring between the controlers, Mellanox QDR/10GigE interfaces are now replacing the previous bonded 1Gb ethernet interfaces. 4 hosts are connected for performance tests: each node is using 4 FC8 ports and 1 IB QDR, Lustre should be able to get 3,2GB/s on each OSS

DDN S2A9900 - 60 disks enclosures

Nice and performant system: we've been able to get more than 5GB/s when writing on Lustre FS other S2A9900...



We also get tons of boxes with SATA and SAS disks for our tests. There are 24 disks in a box.



We can put up to 60 disks in each enclosure. Our test configuration is using 5 enclosures, with 300 disks. Up to 20 enclosures (1200 disks !!!) can be connected to one S2A9900 system, for huge capacity needs...

SMP and rpmbuild

To use multiples processors when builing/compiling RPMs:


edit your .rpmmacros file and define %make macro:


%make make -j 4

Linux ALUA Multipath with EMC CX 4-120

First, on EMC CX 4-120, Check that storage groups on EMC CX 4-120 are configured for ALUA (Assymetric Logical Unit Access): failovermode should be 4

Note: if no agent is running on server side, arraycommpath should be set to 0.

Then, on server side (linux OS is RHEL5.3) we are using following /etc/multipath.conf file:

blacklist {
device {
vendor ".*"
model  ".*"
}
}
blacklist_exceptions {
device {
vendor  "DGC.*"
model  "RAID.*"
}
}
devices {
device {
vendor  "DGC.*"
product ".*"
path_grouping_policy    failover
getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
prio_callout            "/sbin/mpath_prio_alua /dev/%n"
no_path_retry           fail
path_checker            alua
failback                immediate
}
}

Note: blacklist and blacklist_exceptions should use the same kind of definitions: If some devices are blacklisted by devnode regexp, exceptions for these devices have to be defined by devnode keyword too.
Here, we only want to apply multipath on EMC devices. So we use blacklist (by default, all devices will be blacklisted) and blacklist exception (EMC vendor TAG is DGC...) based on device keyword (with vendor/model fields).
So, for all EMC devices, multipath is applied using ALUA rules (priority, checker, ...).

Now, there's an issue with multipath tools (device-mapper-multipath package for RHEL5) and internal settings (hardcoded in tools) are overriding our settings from /etc/multipath.conf.
These internal settings will lead multipath to use default EMC multipath based on tresspass technology, and ALUA won't be used:
multipath errors are generated, tresspass commands are sent to the storage processors, and multipath devices won't work.
The solution suggested is to remove internal settings for EMC from multipath tools. These settings are located in libmultipath/hwtable.c file. Ask for someone who know how to modify/patch/rebuild/ RPMs if you don't know.

At last, here is an example of multipath failover based on ALUA for EMC CX 4-120:

[root@XXXXXX ~]# multipath -ll
3600601609fa02200007deb7d0881de11 dm-7 DGC,RAID 5 [size=1.2T][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=10][enabled]
\_ 7:0:0:0 sdb 8:16  [active][ready]
\_ round-robin 0 [prio=50][active]
\_ 9:0:0:0 sdc 8:32  [active][ready]

Issue when mounting multipath device with label

I got this issue yesterday, trying to mount a multipath device using label:

mount LABEL=ACME /mnt/acme
mount: /dev/sdc5 already mounted or /mnt/acme busy

strace and Google help me to solve this issue:

There's a cache file /etc/blkid/blkid.tab where information about block devices (labels, uuid) are stored. To solve/avoid this issue, just refresh this cache file:

• remove/backup the file
• regenerate it by running blkid command

Doing so, label will be found/used on multipath device first when mounting from label.