Watchdog 100 Climate Monitor and WD-1 Water sensors.

With my recent minor water issue, I decided to add an additional climate monitor to both our Warehouse Data Center and main office server room. The last thing any IT admin wants is a failed HVAC unit or leaking roof taking down critical infrastructure. I have used ITWatchdogs MiniGoose II's in the past with great success and decided to purchase two WatchDog 100's with WD-1 water sensors.

From the ITWatchdogs website:

Specifications 
On-board Sensors: 
Temperature: -22 to 185 °F (-30 to 85 °C), +/- 0.5 °C 
Humidity: 0 to 100%, +/- 2% 

Power: 6VDC power supply (included) 
Ethernet: 10/100 Mbps, RJ-45 receptacle 
Real Time Clock (RTC) with power backup 
Reset IP push-button: restores factory defaults 
Warranty: 1 year (extended warranties available) 

Software Features: 
HTTP/HTTPS: web access 
Alarms: high/low values, multiple thresholds per sensor 
ESMTP: email alerts, ESMTP auth 
SNMP (v1, v2c): gets, trap and clear alerts, MIB 
Paging: email to pager proxy 
XML: meta-tagged sensor values, alarms, config 
Syslog: send debug messages to Syslog server 
Access-control: 3 access levels (admin, control, view only) 
Web cams (optional): Up to four can be displayed 
Compatible with WatchDog Console Aggregator 

I can't wait to get these setup and tested. Hopefully they will never go off, but I would rather be safe then sorry.

Marc

What causes server room AC units to freeze? Skipping maintenance!

It's getting warmer here in Oregon. We peaked at 98F last week, which must seem like nothing compared to other folks in 120F+ climates, but when you're used to it be 70F and overcast, it reminds you why you have AC both at home, in your car, and in your server room. Our warehouse Data Center at work is equipped with two 5 Ton AC units and this weekend, the primary unit iced up. After the unit shut itself down, it did what a block of ice does best in a 90F room: Melt. We don't have a raised floor, so water ended up under the first three racks, storage area, and in the adjoining office space on the other side of the wall. This was only a minor disaster- we could have lost UPS's, PDU's, and server gear. But what caused it?

Dirty, dirty filters.

I'll be honest here, I'm solely to blame on this one. I could spin a story on how my boss never responded to my inquiry about getting maintenance contracts on the units, or how the staff at that facility are supposed to handle building/equipment maintenance, but at the end of the day- it was on me. When I got a $1.1k quote for 1 year of filter changes/inspections, I laughed and told myself I can easily handle this, no need to bother the staff. The filters were only $200 and there's a local supplier down the street. No problem, right? That was... 8 months ago. As it turns out, the filters I forgot to change were the problem. The restriction caused by the dirty filters wasn't allowing enough air to pass through the super cold coils. Add some humidity and bam. Ice block. After a trip to a local  supplier and a quick visit from the HVAC tech to verify everything was ok- it's back up to running at capacity.

So don't skimp or forget about maintenance! And don't worry, spare filters are stocked and a calendar reminder (or two) has been put in place. 

*EDIT*
Sadly the unit froze up again last night, even worse. While we wait on a diagnosis from the HVAC company, I've gone ahead and ordered moisture sensors.

Marc

Dell T110 II Review - The perfect home ESXi, XEN, or GNS3 lab server

I have been slowly paring down the gear I have at home. Like many admins, I just want something simple that just works. I play with the big equipment at work and I don't need a power sucking dual 12 core/256gb ram box at home just to load the latest hypervisor and server OS on to play with. I recently came across a deal for a $249 Dell T110 II server and could not pass it up. It's the perfect lab box and here's why:

1) It's a normal tower form factor. It can easily sit under or on your desk and not be in the way.
2) It's powerful. A simple processor upgrade on my low end sale-model Celeron G550 to a e3-1220v2 makes this a quad-core work horse. Same goes for the ram. It supports up to 32GB of DRR3 ECC.
3) It's quiet. I have it sitting under my desk and can't hear it over my HP micro server.
4) It's well built; the case material feels solid, the internal layout gives plenty of working room, there are two internal USB headers on the motherboard for thumb drives , and it has plenty of PCI-E express slots to add additional network adaptors or a raid controller.
5) It has IPMI or Remote management.
6) It barely sips power. With the latest gen Intel Xeon, 32GB of ram, four hard drives (two SSD/two spindle), and a Intel Pro 1000 PT Quad port NIC, I'm barely drawing 46 watts at idle with half a dozen VM's spooled up. That's nothing! My HP N54l Microserver draws the same amount of power with 6 spindle drives and 1/10th the processing power.
7) It will run anything. Xen, ESXi, Linux, Windows, GN3, it does not matter- it has great driver support. You can add up to 18 NIC ports with 6 port PCI-E nics. You can fill it full of hard drives, install OMNI/ESXi and turn it in to a micro Napp-it all-in-one appliance on the cheap.
8) It has business support. I "loathe" Dell's home support. Business class has always been top notch.


So what's it missing?
First off, I have to knock a solid point off for it having a SINGLE NIC. Really Dell? Even my whitebox Supermicro gear has three (one dedicated IPMI and two gigabit nics). There's plenty of room to add additional though.
Second, and this is more my fault for not spending the extra $20, but it only has four hard drive bays. It would make a great FreeNas/Omni ZFS box if it had 6 + the two 3.5" bays.
Third, the bios is straight out of the 90's. It's just like every other server motherboard thought, so I can't really complain. Having the power state options not be under the power settings menu and instead have it hidden under something completely unrelated is just unnecessarily confusing.
Fourth and finally, it lacks USB 3.0- this can be fixed via addon card, but by the time I add this, an additional NIC, ram, hdd's, and processor- I'm close to $1k in to this box.

It's still a killer little box and one that I hope will last me for years to come. I love the tool-less hard drive trays and that they even included the correct length SATA cables for all four HDD's. The power supply is pretty low rated, but unless you are tossing a dozen spindle drives in here and maxing everything out,  I can't see it being a problem. I'm not sure why they still include a DVD rom, I personally would rather have more hard drive bays or save the $10 it costs to add in. This deal pops up every year it seams and similar ones throughout the year. Sure it may be $289 next time, but I would buy another if I had a need for a second lab box.

(Air ducting removed for photo.)

Cisco IPSEC site-to-site VPN connected but not passing traffic- solution

 Last week at work we finished up a major storage system upgrade. Two days later, our site-to-site VPN started to act up. Coincidence? I figured it was the construction and flood of trucks from the local provider in the area lately, and put it off until Monday. No users were complaining and I was working remotely Friday (the Anyconnect VPN was stable) and could access both sites just fine. After more testing, I realized I could ping site A router to site B router and site B to site A, I quickly realized there was nothing wrong with the physical fiber link between the sites. Logging out/ restarting the VPN allowed the connection to work for a short period of time, and the only downtime the ASA logged was when I reset the VPN or restarted the adapter. The tunnel otherwise would stay up. Some quick Google'ing returned this:


http://networkengineering.stackexchange.com/questions/2155/site-to-site-vpn-tunnel-up-not-passing-traffic

Our setup has the exact same issue: The tunnel stayed alive, but no traffic would pass. A quick look at the crypto security-association lifetime and kilobytes showed that we had a 8 hour and 4.5GB transfer cap. Our site-to-site fiber line can do 18.5GB in~ 1 hour on a congested day.  It appears as though our new storage system upgrade, which included site-to-site replication, was maxing out the security-association lifetime. Changing this to 8.6.5 fixes the problem, but so does increasing the kilobytes threshold. Changing ours to just over the peak transfer rate should prevent this from happening again until I can schedule downtime to upgrade the firmware. So far it has returned to its normal state and we can resume the replication.

Best of luck!
- Marc

Using a Raspberry Pi for a Nagios display

When I heard about the Raspberry Pi being released, I preordered waaaaay to many. My house has two for XBMC and a third for a side project. I gave one away and used the last here at work. People ask quite a few questions when they see it, so here is the setup.

Our Pi is just browsing two websites, Cacti's weathermap and Nagvis's picture- hostesd off the Nagios server.

Hardware:
Raspberry Pi Model B 512MB
4GB SD card
Wall-wort charger
HDMI cable
Any 1080p TV of your choice

http://www.cacti.net/
http://www.nagvis.org/

It's low power and small footprint make it ideal for this task. I'm hoping to move our kitchen computer over too.

Let me know if you have any questions!

IOMETER not showing all physical drives

I started to benchmark a new SSD only test SAN at work this week and after running several tests configuered in a RAID setup, I wanted to test the physical raw drives themselves. After deleting the partitions, I could not get the disks to show in IOMETER. The general solution to this is to run IOMETER as administrator under S2K8R2. This did not work for me, as the drives still had some partition reminants from the RAID. The solution: Use diskpart to clean the drives.

1) Open an Administrative CMD prompt- (right click CMD- run as administrator)

2) # diskpart

3) # select disk 1

4) # clean

Repeat until all the disks you want to test are clean. DO NOT CLEAN YOUR OS DISK.

5) # exit

Make sure IOMETER is being ran as administrator and all your disks should now show. Good luck!

Marc

Flashing IT firmware on a LSI 9211-8i HBA from a USB stick

Recently I needed to test different firmware versions on some LSI 9211-8i PCI-E 8x SAS 6Gbps HBA's for work. Updating the firmware seemed like a momentus pain afer reviewing the plethra of guides online. Here is my simplifyed version:

Step 1: Download the HP Flash Utility and MS-DOS system files

LINK

Step 2: Install the HP Flash Utility and extract the MS-DOS system files


Step 3: Download the latest LSI firmware: 9211_8i_Package_P16_IR_IT_Firmware_BIOS_for_MSDOS_Windows

LINK

Step 4: Extract the LSI firmware zip.

 Step 5: Follow the great guide from the link in Step 1 on Seven Forums for creating your dos bootable USB thumb drive.

Step 6: Copy the following files from the LSI firmware package to your thumb drive.

2118it.bin - Under Firmware\HBA_9211_8I_IT
sas2flsh.exe - Under sas2flash_dos_rel
mptsas2.rom - Under sasbios_rel

Step 7: Plug your USB stick into the system you are flashing the 9211-8i controller in and boot off the thumb drive.

*IMPORTANT: DO NOT TRY TO FLASH MORE THEN (1) HBA CARD AT A TIME. Remove any additional controllers and repeat the next steps. You could perminatly brick multiple cards!*

Step 8: View the installed cards using # sas2flsh -listall

(Please ignore the onboard LSI 2308 controller)

 

Step 9: Delete the existing firmware # sas2flsh -o -e 6

Step 10: Update the firmware and bios # sas2flsh -f 2118it.bin -b mptsas2.rom

 All done!

 

There, that was not too bad. You can check the update by running # sas2flsh -listall and power cycle the machine- don't forget to remove the USB stick.

 

Marc