Wednesday, January 18, 2012

The Network is Failing!

Last summer I was faced with a interesting challenge. My old work brought me back to replace their aging servers which I originally installed in 2006. The old setup was barely keeping up with the business as it grew. Two Dell PE2900's served as the primary SBS2003 and Fileserver. Their primary storage array was comprised of 320gb Western Digital hard drives in a Raid 5. This worked great for about 3 years until the average file size grew over 350mb and the engineering staff doubled. Needless to say they also outgrew SBS2003 and have been painfully living with the now 6yr old install still grinding away.

I had to find a new solution which fit the following criteria:
Powerful
Stable
Expandable
and on a shoe string budget...

I had recently built out a $100k virtual environment for another client and this seemed like quite the challenge. I wanted to give them the same redundancy, performance, and expandability that I have previously found in commercially available solutions.

Step 1: Virtualize it. vSphere 5
I am a big virtualization fan and I can think of no better way to provide a powerful, stable, and expandable environment. As they grow, they can move away from the limitations of the free version and easily setup a HA environment. The hardware is fairly common:
- Two Dell R710's , Dual Quad e5620, SD card, dual powersupplies, 3yr warranty.
- 32gb ram each (upgradable to 256gb each)
- No local storage
- 4 Quad port Intel 1000 nics

Step 2: We need storage! ZFS SAN
I starting working with ZFS just over two years ago after reading about it while researching a home fileserver build. What started in my esxi lab soon moved to a 5tb fileserver to replace my original WHS, and even then I was never able to fully take advantage of the great features of ZFS.
Benifits of ZFS:
-Built by Sun and included in OpenIndiana 151a- the latest release from the Illumos project.
-Software like Raid with no write hole and incredible speeds.
-Very inexpensive to deploy, no special hardware.
-All raid sets supported (though named differently)
-Data encryption, file system level snap shots, and de-duplication.

After seeing the success stories of sites like www.zfsbuild.com and watching Gea's Napp-It web interface evolve, I decided it was worth setting up a test system for a proof of concept. My own personal success with it was a great starting point, but I wanted to really test the speed and redundancy of ZFS. We need stability and redundancy here, as this could easily be a single point of failure.

Step 3: Networking- Tie it all together!
This is where it gets fun. I planned originally on a basic two network design, separating the SAN and LAN network. Due to the projected growth and the ability to easily expand, I decided to add a third. The end design includes a third network, dedicated to management. This allows me to keep the traffic away from the end user, and keep the end user away from my management interfaces. Nothing like having someone find a password list and causing hours of damage for "curiosities sake". (It happens more then any IT person cares to admit.) Plus I now am free to use those spare IP's for more end users. Due to the budget, I went with HP 1800-24g switches. These are a low cost, web managed, gigabit switch that supports vlans, trunking, and LCAP.

Network layout:
- x.x.20.0 Management
- x.x.15.0 LAN
- 10.254.11.0 SAN

Step 4: VPN! Working from home has never been so sweet!
Theoretically this should fall under networking, but I have been super impressed with OpenVPN lately. Years ago I setup a small server to VPN in to my home network. The server crashed, I never rebuilt it, and moved to using other hardware based VPN clients for offices or the SBS included quick connection. This worked ok, but I was never really happy with it. Since they will most likely split to a multiple office setup here soon and I want to co-locate a backup server or two, I needed something slightly more robust which would support IPSEC and Road Warrior style connections. I have ran PFsense at home for a number of years and with the latest 2.0 release, setting up a secure VPN could not be easier.

This is just a very brief overview and I look forward to documenting each step.


Here are some quick pictures of the finished setup:







No comments:

Post a Comment