A ESXi colo lab build. Part 1: Hardware

Happy Holidays! It's always a busy time of year around home, but thankfully I got to take 10 days off work! I figured it was time to update the blog, with work related things.... wait a second I'm supposed to be relaxing! :)

Moving on.

Since starting my new job nearly a year ago my home lab has been kept busy running several test environments for work. I watched my powerbill go up $30/m, despite using fairly power conservative hardware. Not to mention the $50/m second DSL line for firewall evals (nothing like the real world to proof something out.) Ok ok, maybe I could have skipped the 2nd internet connection, but still! At work, the warehouse data center has finally started to shape up. Our power work was just completed last month and everything seems to be running smoothly. No better time then now to colo a new lab box :).

Plus who wants this in their living room?

(Old home lab, testing HP switch gear and FreePBX phone system.)

Like 90% of everything else I build, I started by gathering recycled parts. After months a few months of cleaning and saving random bits from recycled systems before they headed to the e-waste bin, I finally came up with a random assortment of project leftovers, e-waste, and some "new" parts. The chassis was one I bought for a previous ESXi + OI all in one build that ended up shelved. The motherboard was a open box special off newegg, and the ASA/HP switch were left overs from a early switching lab. Now I realize to most people this is not what you would find headed for recycling and I would not let it. I try to keep a small stock of parts like this for in house use. You never know when you will need that 32GB of DDR3 or LSI HBA's.

System specs:

Supermicro SC826E1-R900 with hotswap powersupplies

(2) Intel Xeon x5650 2.66/12mg 6 core processors

Supermicro X8DT3-O with IPMI 

(12) 16GB DDR3 ECC REG

(2) LSI 9211-8i HBA

(1) LSI 9211-8e HBA

(2) Intel 160gb x25

(5) Crucial M4 256gb

(5) Crucial M4 512gb

(2) HP 16GB Thumbnail flash drives for ESXi install

Storage:

Supermicro SC846A-R900B 24 bay JBOD

24x Seagate 1TB 7200rpm drives

Networking:

HP 1910-24G

Cisco ASA5505-SECPLUS

Backup:

Synology DS219j

2x3TB drives

I recently upgraded out licensing from vSphere Essentials to Essentials plus, but kept the old licensing for two test servers and this lab box. Once all setup, I should be able to VPN in and setup how ever many test boxes, maybe a game server or four, haha. 

(Early picture with 48GB)

I can't wait to get this racked. Not only will it vastly increase the number of VM's I can run, but it will not longer cost me ~$80/m to run at home. I can not stress this enough- having a lab environment to test out routing issues, build server environments, and even built images in that is not directly associated with your production environment is critical! I got away with a Quad Core/32gb ram and a small HP n40l SAN for 15 machines, but it was painful.

Next up: Racking and storage configuration. I plan on building that ESXi + OpenIndiana + ZFS/NAPP-IT pass through All-In-One system and using it. Once everything is setup, it is time to play!

Lexmark scanning to email with Gmail

I recently replaced several of our printers with Lexmark X548 multi-function units. I finally got a chance to setup scan to email and hit an all to common bug. We use Google Apps for Business and no

matter what settings I tried, I could not for the life of me get authentication to work. I kept getting 530 errors. Finally I found a post on a unrelated printer suggesting I try a Google SMTP server I had never heard of: aspmx.I.google.com.

Using the following settings, it works perfect.

Primary SMTP Gateway: aspmx.I.google.com

Primary SMTP Gateway Port: 25

Use SSL/TLS: Disabled

SMTP Server Authentication: No Authentication Required

Device UserID: username@yourdomain.com

Device Password: password

Hope this helps someone!

The HP Microserver N40l- Your next ZFS based home NAS!

I know I am posting this late in the scheme of things, but a colleague at work asked me this week what I use at home for my NAS. At first, I recommended a Synology DS1512j, but not everyone has that kind of money. He wanted something cheap and expandable. We use Solaris/ZFS on a daily basis to store petabytes of data and I adopted Openindiana 151a as my home fileserver days after its release. Shortly after, HP released the 2nd revision of its Microserver, the n40l. What a great combo! A low powered server that natively supports 4 hot swap disks for $249 on sale? Why not?

Here is the quick run down:

Hp N40l
Amd Turion II Dual Core 1.5ghz
8gb 2x4gb Unregistered DDR-3 ECC ram max
1000G-baseT ethernet
USB/ESata
4x 3.5 Internal hard drive bays
1x 5.25 ODD bay

My personal config adds:
6x 3tb 3.5" Hitachi drives, 2 in the ODD bay
1x LSI 1068e HBA
2x 160gb Intel x25 SSD's in the ODD bay
1x 80gb Intel x25 SSD in the space between the ODD bay and lower case
1x Intel 1000G-baseT PCI-E 1x nic

Running:
OpenIndiana 151a
Napp-It (www.napp-it.org)

This gives me 12TB of storage with Zil and Write cache drives at 30watts! I was so happy with it, I bought a second when they came on sale for $199. It is currently my ESXi NFS datastore configured below:

N40l
8gb Unregistered DDR3 ECC
LSI 1068e
Intel Pro1000 Nic
4x 1.5tb Seagate 7200rpm in RAIDZ
4x 160gb Intel x25 in Raid10
Icydock 4x 2.5 in one 5.25 bay

This serves up 220mbps Write and 680mbps Read! Considering these are older Sata 2 drives and controllers, I am perfectly happy, considering I am limited to a single 1gb link to my ESXi host.

Anyways,
If you are looking for a great home NAS, look no further then a N40l running either Solaris/FreeNas.

pfSense- retiring the google hackery for a Cisco 881w

Saturday was a sad day.

My wife came and told me the closet was screeching at her. After investigating, it was the fans in the 1U powersupply of the Google pfSense appliance. I had no 40mm fans in stock, Fry's was out and my only choice was to replace it. I have had a Cisco 881w kicking around, waiting to be installed at my parents house a state away, along with a spare n40l Hp Microserver for remote backups. Thankfully I had it configuered and ready to go. All I had to do was change some IP addresses and plug it in. I did disable the built in AP, seeing as we have a Unifi LR Access Point that does a great job covering our house and shop.

The worst part? I do not think I ever took advantage of all the features pfSense had to offer. The 881w is a solid little router and I do not think I will ever replace it with the old Google appliance. My next system will be smaller, maybe a alix board? Who knows! Anyone want to buy a Dual Core Atom and 2gb ram?

Hp Procurve Inter-Vlan routing with a Cisco ASA firewall

Long-winded network post ahead! You have been warned.

As part of my network overhaul here at work, I wanted to transform our current semi-flat network in to a multi-teared, access controlled, dynamic network that could grow with the company. Our existing network has been plagued with broadcast storms caused by the rouge engineering DHCP server being accidentily connected to the office network. To do this I purchased new switch gear that supports L3 routing and VLANs. This new gear allows me to seperate our large broadcase domain in to smaller, department based broadast domains using VLANs and Inter-vlan routing. The existing network gear, while functional, lacked the capability of Inter-vlan routing and strugged under our daily office load with only two VLANs. I can't say I will miss the old Netgear switches, but they were barely able to support the traffic when we were a 30 person company and are unstable with the 70+ now.

Wanting to keep a fairly tight budget, I ended up choosing HP Procurve 2510 and 2520 POE switches for distribution and a 5406zl loaded with 1Gb modules for my core. The 2510/2520 are layer 2 gigabit switches and the 5406 is layer 3. If I had a larger budget, EDU discount, or was purchasing a huge lot of gear, I would probably have gone Cisco 3750G/2960G. The HP gear is very competitive, offering a lifetime warranty, lifetime support, and the cheapest 10G-baseT I could find. I have worked with HP in the past and have found it very simmilar to manage. The menu based cmd line interface makes it a breeze for the novice, but I still prefer the straight old cmd line.

My firewalls were a tough choice. I wanted something that could support 250+ VPN SSL vpn connections, a Gigabit Metro-E line, a 100Mb EDI line, and have enough throughoput to handle all of this. After looking at Forigate, Juniper, and Cisco, I ended up choosing four Cisco ASA 5515-x's. Each site will have two, setup in Active/Active serving up a maximum of 500 SSL VPN connections per site. I sacrificed the ability to load balance across two or more internet connections, but our EDI line makes up for that. These, at least for now should be able to handle everything we throw at them.

In the last few weeks, I setup all of the HP switch gear in a test enviroment, along with a ESXi host with multiple quad port nics. I wanted to simmulate having multiple machines across multiple switches to ensure my configs would work. Starting out, I got everything up and working. I could ping between Vlans, but I did not have a DHCP server to test ip helper-addresses or a internet connection. This week I added a Server 2008 R2 box and setup DHCP/AD/DNS and connected a spare Cisco ASA 5505 running 8.4. After a few hours of research through somewhat helpful posts, I came up with the following basics to using Inter-vlan routing on HP Procure switches with a Cisco ASA.

Helpful tips:
1) Your core must be a Layer 3 switch. In my lab it is the hp2910al-24g. It is not possible to do this without a L3 switch.
2) On the core, there should be no default gateway. I have seen this far to often as the problem in my research.
3) Enable ip routing on the core switch.
hp2910al-24g:# ip routing

4) Once you create additional VLAN's, only use the default VLAN for switch management if possible.
               hp2910al-24g:# config
               hp2910al-24g:# vlan 10
               hp2910al-24g:(Vlan 10)#

5) Assign IP addresses to each VLAN- only on the core!
               hp2910al-24g:(Vlan 10)# ip address 10.1.0.1/24

6) Assign a ip helper-address for your DHCP server to each VLAN on the core switch (except the one it natively lives on) and add each scope to the DHCP server.
               hp2910al-24g:(Vlan 20)# ip helper-address 10.1.0.2
              
              
7) Be sure to TAG (tagged) the VLANS on your trunks (trk 1-24) to the distribution switches, and on the distribution back to the core. Otherwise only local traffic on the untagged ports will flow on the core.
                hp2910al-24g:(Vlan 10)# tagged Trk1

8) Set a static route to your routers IP, (Replacing 10.1.0.1 with your routers IP.)
                hp2910al-24g:#ip route 0.0.0.0 0.0.0.0 10.1.0.1

9) Set a static route on the ASA back to your core switch: (Where 10.0.0.0 255.0.0.0 is your inside subnet and 10.1.0.254 is the core switch. My router is plugged in to VLAN 10, which is 10.1.0.0- this must match! Your routers interal IP must be on the same subnet as the core switches VLAN IP.)
                ciscoasa5505:#route Inside 10.0.0.0 255.0.0.0 10.1.0.254

10) ALWAYS use the IP of the VLAN as the DHCP default gateway- otherwise nothing will work!
                Example: Vlan 20- IP 10.1.1.254
                                 xptestbox:# ipconfig -a
                                                     IP:10.1.1.100
                                                     Subnet: 255.255.255.0
                                                     Gateway: 10.1.1.254
                                                     DNS:10.1.0.2
11) Restart everything once the configs are made and SAVED.
                 hp2910al-24g:#wr mem
12) Enjoy your working network!

Example configs:

2910al-24g:

; J9145A Configuration Editor; Created on release #W.15.08.0012
; Ver #02:11.05:16
hostname "HP-E2910al-24G"
module 1 type j9145a
trunk 23-24 trk1 trunk
ip route 0.0.0.0 0.0.0.0 10.1.0.1
ip routing
snmp-server community "public" unrestricted
spanning-tree Trk1 priority 4
vlan 1
   name "DEFAULT_VLAN"
   no untagged 1-22
   tagged Trk1
   ip address 10.0.0.254 255.255.255.0
   exit
vlan 10
   name "VLAN10"
   untagged 1-10
   tagged Trk1
   ip address 10.1.0.254 255.255.255.0
   exit
vlan 20
   name "VLAN20"
   untagged 11-20
   tagged Trk1
   ip address 10.1.1.254 255.255.255.0
   ip helper-address 10.1.0.2
   exit
vlan 30
   name "Vlan30"
   tagged Trk1
   ip address 10.20.30.254 255.255.255.0
   ip helper-address 10.1.0.2
   exit
vlan 99
   name "VLAN99"
   untagged 21-22
   tagged Trk1
   ip address 10.1.99.254 255.255.255.0
   ip helper-address 10.1.0.2
   exit

2510G-24:
hostname "00005- 2510-24g"
trunk 23-24 Trk1 Trunk
ip default-gateway 10.0.0.254
snmp-server community "public" Unrestricted
vlan 1
   name "DEFAULT_VLAN"
   ip address 10.0.0.253 255.255.255.0
   tagged Trk1
   no untagged 1-22
   exit
vlan 10
   name "VLAN 10"
   tagged Trk1
   exit
vlan 20
   name "VLAN 20"
   tagged Trk1
   exit
vlan 99
   name "Vlan 99"
   tagged Trk1
   exit
vlan 30
   name "VLAN 30"
   untagged 1-22
   tagged Trk1
   exit
spanning-tree Trk1 priority 4

Cisco ASA 5505:

route 10.0.0.0 255.0.0.0 10.1.0.254



Hope this helps someone out there!

Long needed break, kind of.

As it turns out, I am a work-a-holic.
I was supposed to take most of this week off, but I ended up with a few projects that had to be done before I could break away. (Don't we all have this problem some weeks?). I managed to sneak out for Thursday and Firday, but despite my best effors- I swear- I ended up working from home with a flu. Not much of a break sadly and with the upcoming projects, this is probably as good as it is going to get. I am hoping that after our new Desktop support/Jr. Admin starts, well after I hire them, I can stop working 220+ hour months.

On to something fun:

I am getting to spec the hardware for our next system refresh. This is a beyond daugnting task, as our current cluster is based of Dual Xeon X5650s with 96gb ram and 256gb SSDs. All linked together by 40Gb's Infinband. I am planning on switch to E5 xeons, 256gb Ram, and 240gb Intel SSDs. The rest of the changes will all be in the configuration in hopes of giving us a performance boost. I have already had great sucess adding cache and log drives to our Nexanta/ZFS based SAN's.

Some notes for anyone else configuring a ASA 5505 with 8.4 and ASDM 6.4.
Check out both Sound Training and Pete Net Live on Youtube.

Pete's SSL VPN guide: LINK
And Site to Site VPN guide: LINK

Both were a great help and I highly recommend Sound Training's Cisco ASA book. I would wait until they have a refresh though, as 8.3/6.2 and 8.4/6.4 have some significant changes.

One fire at a time... Exchange 0x8004010f

I finished up our migration to Server 2008r2/Exchange 2010 this weekend and despite some pesky little issues, it has gone well.

One which came up was a Send /Receive error in outlook- Error 0x8004010f. Object not found.
After hours of searching, it tuned out this was a common error with nearly as many different solutions for reported issues. I finally found out that my Offline address book setting was blank!
Ogranization Configuration-> Mailbox -> Database Management-> Mailbox Database->*Rightclick Properties->Client settings
I just set it to the default and bam!
This resolved all our send/receive errors.



Oh and if you want to increase the default datastore size for all the mail boxes,
Ogranization Configuration-> Mailbox -> Database Management-> Mailbox Database->*Rightclick Properties-Limits

Here you can change the default storage limits.

Enabling Remote Desktop via a GPO and new job!

First the good news: I have joined the team at Urban Robotics as of Feb 12th as their new IT Manger. I can not wait to start.

Secondly:
A good guide I found for enabling remote desktip via a GPO.
LINK

Oh and for installing sharepoint 2010 foundations on server 2008r2!
LINK

Building a CCNA lab

I received a cisco 678 DSL modem right around when that famous clip on The Simpsons aired, showing a beatup "cisco" box, with many wires, and a old sleeping man in the corner. It was "The Internet". Ever since I have collected random Cisco gear to play with. It was years later I found out about the CCNA, and by that point the gear I had was useless. Recently I started to rebuild my lab with the goal of completing the CCNA, plus volunteering at a ewaste recycler helps :).

I found a great little blog HERE for getting started: Check it out.

My current lab is a bit of a mess, as I have gathered far to much gear for the CCNA.

Current gear:
Switches:
Cisco 3550
Cisco 3500
Cisco 2950-12
Cisco 2950-24
Cisco 2950T-24

Routers:
Pix 515e
CIsco 881w
Cisco 1760
Cisco 1841/256/64mg
Cisco 1841/128/32mg
Cisco 2610
Cisco 2620
Cisco 3620
Cisco 3640

AP:
1230G
Addon cards:
Wic 2T
3x Wic 1dsu T1
2x Wic 1dsu T1 V2
1FE2W
Async 32a with cable

I set my two 1841's and 2950T switch up as just a starter lab to give myself some practice. As I have more time I will build out the full lab once my new rack gets here.

ZFS Replication

Quick note for myself in the future:
ZFS Replication:LINK

Interviews and Server 2008r2 vm templates

I had a great interview with the folks over at Urban Robotics last week. They build custom 3d terrain mapping software, which is processed and stored on a custom cluster/ZFS box. The staff their is just amazing, all weekend I have had this just "too good to be true" feeling. Needless to say I can't wait to hear back from them :).

Tech notes: I recently had to rebuild my lab due to a series of hard drive failures. I was not happy, but it was my own fault in the end. (What do you mean skimping on a controller to pass through to my AIO ZFS VM was a bad idea? Haha, lesson learned- do NOT pass a bunch of discs through as individual data stores and then use them for ZFS. ESXi does NOT like this.) I decided to deploy a template for my Server 2008r2 VMs this time to make my life a little easier. In the mean time, I moved everything to my ZFS box and am awaiting hardware to replace the failed components.

This guide lays out basic template creation- I used it recently when setting up our production enviroment and was very happy with how well they laid everything out. Warning there is basic scripting involved :p.
You will want to decide what is needed in your environment. Example- I decided to not separate my page file, as I do not have a separate LUN for pagefiles in my environment.
Also- I found that when it came time to run sysprep, I could only get it to recognize my unattend.xml if I naviated to the c:\windows\system32\sysprep and then ran sysprep.exe /generalize /unattend:unattend.xml

Here's to a great week! :)

Ubuntu 11.10 with LAMP

I decided to finally get my website back up and decided to swap back to Ubuntu/LAMP. I ran a simmilar WAMP/Server 2003 box last time in the sake of trying WAMP, but I was never happy with the overhead  Windows required.

I found this great guide to installing Ubuntu 11.10. HERE

The install was fast, simple, and worked "out of the box". Mind you, it will need heavy tweaking, but for my personal use, it should work great. Can't wait to get Wordpress loaded up.

Installing Exchange Server 2010 on Server 2008r2

I recently setup Exchange Server 2010 sp1 on one of our Server 2008r2 boxes. It went fairly smoothly after adding myself to the Enterprise, Schema, and Domain Admin groups. Don't forget to install the Microsoft Office Filter Packs HERE. Once installed return to the exchange setup and get any critical updates.

Enjoy!

Here is the guide I used: LINK

Here is a quick guide for setting up a SSL cert for OWA:
LINK

Here is a quick guide for setting up a http to https redirect rule in the firewall:
LINK

Home network layout.

I have been cleaning up my home network layout and wanted to toss it quickly in to Visio. Check it out below.

Setting up Link Aggregation in Solaris 11

Now that you have built that awesome ZFS box, your going to want to be able to access it as fast as possible right? So you do some research, find out that with a managed switch you can combine physical ports to increase your network speed. Link Aggregation does just this and Solaris has its own method. Now mind you, this is really only beneficial where you have multiple machines trying to access the same server at once. Unless you want to put a quad port nic in your desktop, but that's a whole other project :).

Here is a great guide!
http://blog.allanglesit.com/2011/03/solaris-11-network-configuration-advanced/

Mirroring your boot drive in Solaris 11

As part of building a handful of ZFS boxes lately, I have been gathering small helpful tricks or well written guides. Here is another great guide on mirroring your boot drives in Solaris 11.

http://constantin.glez.de/blog/2011/03/how-set-zfs-root-pool-mirror-oracle-solaris-11-express

ZFS SAN Build: Part 1 the hardware

Before putting a ZFS SAN in to production for our new VM environment, I wanted to build a proof of concept server. If you have no idea what ZFS is, where have you been? Check out Sun's offical documentation on it HERE! Or over at Serve the Home
Or over at Hard Forum
Or over at Anandtech

Why ZFS?
- Data security due to real data checksums, no raid-write hole problem, online scrubbing against silent corruption.
- Nearly unlimited datasnaps + systemsnaps and writable clones without initial space consumption due to copy on write.
- Easyness, it just works and is stable without special raid-controller or complex setup.
- It scales to petabyte size with off the shelf hardware.
- It is FREE.

Let me repeat the two most important parts- It is FREE and it just WORKS. You can build and configure a 20TB+ system in a weekend , minimal headaches, and be ready to start enjoying all the great features of ZFS before you could even build a similar Raid 5 array. 

When planning the ZFS SAN, I had the following things in mind:
-Reliable hardware
-Off the shelf parts
-Easy to service

Lets start with the chassis:
There are a dozen great chassis which can fit a number of different users requirements. Norco and Supermicro are two great choices. On a budget? Norco. 24 bays for a $300 price range? Can't beat that for home use! Looking for redundancy? Supermicro, you can not built the quality for the price point.
I went with neither. Originally I was going to use the Supermicro 4u 24 bay hot swap. HERE It has redundant 900w or 1200w powersupplies, great build quality, and a price point around $1000 shipped. It was perfect- until I found this monster at a local ewaste recycler:

34bay Hotswap SATA chassis with 4 redundant powersupplies for $200

It was originally designed for a dual opteron with PCI-X raid controllers. This chassis was around $5k with no drives back in its day. It even worked. It fit every requirement and helped take a significant chunk out of the budget. The one down side? It required custom cables for the backplane- SFF-8470 to SFF-8087. These cost a cool $300... It was weeks after I bought it I found this out, and honestly, I should have bought the supermicro. No one makes rails for this monster anymore either. Lesson 1 learned.

The motherboard, processor, and ram:
I wanted something stable and cheap. I have had great luck with supermicro motherboards. I found this board open box on Newegg for $200 off: Supermicro X8DAH+-F-O. It includes some very nice features like ipmi, dual processors, 256gb ram supported, tons of PCIe 16x, 8x, 4x slots, and nearly half off! It uses cheap, unregistered ecc at around $40 per 4gb stick.

Supermicro X8DAH+-F-O
16gb Kingston DDR3 ECC
(1)Xeon e5420 2.5 Quad

Controller cards:
I wanted to go with LSI from the start and keeping in mind being able to provide the best bandwitdh possible, I chose the following:
Two LSI SAS9201-16i controllers
These two cards use 4x 8087 internal connectors and provide up to 430,000 I/Os per second when connected to the 8x PCIe bus. I choose to use two controllers instead of a expander due to concerns over the single 8087 linking the controller to host.



Hard drives:
2x 250gb WD hard drives (Mirrored rpool)
20 1tb WD RE3 hard drives
2x Intel 320 80gb ZIL drives
2x Intel 320 160gb L2ARC cache drive


Now I know using RE3 drives defeats the purpose of using cheap hard drives, but at $65/e brand new off ebay, these were hard to beat. RE3 are raid class enterprise drives designed for 24x7x365 and have a long proven track record.

The ZIL drives are simply for the ZIL log, setup in a mirror this should create a fast log store, which is important for performance in ZFS.
The L2ARC drives keep the most commonly accessed files and setup in a raid-0, these should provide a super fast store for 320gb of the most accessed data.




Networking:
2x Quad Port Intel Nics
With building this fast of a machine, I wanted to provide a solid connection to the SAN network. Two Quad nics allows me to provide two 4gb's LCAP links to two individual switches. Combined with each servers two dual port gigabit cards, each server has a super fast, redundant back end for storage traffic.

The Network is Failing!

Last summer I was faced with a interesting challenge. My old work brought me back to replace their aging servers which I originally installed in 2006. The old setup was barely keeping up with the business as it grew. Two Dell PE2900's served as the primary SBS2003 and Fileserver. Their primary storage array was comprised of 320gb Western Digital hard drives in a Raid 5. This worked great for about 3 years until the average file size grew over 350mb and the engineering staff doubled. Needless to say they also outgrew SBS2003 and have been painfully living with the now 6yr old install still grinding away.

I had to find a new solution which fit the following criteria:
Powerful
Stable
Expandable
and on a shoe string budget...

I had recently built out a $100k virtual environment for another client and this seemed like quite the challenge. I wanted to give them the same redundancy, performance, and expandability that I have previously found in commercially available solutions.

Step 1: Virtualize it. vSphere 5
I am a big virtualization fan and I can think of no better way to provide a powerful, stable, and expandable environment. As they grow, they can move away from the limitations of the free version and easily setup a HA environment. The hardware is fairly common:
- Two Dell R710's , Dual Quad e5620, SD card, dual powersupplies, 3yr warranty.
- 32gb ram each (upgradable to 256gb each)
- No local storage
- 4 Quad port Intel 1000 nics

Step 2: We need storage! ZFS SAN
I starting working with ZFS just over two years ago after reading about it while researching a home fileserver build. What started in my esxi lab soon moved to a 5tb fileserver to replace my original WHS, and even then I was never able to fully take advantage of the great features of ZFS.
Benifits of ZFS:
-Built by Sun and included in OpenIndiana 151a- the latest release from the Illumos project.
-Software like Raid with no write hole and incredible speeds.
-Very inexpensive to deploy, no special hardware.
-All raid sets supported (though named differently)
-Data encryption, file system level snap shots, and de-duplication.

After seeing the success stories of sites like www.zfsbuild.com and watching Gea's Napp-It web interface evolve, I decided it was worth setting up a test system for a proof of concept. My own personal success with it was a great starting point, but I wanted to really test the speed and redundancy of ZFS. We need stability and redundancy here, as this could easily be a single point of failure.

Step 3: Networking- Tie it all together!
This is where it gets fun. I planned originally on a basic two network design, separating the SAN and LAN network. Due to the projected growth and the ability to easily expand, I decided to add a third. The end design includes a third network, dedicated to management. This allows me to keep the traffic away from the end user, and keep the end user away from my management interfaces. Nothing like having someone find a password list and causing hours of damage for "curiosities sake". (It happens more then any IT person cares to admit.) Plus I now am free to use those spare IP's for more end users. Due to the budget, I went with HP 1800-24g switches. These are a low cost, web managed, gigabit switch that supports vlans, trunking, and LCAP.

Network layout:
- x.x.20.0 Management
- x.x.15.0 LAN
- 10.254.11.0 SAN

Step 4: VPN! Working from home has never been so sweet!
Theoretically this should fall under networking, but I have been super impressed with OpenVPN lately. Years ago I setup a small server to VPN in to my home network. The server crashed, I never rebuilt it, and moved to using other hardware based VPN clients for offices or the SBS included quick connection. This worked ok, but I was never really happy with it. Since they will most likely split to a multiple office setup here soon and I want to co-locate a backup server or two, I needed something slightly more robust which would support IPSEC and Road Warrior style connections. I have ran PFsense at home for a number of years and with the latest 2.0 release, setting up a secure VPN could not be easier.

This is just a very brief overview and I look forward to documenting each step.


Here are some quick pictures of the finished setup:

Static IP's and Solaris 11

No matter how many times I setup up static IP's on my Solaris 11 express or Open Indiana 151a installs, I always come across different problems! Almost every guide has you do it differently and I wanted to post what finally worked for me. This is from late last summer, but I found it relevant when trying to resetup my Napp-IT ZFS All-in-one on ESXi tonight.

First up fire up terminal and switch to root.

Next up disable network auto magic:

# svcadm disable nwam
# svcadm enable network/physical:default

Check the physical adapter state and note the LINK name: 

# dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
bge0         Ethernet             unknown    1000   full      bge0 

Check the logical adapter state: 

# ipadm show-if
IFNAME     STATE    CURRENT      PERSISTENT
lo0        ok       -m-v------46 --- 

Create a logical adapter:

# ipadm create-if bge0
# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
bge0        phys      1500   up       --         --
# ipadm show-if
IFNAME     STATE    CURRENT      PERSISTENT
lo0        ok       -m-v------46 ---
bge0       down     bm--------46 -46

Assign the adapter a static ip of 172.16.1.10/16. 

#ipadm create-addr -T static -a 172.16.1.10 bge0/v4
# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
bge0/v4           static   ok           172.16.1.10/16
lo0/v6            static   ok           ::1/128 

Now verify the existing routing tables:

# netstat -r

Routing Table: IPv4
Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
solaris              solaris              UH        2          0 lo0
172.16.0.0        172.16.1.1      U         3          1 bge0

Routing Table: IPv6
Destination/Mask            Gateway                   Flags Ref   Use    If
--------------------------- --------------------------- ----- --- ------- -----
solaris                     solaris                     UH      2       4 lo0

Add a persitant default route:

# route -p add default 172.16.1.1
add net default: gateway 172.16.1.1
add persistent net default: gateway 172.16.1.1

Now your routing table should look like this:

# netstat -r

Routing Table: IPv4
Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
default              fw01.allanglesit.net UG        2      10466
solaris              solaris              UH        2         12 lo0
172.16.0.0        172.16.1.1      U         6       1810 bge0

Routing Table: IPv6
Destination/Mask            Gateway                   Flags Ref   Use    If
--------------------------- --------------------------- ----- --- ------- -----
solaris                     solaris                     UH      2     156 lo0 

Go ahead and check your resolv.conf:

# cat /etc/resolv.conf
domain  allanglesit.net
nameserver  172.16.1.1

# dig www.google.com 

Now reboot and enjoy your new static IP :)

This has been modified from a guide here: LINK

Busy! Removing default password complexity policy in Server 2008R2

Wow it's January. 2011 flew by. It seems like 2011 was just one project after another! Things around our house have kept me super busy, with the holidays and all- but I really need to spend some time to update this blog with my ZFS fileserver builds.

Quick note: Ever built a small test environment and wanted to disable the annoying password complexity requirements/age in server 2008R2? Here is how:



When setting up a new Windows Server 2008 server either with or without Active Directory you will discover that it has a rather strong policy for passwords.  If you are setting this up at home or in a small business environment and don't want to deal with the complex passwords that are required to meet the policy guidelines, you can edit the policy to disable the complexity requirements.  You can try going to a command prompt and typing 'gpedit.msc' then navigating to Computer Settings\Windows Settings\Security Settings\Account Policies\Password Policy\ section.
Here you will see the 'Password must meet complexity requirements' item.  When viewing the properties of it, usually the Enabled/Disabled radio buttons will be grayed out and you cannot change the values.  If they are able to be changed, go ahead and do it, and save out of the dialog boxes.  If it is grayed out and you cannot change it here, this is how you do it:

1. Go to a command prompt
2. Type 'secedit /export /cfg c:\local.cfg' and hit enter
3. Using notepad, edit c:\local.cfg
4. Look for the line "PasswordComplexity = 1" and change it to "PasswordComplexity = 0"
5. You can also edit "MinimumPasswordLength = 7" to a lesser value if you like.
6. Save the file
7. At a command prompt type 'secedit /configure /db %windir%\security\local.sdb /cfg c:\local.cfg /areas SECURITYPOLICY
8. This will apply the new settings and refreshing the gpedit.msc should reflect the new settings
9. Set your new less complex password!

Shamelessly stolen from here : http://strangenut.com/blogs/clr-programming-blog/archive/2010/07/03/how-to-edit-the-password-policy-on-windows-server-2008-r2-with-active-directory.aspx