Leonid S. Knyshov

In the spirit of Eric Ligman, who always comes up with crazy ways to get customers with fewer than 5 employees enrolled into Open Value volume license agreements (did you hear the one about using $8 licenses for Office Mobile to open an Open Business agreement http://blogs.msdn.com/mssmallbiz/archive/2010/03/18/microsoft-open-license-basics-the-8-way-into-microsoft-volume-licensing.aspx), I decided to illustrate some of my creative ways to save customers lots of money when deploying servers.

I’ve been thinking how to save my customers the most amount of money when buying servers.

Traditionally, as a consultant I am expected to do this:

1. Login to my Dell, HP, or IBM account
2. Configure a server for my client’s needs
3. Sell them the brand new server
4. Most consultants include an OEM Windows Server 2008 license with the server
5. Some consultants also buy the CALs at the same time

Do you have any idea how expensive servers really are? Yes, it makes sense to get massively overpowered servers for some organizations (I own a couple of x366 quad-xeon 8GB and 6x73GB 15K RPM SAS monster servers from IBM, for example, that run approximately 50 virtual machines simultaneously) that actually need that kind of capacity for their workloads, but if you are a smaller shop, why should I sell you a $4000 server if a much less expensive solution is available?

After all, and regardless of the size of your business, all I really need in your server is

• Ability to support 6 or more hot swap hard drives (or at least 4)
• At least 2 RAID arrays
• Redundant hot swap power supplies
• Have battery-backed RAID cache
• Have at least 2GB of RAM (4-8GB for SBS)
• I couldn’t care less about its 64-bit CPU capacity.
• Oh, and I need it run Windows 2003 or Windows 2008 Server.

Note here that hot swap hard drives are mandatory. I refuse to support servers without hot swap drives. Do you know how long it takes to swap a failed drive on a CERC controller that is not hot swappable? Try at least 30 minutes. You also have to figure out which drive it is, power down and pull out the server (technically, the server CAN remain online as the SATA spec is written to include hot swap ability for all drives, but most consultants don’t know that or err on the side of caution when they do), figure out which drive it is, curse at the drive cage (multiple times).

No, I just don’t do that.

If a drive is not user swappable, and it fails, and then the other drive fails, that’s the scenario that makes data recovery companies salivate. I charged a customer $2000 a couple of weeks ago for just such a scenario. Believe me, I was being nice here. It was a piece of cake for me to recover because fortunately the customer was saved by an overzealous RAID controller that shut down the server before real damage was done, but it cost the customer a lot of money.

First rule of working with 24/7 servers – hard drives fail every day. If they can’t be swapped out easily without interrupting the server, they will get ignored. Most of such servers are not configured correctly to report HDD failures anyway. Once the second drive dies, things grind to a halt and you are at the mercy of a data recovery company.

Hot swappable drives come with visual indicators. Green = good, orange = bad. Cabled drives sit silently inside the case until they fry and die.

When a drive fails on a real enterprise-class server chassis, it can be swapped in a matter of seconds while the server remains in production. This is no-brainer information that’s making datacenter admins yawn really wide right now.

Well, from Dell today this configuration requires a minimum of T410 chassis. It comes with an integrated RAID controller known as SAS 6/iR which sounds great in theory and supports both RAID 0 and RAID 1.

I read the datasheet on this controller and it does not have battery-backed cache. Written in black on white is

“Dell SAS 6/iR controllers are designed as a costeffective,entry-level solution for basic RAID-0(integrated striping) and RAID-1 (integrated mirroring)functionality; they support only internaldisks and do not provide the performance advantagesof cache or battery backup. Administratorscan choose between these controllers and the DellPowerEdge Expandable RAID Controller (PERC) 6 family to help optimize RAID configuration andmanagement in their environments.1″

Why is that important? The purpose of this cache is to prevent data loss in the event of abrupt loss of power. If your server is on a UPS and the UPS is sized correcly and the power does not fluctuate in excess of the UPS’s capacity, then you are in great shape. The rest of you will suffer some sort of data loss when data stored in the RAID controller’s cache, which is often up to 512MB, goes *poof* with the server. That’s 512MB of data that you think, and so does the operating system, is safely on-disk but is in reality sitting in the memory of your RAID controller. I am jumping a little ahead of myself, but a database writes data to transaction log first and the actual database later. Based on that, how likely do you think it is to lose transactions not yet committed to the database in the event of power failure. Yep.

In order to get the real RAID controller with real battery-backed cache, which in Dell’s case is PERC 6/E, I have to add it. That’s a $599 option for the 256MB cache and $649 option for the 512MB cache model. I want you to take a special note of this number as I’ll come back to it later.

By the way, I hate most implementations of RAID5. Many consultants consider it to be almost the panacea, slap a 3-drive RAID5 into a box and call it protected against a single HDD failure. You won’t see it in use on my servers unless I have very good reasons to implement it.

So far I am up to about $3500 with (6) hard drives (500GB drives), (1) dual-core Xeon E5502, and (4)GB of RAM. This server started at $779, by the way.

Oh, and that price does not include tax, I can continue, but I think it’s obvious this is a $4000 server already, so I’ll stop here for now.

Wait a second, we didn’t add the operating system! Win server 2008 std OEM is $799.

OK, so now our server is $4264. A similarly configured T610 is actually slightly less expensive, but that’s all Dell marketing and really doesn’t matter much in this case.

Here you go, Mr. Customer, your new server will cost you about $5000 after taxes and shipping.

By the way, the OEM license you purchased is stuck to this server. I am not going to get into virtualization issues related to OEM licenses. We’ll talk about that later.

Then there is the matter of CALs. They are roughly $40/user, so not really that big of a deal. Or so you think.

Now let’s examine how we can meet this same requirement for a lot less.

I need to build a network with a domain controller, a server, and a web server

If I don’t want to experience network interruptions, I need at least two domain controllers/global catalogs.

My minimum config for a DC is very very simple. Why is it simple? The only thing that server does is run the NTFRS service. Since I have it anyway, I usually make it a backup server as well.

My minimum configuration for a primary file server is as above. Why? I have a couple of drives for the OS and transaction logs and the rest for my data. In no event do I let both co-exist on the same array. I want to be able to swap out the OS drive and leave the data array intact. Can you imagine an easier server OS release upgrade than pulling out two drives and putting in two new ones? Didn’t think so.

I love the new SAS controllers. Did you know you can save some serious money with a SAS controller? RAID 1 on SAS for system. RAID 1 on SAS for recent data and databases. RAID 1 on SATA using 2TB volumes for the older data. That’s 6 drives, 3 arrays, and pretty much no chance that anything will go wrong.

In the days of SCSI servers, this was not possible and you had to rely on extremely expensive SCSI drives, but SAS controllers let us mix and match SAS and SATA all day long.

Let’s take a closer look at my requirements. Do I really need today’s generation of server equipment? The truth is, CPU speeds are basically stuck. We are adding cores, but we aren’t increasing per-core performance in a way that means much for SMB workloads.

So here is how I buy servers today.

1. I buy CALs and enroll them in software assurance. That’s mandatory because as soon as you add a new release of a Windows Server, your older CALs are no longer valid to access that server. You didn’t know that, right? However, CALs under SA are indeed sufficient.
2. I buy used servers with OEM licenses on the chassis. Remember I said I don’t like OEM licensing in general? That’s true, but I can buy a dual-CPU server with 4GB of RAM, a couple of hard drives, AND Windows Server 2003 Standard OEM license for about $350. All…day…long! These are typically rack form factor Dell boxes like the 2850. They were the work horses of the corporate datacenter, so they typically come configured to the teeth with PERC cards, dual CPUs etc.
3. Remember I said I need an extra domain controller and don’t really care what runs it? I can buy a 1U server for less than $300 with Windows 2003 Standard OEM license.

But what about hardware warranty? What about it? These servers are so cheap that you can have more than one chassis just sitting there. Swap in the drives and you are back in business.

Want some ridiculously undervalued servers? Buy a server with a Windows 2003 Enterprise license attached to the chassis. That thing is licensed to run up to 4VMs of that operating system on that chassis.

I save my customers thousands of dollars by taking advantage of this information. There is no rule against taking out PERC cards and putting them into new servers, for example.

While I should be very excited about the Windows Server 2008 Foundation R2, I am not really that excited about it on reflection. It’s an OEM license, which means it must be attached to a new server sale. In practical terms, that means I’d configure a new server from a super OEM such as Dell, HP, or IBM, and then sell it to you. It’s a restricted server and indeed is very attractively priced. If you don’t know about your other options, it’s a steal. $279 for up to 15 users is indeed a steal because it costs about $1399 for 15 users (including the 5 built-in users) + OEM license of Win 2008 Std.

However, I can also buy an enterprise-class server chassis with an unrestricted Windows Server 2003 Standard and 15 CALs in software assurance for far less than $1000. Which do you think is a better value if you don’t particularly care about Windows 2008 features and just need an extra domain controller/backup server?

In summary, by buying used servers you can get the feature set normally reachable at $4000 level for less than $400, which is less than the cost of a PERC card by itself. Any new licenses you buy should be bought through Microsoft volume licensing. It’s OK to buy OEM licenses, just be sure to move them into your volume licensing agreement before 90 days expire from your original purchase from the original OEM. In case of Microsoft office, it’s pretty pointless to buy them OEM.

My preference for servers these days tends to be something like the Dell PowerEdge 2850 or HP Proliant DL380 G4. Ask the seller to make sure COA for Server 2003 or 2008 is included. You are basically buying that little sticker and the rest of the server is free. There are tons of older servers out there with Server 2000 that are practically free, but that’s a tricky case. Server 2000 has reached the end of its product support cycle as of July 2010. On the other hand, if you just need an incredibly cheap domain controller, it might not be an entirely awful idea. Note that Exchange 2007 doesn’t like Windows 2000 domain controllers.

By the way, Dell seriously overcharges for its drives. Buy the drive carriers separately on eBay and put your own drives in them. If you use RAID arrays, you don’t really care when the drives will die.

For cheap storage, I usually add an external drives chassis. Feeding the SCSI appetite of enterprise servers is expensive. In most cases, I don’t need that much performance for everything except my SQL and Exchange boxes.

I personally buy my servers racks at a time. My preference is for SAS, but SCSI will do. There is an abundance of powerful servers with OEM licenses out there. My managed customers run on them very happily. If I have any catastrophic failures, such as a motherboard failure, I just have them swap the drives into a spare chassis. This is in a small business environment with non-technical users. Their server is back up and running in less than 30 minutes.

Isn’t that just plain awesome?

Well, it’s not awesome if you charge your customers for on-site visits. My business model is a flat rate helpdesk. The less time I spend per problem, the more profitable my day is.

Filed under:
microsoft by Leonid S. Knyshov