Microsoft.com: What’s the story?
If you’ve ever wondered how microsoft.com uses their technology then you’ll be interested in this post from Jeff Alexander an IT Pro Evangelist for Microsoft Australia. Though unfortunately Jeff’s blog seems to be down at the moment or better said nonexistent as visiting his blog will get the following message (The forum you requested does not exist.), I’ve still got a cached copy of the post so for the time being you can read the main points below thanks to Jeff and his information gathering from the people over at the Operations team at Microsoft.com.
At this point we still don’t use firewalls for MS.COM sites and don’t have any plans on the books to put them in place. Here is the short answer as to why:
- We don’t handle HBI data so we don’t have the need for external logging capabilities. If we did handle HBI, we’d have firewalls.
- We have ~650GB/day of IIS logs just for www.microsoft.com and update.microsoft.com (not including the 6GB/hour for each download server). Just IIS logs are a challenge without trying to parse another ~650GB of firewall logs.
- 5+ years ago, there wasn’t a firewall solution that would scale to our needs and this forced us to focus on network, host, and application security. Based on the success of that work, we’ve not looked further at firewalls even though there are solutions that I believe (haven’t tested) would handled the traffic load (our non-download based web traffic alone can be in the 8-9 Gbps range and ~30 total for internal hosted traffic).
- We also used NLB for load balancing exclusively up until July 2006 and the micro segmentation of networks required by that solution made firewalls an expensive and very complex solution. Again, especially at the scalability that used to be available.
- Application security is critical since a firewall is likely going to allow traffic on the correct port and protocol through to the web servers so IIS/ASP.NET/Applications must deal with these requests gracefully. I realize there are other options/features of firewalls/IPS that provide other options.
In terms of how we protect the sites, we utilize (starting at the outside edge of the network and working in):
- Cisco Guards for DoS detection and automated response
- Router ACLs are in place to block unnecessary ports
- NetScalers for www.microsoft.com and MSDN/TechNet (NLB still for update.microsoft.com) and those also provide DoS protection inherently as well as providing a few other knobs we can turn when required.
- Windows and IIS…rock solid and secure! www.microsoft.com is on Windows Server 2008/IIS7, MSDN/TechNet are migrating to Win2k8/IIS7, and update.microsoft.com is on Windows Server 2003/IIS6. We do all the normal shut-off-unused-services practices that line up with MS published security guidance and we utilize GFS images to ensure standardized builds of systems.
- Automated Netmon/Perfmon captures for attack analysis on NLB systems when SYN floods occur (event trigger). We’ve not yet done this for NetScaler systems, but we are noodling on how in our copious spare time :).
- We do run AV on our servers when we can. At times product adoption means we don’t install it, but we do normally run AV.
- Application security as mentioned. ACE is very good resource for this aspect. ACE is an internal team that does threat modelling for applications.