Windows 8 and 8.1 Forums

Hotmail tips the scales

  1. #1

    Hotmail tips the scales

    If you take a look through Dick Craddock’s blog post from earlier this year, A short history of Hotmail, or A peek behind the scenes at Hotmail, by Arthur de Haan, you’ll see that Hotmail has experienced enormous growth since it started 15 years ago. Today it delivers a service that is localized to 59 markets, encompassing well over a billion inboxes that generate over two petabytes (2 million gigabytes) of new storage each month, and stores hundreds of petabytes in total. You’ll also see that through the years Hotmail’s back-end architecture has been upgraded and reworked multiple times to allow for this rapid growth and to incorporate advances in hardware. Given the operational challenges and trade-offs required to run a service of this size, developing and managing our systems at scale is second nature to the Hotmail team.

    In this video, Hotmail engineering team leads Dick Craddock, Mike Schackwitz, and Phil Smoot (me) discuss the major issues that arise from running such a large service.


    Next, let’s dig into how we run Hotmail and how we think about development at scale in general. 

    For starters, while some of the more than 360 million people who come to Hotmail each month may think about the web UI they see and use, there is a whole lot more going on behind the scenes to run this service.

    Under Hotmail’s hood

    Hotmail has lots of users, that’s a given. But with that comes the need to support diverse user scenarios.  People access Hotmail from just about every country on the planet, in dozens of languages, from multiple devices (phones; PCs), on multiple browsers running on multiple operating systems (Windows, Mac, Linux, Unix). And each month they add new accounts and contacts, create filters, share and download photos, import email from other services, and send and receive tens of billions of messages. For all of this, they look to Hotmail for a fast, secure and seamless experience. To make this happen, there are about 100 different services that Hotmail is running all the time.

    These 100 or so different service types run on tens of thousands of servers in data centers around the world.  From the ground up, they are grouped into different classes:

    • Manageability services let us operate the system with very little administrative support, the key to running a large number of servers.  Our design goal is a “self-healing” system, and our management services help us do this. They automate software deployment to our servers, monitor the health of our servers, automatically repair failing servers and rebalance the system as necessary, all without any human involvement.
    • Storage services uses tens of thousands of servers to store our users’ data, including 3-4 copies of each piece of data for redundancy and backup purposes. Our data systems rely on automatic replication, consistency, and fail-over algorithms to keep all of our data correct and always available.
    • Message delivery services deliver mail to and from Hotmail. Today we process over 8 billion messages a day with 2.5 billion messages being delivered to the inbox (the difference between those two numbers is primarily blocked spam).
    • Anti-abuse, safety and privacy services protect our users and identify spammers. Our spam prevention incorporates dozens of systems for filtering incoming mail, integrating third-party block- and safe-sender lists, recording what email users report as Junk, and handling end-point reputations systems (i.e., where did the message come from, or where is it directing the user?). Our anti-virus prevention removes viruses from the messages before they get to the user.
    • Data synchronization services exchange data between our services and user’s devices like PCs and phones. These include POP3, ActiveSync, and DeltaSync for synchronizing data with PC applications like Outlook and Windows Live Mail, and with mobile devices like Windows Mobile, Blackberry, and the iPhone. These also include services that exchange data with other internal data services like instant messaging, billing, and the Windows Live ID authentication systems.  We also support services that aggregate data from other email services and social networks like Twitter and Facebook.
    • Site maintenance services run in the back ground, cleaning up after the party. These include data warehouse services, which track feature usage, and system performance.  They include system garbage collection services that remove deleted and junk mail from the system, and load balancing services that ensure that storage, CPU, memory, and networking demands are distributed over the entire network of servers.
    • Application services like mail, calendar, contacts, and instant messaging are our web-based applications that consumers use directly, driving tens of billions of page views per month. These applications implement a variety of techniques including caching, geographical data placement, bandwidth detection, etc., to ensure that the application performance meets our goals in all markets.
    Hotmail clusters – how do we put all these services together?

    Hotmail uses what we call "clusters" to build out and manage all of these different services.  A cluster is a management unit of computation, storage, and memory caching servers grouped together in a network unit.  Clusters allow building and running the Hotmail operational system in a repeatable and predictable manner at ever-growing scale. The cluster contains everything necessary to manage and run a set of services for a set of users. This design provides good performance, as everything the user needs is in one place.  The design also minimizes the impact of system outages to only users on a cluster that might be experiencing a problem. Hotmail has hundreds of clusters, and is adding dozens per year to keep up with the needs of our users. Additional details on cluster architecture can be found at:

    Tips for developing at scale

    If you’re planning to start your own billion-user service, look no further!  The Hotmail developers have come up with their top five scaling suggestions – things to keep in mind when writing code for lots of users.

    1. Keep the overall design as simple as possible. The goal is to fail fast in simple ways. Over time, 80% of all work is maintenance done by people who didn’t invent the system, and often the system becomes so large the entire end-to-end experience cannot be understood.
    2. Remove all single points of failure from your designs. No single component failure should affect the performance or availability of your system. Then, plan and practice and test these failures as normal occurrences. When your service fails, don’t wake up administrators at night.  Ensure the system handles failure without human intervention. 
    3. Build in performance testing from day one.  Big composite systems spread over the earth can become expensive quickly.  Add in large distances and the constant speed of light, and performance can become very slow with just a simple extra network round-trip or additional disk IO.
    4. Automate everything. Humans don’t scale well, they are expensive, and make lots of mistakes.
    5. Isolate composite system failures.  When a neighbor system fails, back off so the neighbor can recover.  And make sure you don’t get tangled up in your neighbors mess.

    Phil Smoot
    Partner Development Manager, Windows Live Hotmail


      My System SpecsSystem Spec

  2. #2

    I remember when Hotmail was a baby. It was more like a spam trap than a valid e-mail account. It certainly has grown up into a respectable "adult" and my preferred e-mail service.
      My System SpecsSystem Spec

  3. #3

    Pembrokeshire, South Wales, UK
    Posts : 652
    Windows 7 Ultimate SP1 32 bit/ Windows 8.1 64 bit

    I still have the free version, I only got an account to use as my address when in the newsgroups many years ago. I do have some of the newsletters come in on it, the Microsoft ones, mainly security and the At Home ones, I get most of them through my RSS Feeds now though.
    I can't even remember when I last went to the site, I just have the account set up in Windows Live Mail and it's synchronised there.
      My System SpecsSystem Spec

  4. #4

    Tropical Island Pair a Dice
    Posts : 3,030
    Windows 8.1 Pro x64/ Windows 7 Ult x64

    Same here, haven't used Hotmail in many years, but still have an account, so I can get into Technet and a few other MS sites.
    Did use Windows live mail for a little while to gather email from a couple different accounts.
      My System SpecsSystem Spec

  5. #5

    Personally, I stopped using Hotmail years and years ago. Mainly because I have my own domain, so it was better for me to create my own personalized email address.

    When Gmail went into closed beta, I was fortunate enough to be one of the first few to play with it. I thoroughly enjoyed it and it became one of the many email addresses I still use to this day. Enter Google Apps and everything just came together. I could not only continue to use my personalized email address, but I could do so by using the the Gmail interface! I still occasionally sign into my Hotmail, but it's really only used for Windows Live Messenger at this point. When I even bother to sign into it of course.

    Although I don't use Hotmail...well...ever really, I can't really say I've been impressed by anything new within the last 10 years. It definitely looks different, but I haven't bothered to actually use/play with it in so long. Maybe I'll take a look at it sometime soon
      My System SpecsSystem Spec

Hotmail tips the scales
Related Threads
Manage the pop-up help tips in Software and Apps
Hi I installed Windows 8.1 Update 1 64-bit. What command should I use to remove a pop-up help tip shown when I run an app? If I wanted to eliminate the appearance of these Apps tips from every account, how should I do? Thanks
Tips for new user in General Support
Hi there! I have recently bought my very first Windows 8 touchscreen laptop. I have had a Macbook Air in the past 2 years and prior to that a Windows 7 desktop. I want a cheap and "familiar" computer which i can call my own. What i want to ask & discuss here are some tips & tricks on how to make...
I just installed Windows 8.1 in a virtual machine and I can't seem to get rid of this annoyance (see "Switch between apps" and the orange arrow on the left): 41823 I am not on a tablet and can't swipe from the side. All I can do is click on it but it comes right back and obscures my view....
I looked forward to opening 'publisher' for the first time to see if I could produce an item to fit into a magazine/newspaper. I made my initial box-size and then discovered that IT'S ALL IN IMPERIAL! Considering that most of the people in the world use Metric and don't mess up our engineering and...
Eight Forums Android App Eight Forums IOS App Follow us on Facebook