Hotmail tips the scales

If you take a look through Dick Craddock’s blog post from earlier this year, A short history of Hotmail, or A peek behind the scenes at Hotmail, by Arthur de Haan, you’ll see that Hotmail has experienced enormous growth since it started 15 years ago. Today it delivers a service that is localized to 59 markets, encompassing well over a billion inboxes that generate over two petabytes (2 million gigabytes) of new storage each month, and stores hundreds of petabytes in total. You’ll also see that through the years Hotmail’s back-end architecture has been upgraded and reworked multiple times to allow for this rapid growth and to incorporate advances in hardware. Given the operational challenges and trade-offs required to run a service of this size, developing and managing our systems at scale is second nature to the Hotmail team.

In this video, Hotmail engineering team leads Dick Craddock, Mike Schackwitz, and Phil Smoot (me) discuss the major issues that arise from running such a large service.

njs.gif

 

Next, let’s dig into how we run Hotmail and how we think about development at scale in general. 

For starters, while some of the more than 360 million people who come to Hotmail each month may think about the web UI they see and use, there is a whole lot more going on behind the scenes to run this service.

Under Hotmail’s hood

Hotmail has lots of users, that’s a given. But with that comes the need to support diverse user scenarios.  People access Hotmail from just about every country on the planet, in dozens of languages, from multiple devices (phones; PCs), on multiple browsers running on multiple operating systems (Windows, Mac, Linux, Unix). And each month they add new accounts and contacts, create filters, share and download photos, import email from other services, and send and receive tens of billions of messages. For all of this, they look to Hotmail for a fast, secure and seamless experience. To make this happen, there are about 100 different services that Hotmail is running all the time.

These 100 or so different service types run on tens of thousands of servers in data centers around the world.  From the ground up, they are grouped into different classes:


  • Manageability services let us operate the system with very little administrative support, the key to running a large number of servers.  Our design goal is a “self-healing” system, and our management services help us do this. They automate software deployment to our servers, monitor the health of our servers, automatically repair failing servers and rebalance the system as necessary, all without any human involvement.
  • Storage services uses tens of thousands of servers to store our users’ data, including 3-4 copies of each piece of data for redundancy and backup purposes. Our data systems rely on automatic replication, consistency, and fail-over algorithms to keep all of our data correct and always available.
  • Message delivery services deliver mail to and from Hotmail. Today we process over 8 billion messages a day with 2.5 billion messages being delivered to the inbox (the difference between those two numbers is primarily blocked spam).
  • Anti-abuse, safety and privacy services protect our users and identify spammers. Our spam prevention incorporates dozens of systems for filtering incoming mail, integrating third-party block- and safe-sender lists, recording what email users report as Junk, and handling end-point reputations systems (i.e., where did the message come from, or where is it directing the user?). Our anti-virus prevention removes viruses from the messages before they get to the user.
  • Data synchronization services exchange data between our services and user’s devices like PCs and phones. These include POP3, ActiveSync, and DeltaSync for synchronizing data with PC applications like Outlook and Windows Live Mail, and with mobile devices like Windows Mobile, Blackberry, and the iPhone. These also include services that exchange data with other internal data services like instant messaging, billing, and the Windows Live ID authentication systems.  We also support services that aggregate data from other email services and social networks like Twitter and Facebook.
  • Site maintenance services run in the back ground, cleaning up after the party. These include data warehouse services, which track feature usage, and system performance.  They include system garbage collection services that remove deleted and junk mail from the system, and load balancing services that ensure that storage, CPU, memory, and networking demands are distributed over the entire network of servers.
  • Application services like mail, calendar, contacts, and instant messaging are our web-based applications that consumers use directly, driving tens of billions of page views per month. These applications implement a variety of techniques including caching, geographical data placement, bandwidth detection, etc., to ensure that the application performance meets our goals in all markets.
Hotmail clusters – how do we put all these services together?



Hotmail uses what we call "clusters" to build out and manage all of these different services.  A cluster is a management unit of computation, storage, and memory caching servers grouped together in a network unit.  Clusters allow building and running the Hotmail operational system in a repeatable and predictable manner at ever-growing scale. The cluster contains everything necessary to manage and run a set of services for a set of users. This design provides good performance, as everything the user needs is in one place.  The design also minimizes the impact of system outages to only users on a cluster that might be experiencing a problem. Hotmail has hundreds of clusters, and is adding dozens per year to keep up with the needs of our users. Additional details on cluster architecture can be found at:


Tips for developing at scale

If you’re planning to start your own billion-user service, look no further!  The Hotmail developers have come up with their top five scaling suggestions – things to keep in mind when writing code for lots of users.

  1. Keep the overall design as simple as possible. The goal is to fail fast in simple ways. Over time, 80% of all work is maintenance done by people who didn’t invent the system, and often the system becomes so large the entire end-to-end experience cannot be understood.
  2. Remove all single points of failure from your designs. No single component failure should affect the performance or availability of your system. Then, plan and practice and test these failures as normal occurrences. When your service fails, don’t wake up administrators at night.  Ensure the system handles failure without human intervention. 
  3. Build in performance testing from day one.  Big composite systems spread over the earth can become expensive quickly.  Add in large distances and the constant speed of light, and performance can become very slow with just a simple extra network round-trip or additional disk IO.
  4. Automate everything. Humans don’t scale well, they are expensive, and make lots of mistakes.
  5. Isolate composite system failures.  When a neighbor system fails, back off so the neighbor can recover.  And make sure you don’t get tangled up in your neighbors mess.
Phil Smoot
Partner Development Manager, Windows Live Hotmail


aggbug.aspx

More...
 
I remember when Hotmail was a baby. It was more like a spam trap than a valid e-mail account. It certainly has grown up into a respectable "adult" and my preferred e-mail service.
 

My Computer

I still have the free version, I only got an account to use as my address when in the newsgroups many years ago. I do have some of the newsletters come in on it, the Microsoft ones, mainly security and the At Home ones, I get most of them through my RSS Feeds now though.
I can't even remember when I last went to the site, I just have the account set up in Windows Live Mail and it's synchronised there.
 

My Computer

System One

  • OS
    Windows 7 Ultimate SP1 32 bit/ Windows 8.1 64 bit
    Computer type
    PC/Desktop
    System Manufacturer/Model
    Zoostorm/ Asus K55A Notebook
    CPU
    Intel Core 2 Duo E4600 @ 2400 MHz
    Motherboard
    Foxconn 45CMX/45GMX/45CMX-K
    Memory
    2048 MB (2 x 1024 DDR2-SDRAM)/8GB DDR3
    Graphics Card(s)
    Intel Corp 82945G Integrated Graphics Controller
    Sound Card
    Onboard Realtek
    Monitor(s) Displays
    Hanns.G HH221 22" Widescreen
    Screen Resolution
    1920 x 1080
    Hard Drives
    ExcelStor (250GB)
    Keyboard
    Microsoft Natural Ergonomic Keyboard 4000 (UK)
    Mouse
    Microsoft Optical Wheel Mouse
    Internet Speed
    Just under 4MB download it's ADSL.
    Other Info
    Epson Stylus SX415 All-in-one Printer,
    Seagate Expansion 500GB External Desktop Drive
Same here, haven't used Hotmail in many years, but still have an account, so I can get into Technet and a few other MS sites.
Did use Windows live mail for a little while to gather email from a couple different accounts.
 

My Computer

System One

  • OS
    Windows 8.1 Pro x64/ Windows 7 Ult x64
    Computer type
    PC/Desktop
    System Manufacturer/Model
    76~2.0
    CPU
    Intel Core i5-3570K 4.6GHz
    Motherboard
    GIGABYTE GA-Z77X UD3H f18
    Memory
    8GB (2X4GB) DDR3 1600 CORSAIR Vengeance CL8 1.5v
    Graphics Card(s)
    Sapphire HD 7770 Vapor-X 1GB DDR5
    Sound Card
    Onboard VIA VT2021
    Monitor(s) Displays
    22" LCD Dell SP2208WFP
    Screen Resolution
    1680x1050
    Hard Drives
    Samaung 840Pro 128GB, Seagate 500GB SATA2 7200rpm 32mb, Seagate 1TB SATA2 7200rpm 32mb,
    PSU
    Corsair HX650W
    Case
    Cooler Master Storm Scout
    Cooling
    Corsair H80 w/Noctua NF P12 12cm fan, case fans 2X14cm
    Keyboard
    Logitech Wave
    Mouse
    CM Sentinel
    Internet Speed
    Abysmal
    Browser
    Opera Next
    Other Info
    Dell Venue 8Pro: Baytrail Z3740D, 2GB Ram, 64GB HDD, 8" IPS Display 1280 x 800, Active Stylus.
    Haswell laptop: HP Envy 17t-j, i7-4700MQ, GeForce 740M 2GB DDR3, 17.3" Full HD 1920x1080, 16GB RAM, Samsung 840 Pro 128GB, 1TB Hitachi 7200 HDD,
    Desktop: eSATA ports,
    External eSATA Seagate 500GB SATA2 7200rpm,
Personally, I stopped using Hotmail years and years ago. Mainly because I have my own domain, so it was better for me to create my own personalized email address.

When Gmail went into closed beta, I was fortunate enough to be one of the first few to play with it. I thoroughly enjoyed it and it became one of the many email addresses I still use to this day. Enter Google Apps and everything just came together. I could not only continue to use my personalized email address, but I could do so by using the the Gmail interface! I still occasionally sign into my Hotmail, but it's really only used for Windows Live Messenger at this point. When I even bother to sign into it of course.

Although I don't use Hotmail...well...ever really, I can't really say I've been impressed by anything new within the last 10 years. It definitely looks different, but I haven't bothered to actually use/play with it in so long. Maybe I'll take a look at it sometime soon :)
 

My Computer

Back
Top