Internet Explorer Performance Lab: reliably measuring browser performa

In this post we take a look at something we all care very deeply about--as engineers and as end-users--real world web performance. We do a huge amount of work to get beyond the basics of anecdotes and feel as we work to build high performance web browsing. This post is authored by Matt Kotsenas, Jatinder Mann, and Jason Weber on the IE team, though performance is something that every single member of the team works on. --Steven

Web performance matters to everyone, and one engineering objective for Internet Explorer is to be the world’s fastest browser. To achieve this goal we need to reliably measure browser performance against the real world scenarios that matter to our customers. Over the last five years we designed and built the Internet Explorer Performance Lab, one of the world’s most sophisticated web performance measurement systems.

The IE Performance Lab collects reliable, accurate, and actionable data to inform decisions throughout the development cycle. We measure the performance of Internet Explorer 200 times daily, collecting over 5.7 million measurements and 480GB of runtime data each day. We understand the impact of every change to the product and ensure that Internet Explorer only gets faster. This blog post takes a deep look at how the IE Performance Lab is designed and how we use the lab to ensure we’re continually making the web faster.

In this post, we present:

Overview of the IE Performance Lab
Lab infrastructure
What (and how) we measure
Testing a scenario
Results investigation
Testing third-party software
Building a fast browser for users

Overview of the IE Performance Lab

In order to reliably measure web performance over time, a system needs to be able to reproducibly simulate real world user scenarios. In essence, our system needs to create a “mini version of the Internet.”

The IE Performance Lab is a private network completely sealed from both the public Internet and the Microsoft intranet network, and contains over 140 machines. The lab contains the key pieces of the real Internet, including web servers, DNS servers, routers, and network emulators, which simulate different customer connectivity scenarios.

Although this may appear complex at first glance, this approach allows all sources of variance to be removed. By controlling every aspect of the network, down to individual packet hops and latencies, our tests become deterministic and repeatable, which is critical to making the results actionable. In the IE Performance Lab, activity is measured with 100 nanosecond resolution.
2604.ieplrmbp_2D00_image1_2D00_hi_2D00_res_5F00_thumb_5F00_462149C4.png
Diagram shows content servers connected to Network emulators, connected to DNS servers, connected to Test clients, connected to Raw data storage, connected to Data analysis, connected to SQL database.

This type of network configuration also provides a great amount of flexibility. Because we’re simulating a real world setup, our lab can accommodate nearly any type of test machine or website content. The IE Performance Lab supports desktops, laptops, netbooks, and tablets with x86, x64, and ARM processors, all simultaneously. Similarly, because the lab uses the Windows Performance Tools (WPT), we can run the same tests using different web browsers, toolbars, anti-virus products, or other third-party software and directly compare the results.

WPT provides deep insight into the underlying hardware. Using WPT, we can capture everything from high-level CPU and GPU activity, to low-level information such as cache efficiency, networking statistics, memory usage patterns, and more. WPT allows us to measure and optimize performance across the stack to ensure that the hardware, device drivers, Windows operating system, and Internet Explorer are all efficiently optimized together.

A single test run takes 6 hours to complete and generates over 22GB of data during that time. This highly automated system is staffed by a small team that monitors operations, analyzes results, and develops new infrastructure features.
Lab infrastructure

The Performance Lab infrastructure can be broken into three main categories: Network and Server, Test Clients, and Analysis and Reporting. Each category is designed to minimize interaction across components, both to improve scalability of testing and to reduce the possibility of introducing noise into the lab environment.
1526.ieplrmbp_2D00_image2_5F00_53F38FBF.jpg
A large room full of computers

Here’s a view of the IE Performance Lab, including a number of test and analysis machines on our private network.
...

More..
 
Interesting, thanks :)

A Guy
 

My Computer

System One

  • OS
    Windows 10 Home x64
    Computer type
    PC/Desktop
    System Manufacturer/Model
    Custom
    CPU
    INTEL Core i5-750
    Motherboard
    ASUS P7P55D
    Memory
    KINGSTON HyperX Fury Black Series 8GB (2 x 4GB) 1866Mhz
    Graphics Card(s)
    EVGA GTX750
    Monitor(s) Displays
    LG 27MP33HQ 32" IPS LED
    Screen Resolution
    1920 x 1080
    Hard Drives
    Samsung 840 Evo 120 GB, 2 x SEAGATE 500GB Barracuda® 7200.12, SATA 3 Gb/s, 7200 RPM, 16MB cache
    PSU
    ANTEC TruePower New TP-550, 80 PLUS®, 550W
    Case
    ANTEC Three Hundred Illusion
    Cooling
    COOLER MASTER Hyper 212 Plus, 3 x 120mm 1 x 140mm Case
    Internet Speed
    20 + Mbps
    Browser
    Vivaldi
    Antivirus
    Avast
And the results are?????????
 

My Computer

System One

  • OS
    Windows 8 CP 64
    System Manufacturer/Model
    Me
    CPU
    AMD Athlon X2 2.4G
    Motherboard
    ASRock
    Memory
    6GS
    Graphics Card(s)
    ATI HD 5450
    Screen Resolution
    1280x1024 75hz
    Hard Drives
    WD Black 80g SATA3 x2 System/Backup
    Keyboard
    MS
    Mouse
    MS
    Internet Speed
    10 mgs
    Other Info
    This is just my test machine.

My Computer

System One

  • OS
    Windows 8.1 Pro
    Computer type
    PC/Desktop
    System Manufacturer/Model
    ASUS
    CPU
    AMD FX 8320
    Motherboard
    Crosshair V Formula-Z
    Memory
    16 gig DDR3
    Graphics Card(s)
    ASUS R9 270
    Screen Resolution
    1440x900
    Hard Drives
    1 TB Seagate Barracuda (starting to hate Seagate)
    x2 3 TB Toshibas
    Windows 8.1 is installed on a SanDisk Ultra Plus 256 GB
    PSU
    OCZ 500 watt
    Case
    A current work in progres as I'll be building the physical case myself. It shall be fantastic.
    Cooling
    Arctic Cooler with 3 heatpipes
    Keyboard
    Logitech K750 wireless solar powered keyboard
    Mouse
    Microsoft Touch Mouse
    Browser
    Internet Explorer 11
    Antivirus
    Windows Defender, but I might go back on KIS 2014
Back
Top