When we kicked off this blog 8 months ago, I talked about how we’d be responding to comments and as part of that gathering feedback from all of you. This blog is clearly an important channel for feedback, but we also gather a lot of other types of feedback, and I wanted to share more about how we bring this together and use all of this feedback to improve our products. This is particularly relevant right now as we just completed the rollout of Hotmail to our entire customer base, and we’re getting a lot of feedback on the beta of Windows Live Essentials (our suite of programs for your Windows PC). Both of these are great opportunities for us to hear from you and make improvements to Windows Live.

Balancing all the feedback

We have a responsibility when we design any product or service to be thoughtful about what we are trying to accomplish, the improvements we are making for customers, and the context behind those improvements. This is definitely a complex task as we have to weigh factors like how many customers will benefit, whether customers will get more benefit from one feature or another, the effect of added features on our release schedule, the impact of any change on customers who know how to use the existing products, and all the other tradeoffs that are made in the process of developing products.

We also need to balance competing requests from people who want more and more, against feedback from others who are happy with what we have today. And then there’s the simple fact that we just can’t do everything. Our goal is always going to be to create the best possible set of products and services for customers, but it’s always going to be a complicated equation.

Feedback comes from many places. First, we have the product telemetry that is built into our products, including data about setup failures, product crashes, script errors, and instrumentation that shows which services are being used how often. These provide a quantitative view of how you’re using our products and some of the issues that you’re encountering. Of course, any information we collect is governed by our privacy policy.

We also get a lot of input through the Windows Live Solution Centers (our support and help forums) – this is a mix of quantitative feedback (how many people hit a particular problem) and qualitative feedback (verbatim comments from people in the forums, and conversations with support professionals).

We also track issues reported through social media avenues (including this blog, Messenger, Facebook, and Twitter), and direct feedback from our MVPs and from customer research–focus groups, surveys, informal discussions with customers, and more.

Feedback can span a broad spectrum – from something that is simply not working right, to suggestions on how we have designed specific features; from new things you want us to create, to less tangible things like our branding. We have to bring this all together and determine a balanced response. This means we sometimes need to change our approach as we go along, and figure out how best to communicate that change. And even when we choose to make changes to a product– the change can be immediate, or it could be in a few weeks or months.

In Windows Live, figuring out how best to balance all the feedback we receive is quite important, because often we get contradictory feedback. One person’s “improvement” is another person’s “unnecessary feature.” A good example in the recent Hotmail update is the update itself – we had some customers who wanted the change “right away” and wondered when we would deploy, and others who wanted us “not to change anything.” Managing this conflicting feedback is part of designing software on a large scale. In the end, we try to find the right balance by picking the right features and changes, on the right schedule, with the right communication plan.

As we look at feedback, we find it falls into three general areas:

Things that aren’t working as planned

First, there are times when something that we built isn’t working as you (or we) expect. This could be that our product isn’t living up to what we intended to deliver, or it could simply be a bug or issue that we didn’t catch in our internal testing, or that only became visible when the product was released to the public.  In general, these are things that we want to make sure we fix. During a beta, we get a lot of feedback that falls in this category, and many of the issues are new issues which we only uncover through the beta process. These can be things that don’t get discovered until we have millions of people using our products, and that’s one of the important reasons that we release betas. As we make fixes, some of these will get packaged in a refresh to the beta, some will come in the final release, and others get fixed within our back-end infrastructure and don’t require you to download anything new. We measure our response to these reports by changes in the software that address the issues.

Things we would have liked to do

The next category of feedback is suggestions for features that are aligned with the goals we’ve set for the product, but, for one reason or another, we chose not to do. These are things we would have liked to do and probably thought about, but we chose to do other features first because we believed most of you would find them to be of higher value. These are often some of the hardest trade-offs to make because we are balancing different perspectives. Something that is critical for one person may be only nice to have for another. The good news is that we intend our value propositions to be enduring and so we will get to a number of these updates in the future. So we will consider these features and suggestions as we plan future updates to our software. We try to avoid promising features in next releases, and instead talk about new features when we have a clear plan to deliver them in our products.

Things we decided against

The last category of feedback is for feature requests that conflict with other feature requests, or that don’t fit into the larger goals of our product offering. We look over this type of feedback carefully, and when we know something does not fit into our product scope we do our best to make sure we’re communicating this clearly. While not everyone will agree with our decisions, we do try to be clear about our reasoning, and let you know what to expect.

Letting you know where we’re headed

Hopefully that provides a little more insight into the different types and categories of feedback we receive, how we use it, and how you should expect us to respond. While we cannot respond to every reported issue or suggestion, we do read through all the feedback and will use this blog to summarize what we’ve heard, and summarize any changes we’ve made. In subsequent blog posts we’ll cover some of the recent feedback we’ve been receiving on Hotmail, SkyDrive, Messenger, and the rest of Windows Live Essentials, and how we’re using this feedback to release additional updates to our products.

As always, please continue to send us your thoughts – here on our blog, in other social media and in our support forums. We won’t respond to every suggestion, but we’re definitely listening and bringing this into our discussions on what to build next.

- Chris Jones

Vice President, Windows Live Engineering

Technorati Tags: Windows Live,Windows Live Essentials,Hotmai,Messenger,Windows Live Solution Center,Feedback,Enthusiast,Consumer