I can be on Desktop, immersive IE, or another app and get a notification of email from here about something random. I can click on that and view it, or I can just swipe over to the Desktop where Outlook is already open and I can already start reading the email, then open IE on the Desktop (as that's the default action in my case) and respond to something. If I needed to open Photoshop to edit something, I swipe open the Charms bar then hit Start (with a Microsoft Touch Mouse) and scroll a bit over to the Photoshop tile. Opens on the Desktop. If I need to view a photo, hit the lower left corner to Start, hit the Pictures tile that opens again, on the Desktop with File Explorer. I go to Start twice in this scenario and only need it for a few seconds each time. Not even close to being a big deal or annoyance. Unless if you have ADHD, then it would make sense...but even then....
Now I'm very used to the Start Screen and am using it like the Windows 95 model, empty Taskbar that is meant to switch tasks, a Start Screen meant to start something from a Desktop app, metro app, or File Explorer location.
Seeing as use Photoshop or AutoCAD or Programs and Features often, they're not the first thing I want to see when I go to Start. I open my group of apps daily except People, Calendar, or Photos so I want to see the live tile updates when I go to Start right away. I also use File Explorer locations daily as well, so it's what I want to see. If I have a program that I don't have pinned to Start, there is All apps I can go to and visually and quickly get to something with semantic view.
I don't have anything pinned to my Taskbar, as again, I have a Touch Mouse and I'm so used to using the Start Screen to launch things when I need them, it makes the Taskbar a little more effective I think. I find it reduces clutter when I had Outlook near the middle of the Taskbar, Photoshop in All Programs and would had been next to Task Manager at the end of the Taskbar, Explorer next to the Start button, and IE 9 next to Explorer. It's easier for me to switch tasks when the programs are clustered next to each other.
Then there's the fact that it's faster to open the programs themselves are there are larger targets to click on versus having to do a little more precise targeting.