How much the hardware matters and how much the software matters depends on what you're doing.
* If you are using a computer to do tasks that are not resource demanding: such as browsing the web, checking e-mail, or typing up documents in office software, then you need a computer powerful enough to perform well enough for the operating system itself. Once a CPU is fast enough to render the desktop UI, a webpage, or a word document, it becomes moot to worry about upgrading it any further. In this case, you would only notice a speed increase by using a faster harddrive (or SSD) or even a RAID setup. This would speed up OS and application start time which is the speed you'll notice the most. Both Linux and Windows can do this very effectively, however MS Office does have more features and support than LibreOffice if you go far with it. Even an old Windows XP computer or a small netbook could handle this job exceptionally well.
* If you are using a computer to play demanding video games, convert or edit videos, design 3D models, manipulate high resolution photos, run virtual machines, or even do emulation: having an i7 over an i3 in this case, for instance, would be largely noticeable. Larger RAM capacities would also matter more here. This is also where your choice of graphics acceleration may come into play, especially with 1080p+ resolutions being prevalent in this department. In this situation, a modern version of Windows has a pretty significant advantage over Linux, though Linux HAS been used with proper knowledge of its tools, but it's not common. Windows 7/8 64-bit are almost a must nowadays for this.
* If you're running a simple server. One to hold your network's files, or even host a simple website... a low end machine running a Linux terminal may very well be all you ever need.
Of course there's in-betweens in all of this. But this is the very basic idea of how it boils down. I mean there are exceptions to all the rules, for instance someone mentioned the lack of DRM support in Linux, that even the average joe users who don't need a powerful machine may still not enjoy Linux. Some printer or some scanner may not work right, whatever. You always have to evaluate and plan ahead what you will do, and what you might end up doing when selecting hardware and software both.
As for me personally? I run Windows 7 64-bit here on my desktop (in my specs), but my laptop runs Windows 8 Pro 64-bit with Ubuntu 12.10 64-bit on dual boot. I reboot to whichever one suits my fancy at the time, and I have my uses for both.