I wouldn't worry about it unless you've noticed it affects performance. Probably that memory can still be freed whenever some application needs it. I have 3GB ram and it uses 50%.
I guess you tried turning off Avast and Rainmeter?
Just a kind of guess but maybe you have some rogue shell extension installed that's causing Explorer to do too much work. Have you installed any low quality homemade programs that put themselves on the Explorer context menu or show fancy icons for their files?
I know this is old but for anyone else, I had the exact same problem and this seems to be why:
My backup drive was on a laptop that's often shutdown or goes to sleep. Also I had put a really big file (80MB) on my desktop.
When File History was run it would get to the big file and slowly work...