I know this is old but for anyone else, I had the exact same problem and this seems to be why:
My backup drive was on a laptop that's often shutdown or goes to sleep. Also I had put a really big file (80MB) on my desktop.
When File History was run it would get to the big file and slowly work...