Yes, leave the prefetch folder alone. Actually that applies to the Windows folder and all of it's subfolders as well. Your policy should be a strict hands off unless you really understand what you are doing. That level of knowledge does not come easy.
The idea that routine "cleaning" of the prefetch folder was beneficial originated in the early days of XP. There were a number of perceived problems with prefetch that periodic deletion of the folders contents would solve. But it was all based on a misunderstanding of how prefetch actually worked which is quite different from how it is often described. Cleaning of the prefetch folder was a bad idea in XP and even more problematic in more recent versions of Windows.
Cleaning of the prefetch folder is a myth that just refuses to die. It was copied (as in copy and paste) from one tweaking website to another, most of whose authors had no real understanding of what they were talking about. The truth has been known for a long time but the myth can still be found in some relatively recent articles, and by authors who should know better.