No you can't easily bypass your router and go PC to PC without it.
A router does DHCP, a standard switch does not. Without DHCP the PC's will not acquire an IP address and not be able to communicate with each other. With no device handling DHCP you would have to set each PC manually with static IP address etc.

It sounds like you need sort out why your new Router is under performing.