Did your computer have Windows 7, and do an upgrade directly from Windows 7? That sometimes will cause problems as some programs and hardware may not have been fully compatible with Windows 8/8.1 when you upgraded.
I see your system specs, and it should drive Windows 8 just fine, even if you shut off the fast boot.
What I would do first, is see if a service, or startup program could be causing the problem. Sometimes a running program can cause a very slow restart.
To disable non-Microsoft services
1. Open the charms bar on the right side
type "msconfig" in the search box then open it.
Click on the "Services" Tab
Click "Hide all Microsoft Services"
Disable all the services listed.
2. To disable all startup programs
Right click on the task bar, Click Task Manager
Click on the startup tab
Click on each startup program, and click on Disable at the bottom right side of task manager.
During both of them, it will prompt you to restart your computer.
The first restart may be slow as all the services are currently running.
Once the computer is booted up, login, try to restart the computer again, to check restart speed. If your restart speed is faster, then you know that it was a program or service causing the slow restart.
If it doesn't solve the problem, then you may have to do a Refresh, or a full Windows reinstallation.
Although, what I would do in this case, is if you have another spare hard drive handy, Install a clean copy of Windows 8 "without activating windows" just for testing purposes. Check and see if the restart is still slow, or if it is fast. If it is a normal fast restart, then maybe a reinstallation of Windows may be a good option, as something is affecting startup, shutdown and restart on your current system.