Hi there and welcome to the forums!
The command should work if VBoxManage' folder is added in the PATH environment variable in Windows.
The other fix would be by using the whole path in command prompt, so running this should work:
This is where VBox is usually installed, change to where your VBox is installed if another folder is used.
"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" setextradata “Windows 8” CustomVideoMode1 1366x768x32
However the machine should be turned off when running the command above.
There is also a way for adding the resolutions entry manually in XML code in your VM .vbox file.
See this: VM Trouble (2)
So open your vbox file and look for <ExtraData>, under it add the resolution:
You can add any other resolutions following the above like this:
<ExtraDataItem name="CustomVideoMode1" value="1366x768x32"/>
name changes (incrementally) with each added line, this means that the name is unique per line.
<ExtraDataItem name="CustomVideoMode2" value="1280x720x32"/>
<ExtraDataItem name="CustomVideoMode3" value="1280x720x16"/>
<ExtraDataItem name="CustomVideoMode4" value="1920x1080x32"/>
... and so on.
VM must be turned off when modifying this file, you are limited to a max of 16 custom resolutions AND they don't work after you've installed the "Guest Additions" so keep that in mind when using this technique.