I fixed everything.
After I confirmed issue was fixed over VPN, it just clicked in my head.
Here is what I believe is/was the issue.
MY internet connection is PPPoE, usually this has a 1452 byte MSS because of the technical makeup of PPPoE, but there is a workaround to use 1460 byte MSS when using 1508 byte MTU with jumbo frames, which is what I have been doing for months on my router.
After the VPN test worked, I made a post on my isp's forum blaming them but right after I made the post it clickedm, and I disabled the VPN, then disabled the jumbo frames and 1460 MSS overide, so now I am back on a 1492 byte MTU and 1452 byte MSS to match PPPoE spec and I just tried to download the IE11 update and it came down fast first time, so the reason looks like was my network configuration.