The answer is truly Windows, though. It's about device enumeration in device manager. ADB Debug in system is *NOT* the same as ADB sideload. It's considered a separately enumerated device. But because it's literally the same hardware, Windows gets...."confused" from time to time. When you switch the USB port, it gets a different device address which Windows then goes "oh, hey, I should load a driver for that." It's freaking stupid.
Sounds similar to what happened when I connected my Taimen to my computer using a USB BT dongle - connected perfectly fine the first time - but after disconnecting, it refuses to connect again. It has gotten confused about the phone lol.
And this is what confused the $#!+ out of me. One day everything works fine, next day it all goes to pot. I'm seriously considering putting Ubuntu on my spare system and using that for future OTA flashes. At least Linux is somewhat readier to use.