Apparently my first guess was right, it was the BSM.
I had to use the old blower resistor, the new one was a bust and was sending voltage to the blower telling it to spin even when the car was not fully on, that was interesting.
The BSM on these cars have something like a resistor that fails and it stops sending voltage to the blower/resistor/etc, aka no power. Apparently they are glued in so you can't easily replace them, too expensive for a new one, junkyard was the best bet. Managed to do it myself, sorted it out, worked like a charm.

I read it on a few forums now so it must at least be an uncommon occurrence.
The symptoms are very similar to the blower resistor failing. You can still change the power of the fan/instruments work/etc, but nothing works. You may experience the blower working/not working a few times before it fully fails. However, the typical blower resistor normally still allows max setting to work before fully failing (not always, but very common). You can also splice two certain wires together in the BSM and the blower will work, though it's a temporary thing and I don't recommend it for a long period of time. (The two wires are here:
Again it is MORE for testing purposes and maybe if you need heat for a couple days to get to the junkyard/etc instead of just fully replacing the BSM. I would also highly advise doing it with a fuse too. It is at least a test to know if you need a new fuse box or not as the BSM is often a moderate price to pay.
I ended up leaving the splice in but removing the fuse, so if the issue ever occurs, I can plug in another wire. Since I had the car for 4 years, I probably don't think I'll need to do it again for the remaining life of the car.