Its both. Replacing the factory speakers will make a tremendous difference, but to see the greatest change you would need to replace the HU and bypass/replace the factory amp. Replacing/Bypassing is a last resort though. You should be fine doing speakers and HU.
<span style='color:gray'><span style='font-family:Courier'>1995 e36 3 Series RIP
1999 e46 3 Series BURN IN HELL
2000 MR-S Roadster= Current</span></span>