I think you'll like the drive of the E36 a little bit better than the E46, but don't interpret this to mean the E46 has a "bad" drive. The E46 is a larger car and you do notice it, just as the E39 is larger still.
I think you will notice a definite drop in power from an E36 328i to a E46 320i. Given a choice of the two, I'd go for the 328i. I own an E46 330xi.
Why not "jazz" up your car to make it feel new again. How about turbo or super charging it. You can get the interior spray painted (the seats if they are leather) to make them look brand new. You can get a deep polish done on your exterior to refreshen its looks. Maybe swap out some lights for some new ones.