I have the same problem. You should be able to turn off 'auto' and reduce the blower level to the minimum without the climate control taking over as far as I know and when I do this, I get the same symptoms as you. I'm wondering whether this could just be loose connections as my fans tend to cut out for a short time when I hit a bump in the road. I'm having it serviced by a dealer in the next couple of weeks so will see if they can advise.
Just had the car serviced at the dealer and, as predicted by other posts on the board, it was a faulty sensor (Apparently happens quite a bit. The part was £50 + fitting so not a cheap fix, but certainly cheaper than a new climate control unit. If you need the full name of the unit, let me know and I'll dig out the docs.