Bringing outdoor rabbits inside to play

At the moment our buns are kept in the outhouse but when I get in from work at night I bring one of them inside until I go to bed to hop around in the front room and have some space and the other is let out of the hutch to have the run of the outhouse. (We alternate which one comes in because they've been fighting). Anyway I was reading today that you shouldn't bring outdoor rabbits in for more than an hour because when they go out again the sudden change in tempreture is too much of a shock for them? They're spend majority of the time in the outhouse so don't see that the tempreture should be a shock.

We bring them in at the weekends too so we can spend time with them so don't want to just leave them in the outhouse and not see much of them.

The temperature in an outhouse (unless it's heated?) will be vastly different to the temperature inside a house. I, personally, wouldn't even bring them inside at all during the winter when the temperature change is so drastic. It "might" not harm them but it can, and that possibility would be enough to make me not want to risk it.

I'm in a similar situation at the moment and it really is sucky. I'm in the process of going all out to insulate a shed so that it will be more comfortable for me to spend time in there with them, rather than the other way round.