Hmmm hadn't thought about that really - mine don't seem too worried about them and I don't expect the immediate neighbours to be partaking. I will probably cover them up well so they feel more secure and/or sit with them.
If you are bringing them in and then putting them out again I guess you need to even out the temperature a bit, turn off the heating or at least the radiator in that room. But I wouldn't have thought a few hours would hurt anyway.