Best of both
Mine have a shed and aviary outside now, which they are in when we are at work (which is 5 days a week full time) and our house is really too small, untidy and unsafe for unsupervised buns. Although our wires are protected, Sprocket has managed to jump onto a shelf before and I don't trust them to be in all day on their own. If we kept them in a cage indoors, they would have far less freedom than they get in their outside house. Whereas they can enjoy being outside, still have plenty of shelter from the elements and be safe now.
They get indoor time usually from 17.30 to 22.00 ish and more at weekends. This means that we get time to socialise with them, we can keep a closer eye on them and pick up on any illnesses etc and also supervise them inside. I have done it this way for years and have never noticed any ill-effects healthwise for the buns. Except now that we have S&W, they have both had snuffles this year (our first year of having them). I think they were probably predisposed to it, from what the vet said when we adopted them, having grown up in a petshop and having both had problems when they were younger.
I can't say whether bringing them in every night for a few hours does any harm. I know a lot of people have had buns with snuffles recently and a lot of those are house buns, so my feeling is that it probably doesn't help, but I also don't know if it makes it any worse. I do know that we are able to monitor them a lot more closely by bringing them in. I might also look at installing some heating in their aviary/kennel to reduce the extent of the temperature change, and if we ever get a conservatory I might think about moving them inside in the winter if they continue to have these problems.