I don't know what a boy house is? :? do you mean your boyfriend's house?
But taking rabbits on journeys and changing the environment can be stressful for them, so not something you should do unless necessary (e.g. vet trip or bonding). They are a prey species, and that means that they have to know their territory perfectly to ensure their survival, so completely changing that environment is stressful, especially when you are a baby, and your instinct is telling you that you need to be in a group to survive, and yet you are alone. You'll likely find that he either hides, or goes to explore - the exploring being finding the possible risks and bolt-holes of the new environment.