It's actually very common for babies to leave mum at 6 weeks (Although a lot of people are leaving litters with mum for longer now), but if you take them away from mum and then rehome them straight away it can cause problems (Such as a weaker immune system), you should give them time to get used to life without mum, but keep them with their siblings and in the same hutch.
Harley came from a very irresponsible breeder who was giving away babies (to children, for free!) at 6 weeks old, to make room for another litter that had just been born :roll: He is the hardiest, healthiest rabbit I've ever met though.