I'm sorry to here about Fiver
It is usually recommended for rabbits to live in pairs, however my boy Sonic lived alone all his life, and was totally content
He sounds similar to Fluffer, any contact always had to be on his terms, otherwise he's get frustrated and be a pain to catch! I would have him out for hours at a time in the garden, and somedays he would totally ignore me :lol: My sisters guinea pig, Finn, became a single boy in Jan, and we've kept a very close eye on him to see if we should introduce a new friend for him, and similarly, we believe that he's actually more relaxed without Jake. I totally understand what you mean, it's really surprised us.
As for bringing him inside, I probably shouldn't advise as Sonic lived outdoors all his life too! He'd make a point of finding the patch where the wind and rain was getting through, and sit in that spot :roll: I would have loved him to be a house bunny, but he would have absolutely hated it! If you can, and he's happy, I would at least try start bringing him inside for a bit every so often?