He's probably just into doing his own thing. I have two buns, one will sit for hours and let me stroke him and cuddle him as long as its on the floor, the other will scarper as soon as I touch her, I cant handle her at all. Its not even like shes scared of me, she'll come when I call and follow me about, she'll climb on me and even sniff my hand - but if I touch her she'll run off and I know I'll have to make it up to her for her to trust me again.
Its all about learning what your bunny likes, it sounds like he doesnt mind being stroked too much but maybe you're getting in the way of his adventures! You can try stroking him when he is eating, although watch he doesnt have food territory issues, thats a good way to lose a hand! Just take things at his pace, he may become a cuddle-bun, but many dont. Just make sure its on his terms.