I agree with the above. It all depends on their backgrounds which you aren't sure of. He could have been really badly treated which would be why he is still nervous.
I'm inclined to think you need to define boundaries. After having a few nervous bunnies - I have learned that they need somewhere that is "safe" and is their retreat. Usually rabbits define this area as their hutch. When humans go in to pick them up - they feel a bit invaded, and that their hutch is not their sanctuary - as at anytime a human can arrive to pick them up.
I would make sure you only go to his hutch for positive reasons - food, hay, treat, a nose rub if he allows. Try not to physically pick him up out of the hutch. Also make sure he is away from the hutch at cleaning time. The hutch is their home and they need to feel safe in their home/space. Not saying that you make him feel unsafe - but you don't know what happened to him in his "previous life". It could have been children poking at him in his hutch or grabbing at him etc. As kids are not always gentle or backwards in coming forwards!
All of my nervous buns are only picked up for health checks and vet visits. They are let out to play without being picked up and go back in by being called. They get fed etc with nose rubs galore and I bend down and let them give kisses if they want to. I sit on the floor and let them use me as a human bouncy castle if they want to! I gradually built up to this - at first I didn't touch them just let them sniff/butt my hand. I decided they would let me know when they wanted stroking etc.
I think he just needs learn to trust you - it can take time but is worth it - I cried :shock: when one of my nervous buns flopped down next to me and let me fuss him for the first time
You could put a chair next to his hutch and just sit and talk to him with the door open and some hand feeding?
Have you thuoght about a rescue bunnny "wife" if you have the time/money etc as this helps nervous buns relax and come out of their shells. And there is no better company than another bunny to snuggle with. :wave: