I'd suggest having a cue to tell him you are about to do something so he knows in advance it's you and what you are going to do. For example I always click my tongue (that noise you make at cats) before turning on the kitchen light where Scamp lives. He knows that noise means the light is about to flash on which might be a bit startling but no need to panic.
I imagine he is reacting to movement, he's not stopping around to see what's making the movement just bolting for cover. So he's not specifically scared of you just he has obviously learnt that sometimes movement results in something scary
By giving him a signal he can learn to differentiate it out a bit so he knows when you give him the signal there is about to be movement that isn't a cause for alarm.
The signal can be anything, a particular phrases, a noise a couple of light taps on the door before you go in. Something loud enough for him to hear but not loud enough to worry him.
You can do the same thing to signal you're about to make any movement, for example you might have a noise that means 'I'm just about to get up'. I have a general 'I'm just about to do something I know is scary and weird but don't worry it's not anything to worry about' which I use for opening the dishwasher (which he hates) and when a car pulls up outside (which he stamps at and runs if he's sat in the window). I can actually stop Scamp mid bolt if I make it when something startles him - he's generally very confident and out going, but he's a wild rabbit I handreared so he's got the instinct to bolt for cover when surprised.
Bolts holes are very important, it's sounds like you've got them covered but make sure he always has ready access and that will give him more confidence.
Never walk straight up to him, meander and aim to one side. Walking up head on is very rude and intimidating in rabbit speak. If you're giving nose rubs, avoid flattening him ears or blocking his sightline, he'll be using those to keep a watch for danger so if you inadvertently stop him he'll be less likely to relax and more likely to move away. You might like to try cheek rubs instead
If he'll come up to you when you are lying down I would use that as a teaching aid. Lie down, use a cue to say I'm going to move, move your hand slightly (not towards him) and then offer a treat. Although he'll move away at first, wait for him to come back. The idea being he gradually comes back quicker and quicker and eventually learns, the signal means you move and he gets a treat and he doesn't leave at all
As he gains more confidence you can make the movement bigger and eventually towards him.
I'd recommend using his regularly dry food portion as treats, so you don't get a podgy bunny!