Maybe its time for you to take a bit more control, such as by blocking off those hard to reach areas, and maybe by restricting his space whilst he is inside so that he has to 'deal' with you which will hopefully help him start to feel more at ease when he's around you and you don't do anything he perceives as bad.