Aww sweatie please dont feel disheartened Jims is a very wriggly bun and has been since we adopted him in Feb 09 even the vets have trouble sometimes. We can not pick him up. But he is the most affectionate bunny i know! He follows me around, plays like a puppy, jumps all over me and loves nothing more that to snuggle on the floor or bed, he snuggles right in to me and licks my nose. Obviously i would LOVE to be able to cuddle him but i think because trying to pick him up stresses him out so much it is probably best not to?
In regards to the taking him out of his cage i agree with the other posts, bunnies need a place they can call their own so he probably perseves you as 'invaiding' his space when you try to grab him, we have Jims cage on the floor so he can go in and out as he pleases and if he is in his cage try to leave him alone as much as possible and let him come to us.
Just a thought, have you ever thought about having him in the house as a house bun it really brings out their personalities.