I woulnd't call it "holding a grudge", it's more like that they get cautious very fast. Had to medicate a doe (eyedrops) which absolutly hates being picked up, on the second day it was already a bloody (my blood) fight to get her out of the hutch, after the 2 weeks it needed half a year for her to forget about it and demand noserubs again.
Accidents didn't matter that much, they really do resent it when they know you do it on purpose, and have kind of second sight when you're up to it again.
Another thing is when they provoke an action, like refusing to go into the hutch at nightfall, first I try to bribe them, then I shoo them, and after running around the garden with flashlight and a long stick, cursing like hell (yes, I do think my neighbours somewhat worry..), for an hour I finally lose my temper and catch them. They don't really hold a grudge for that, they know they had it comming.
It's true, we are just mammals like rabbits, since I have them I can cope better with upsetting behaviour of some people.