You can pull off the yellow and damaged leaves, I don't worry if they have a bit of brown 'rust' on them (perhaps I should?)
Do make sure they are from trees that have not been sprayed or treated in any way.
I don't wash mine but they are from my garden.