Of course you can brush them
Was the person who told you someone who shows? I know some rabbits exhibitors don't use brushes on their rabbits as it could thin their coat, but I assume you are not planning on showing your rabbit. The plucking method is great for moulting rabbits