The main problem with the gravel is that the rabbits may find it uncomfortable to sit / stand on, and there may be issues with their hocks or toes due to uneven balance and wear on joints, etc. A more solid surface provides a more natural balance position for them, for sitting and running on. Long term issues with sore hocks and long nails can lead to other issues with hips and spine (eg arthritis). I wouldn't think that a small patch of gravel would be an issue, but they should be mainly on a more stable substrate.
Grass will (eventually) grow through gravel without much soil as long as it is open to the rain and sun. Any area that isn't open to the elements can be covered in old paving slabs or wood planks (maybe bedded on sand if needed) - as long as the rabbits can't get their legs trapped in any gaps and the structure doesn't move. I imagine that removing several tonnes of gravel is going to be hard work, and there's no treason why it can't just stay as a base for something else. It will also make it harder to dig through for the rabbits - and provide extra protection from predators digging in.