I had the same worry when my goats ate some, the vet called the London poisons unit and we were told it was toxic, including leaves, but they had never had any reportings of poisoning. We took the goats in for a couple of days and fed hay only, and Waffle was the only one to get severe diarrhoea on the 3rd day. She stayed bright within herself and was fine after a few days of pro-rumen (ruminant probiotic). I think really it is only a problem in large quantities in goats, but keep and eye on him and maybe give him some biolapis or fibreplex if you have any.
Hope he is OK.