Yea, just don’t go drinking 1.2L of soy milk a day.
I think the best argument against a vegan diet though is that most people do it without education or planning and encounter nutritional gaps. B12, iron, protein, etc.
Secondary Hypogonadism due to Excessive Ingestion of Isoflavone in a Man - PMC
A 54-year-old man had been drinking approximately 1.2 L of soy milk (equivalent to approximately 310 mg of isoflavones) per day for the previous 3 ...