I wasn't a bodybuilder or anything, but my metabolism is completely fucked because of high school sports.
Football (170) -> Wrestling (155) -> Baseball (165-170)
If I could go back in time, I would punch my wrestling coach in the face and then punch myself in the face.
I have to work really hard to maintain a certain weight, and I can't eat (or drink) like complete excrement without suffering the consequences...which sucks.
I'm not sure if you can necessarily attribute permanent change to that........
Might be natural decline of test levels, other factors at play. I mean yeah,It certainly doesn't help if you yo-yo dieted and of course that effects on the endocrine system as well in the short term but a lot of that stuff can be reversible over time. The biggest danger though when you're in HS though is your growth plates fusing prematurely, and that obviously is not reversible. Lack of nutrients and unnecessary fluctuations with your endocrine system can do that. I'm almost positive i'd be 2-3 inches taller if I didn't basically survive off french bread pizza, skittles and dr pepper for most of my high school years.
I'm willing to guess your situation has to do more with genetic factors than anything else. Some people have their T levels take a huge dump in their early to mid 20s out of nowhere. Of course there's the thyroid too..which can often be the culprit.