It probably in total took me 4 years to recover from hypogonadism (verified by blood tests along with improved physical signs). Two of those years were trying other diets like BS paleo/keto protocols, which only made things worse. The other two years were following Peat. But I bought into the idea that I needed thyroid supplements to increase my temperatures even though my TSH and free T4/Free T3 were completely normal. I bought into the stop the thyroid madness crowd as well. Then I tried high liquid Peat for a while based off of people asking him what an ideal diet would be and he claimed 2 quart of milk/1 quart of OJ. People can say that other's take him out of context or whatever, which is partly true, but when you have a good percentage of people who follow Peat's work drinking copious amounts of fluids, you have to wonder if it is all just a misinterpretation or if some of what Peat "recommends" is really not optimal for some individuals.
Tailoring my diet to whole nutritious foods and fluids only when thirsty was the best thing I've ever done. Also focusing on sleep and sunlight. My point in writing about that it can take so long to recover has to do with the fact that we have a multitude of factors keeping us down and troubleshooting all of these/finding what works for us can take a long time. Often times it is not just one factor, but many factors that cause stress and poor nutritional status. Some examples: PUFA stores that can take years to deplete, our lifestyles are poor (too much working under fluorescent lighting and not enough sunlight), poor air quality, constantly conflicting evidence in nutrition that may make us sway wildly from one strategy to the other, etc... Oftentimes diet is not the #1 factor bringing somebody's testosterone levels down. I know a guy who had low T and tried everything, every kind of diet, etc. Simply quitting his corporate office job to work as a laborer and be outside all day tripled his testosterone levels, and he confirmed that his libido/function improved as well.
Every one of us has responded to the OP's thread with some good suggestions, but that doesn't mean that those suggestions will work for him. It is going to be up to his own experimentation and patience finding what works for him. And that can take time. This is why I think he has a lot of thinking to do for whether or not he should come off of TRT. Regardless of whether his diet is optimal or not, there are a number of reasons why it may take a long time for his levels to come back to normal.
Do you take thyroid supplements now?