I have the same experience with drinking full-fat milk (4.2-4.5% fat). I get the "dairy opiate syndrome", grumpiness, low motivation, just feeling overall weird and like I want to go hide in my room all day.
However when I get raw milk and skim off all the fat, then boil it with ginger and cardamom I feel stronger, nourished and motivated. I think the devil is in the details when it comes to making dairy work!
edit: Also milk works best for me eaten on its own as a meal, no mixing with fruit.
Lol, described my symptoms to a T! Fortunately strong black coffee fixes this issue quick but if I keep drinking raw milk even several cups can send me down for full day.