When asked if everyone should avoid gluten Ray answered yes.
It does seem like some people are more sensitive to gluten than others but imo if you and your family are having issues it probably wouldn't hurt to try completely gluten free diet to see if it makes a difference.
This article seems to say gluten is good for you.
Low-gluten diet linked to heart attack risk - National Library of Medicine - PubMed Health