I think it's just a matter of what you are used to. I know some people who stopped drinking milk, and now they can't stand it. I think the body loses the ability to digest the milk (and milk fat in whole milk) once you stop drinking it.