[ZenGum]

Here's my 1.63 cents (converted from 2 Australian cents):

positive spin:
I note that he mainly does this in public. I recall reading psychological studies that found when a man and woman are together in a public place, the woman spends much more time (70-80% or even more) looking directly at the man and the man spends much less time (25-50%) looking directly at the woman. When not looking at the man, the woman generally looks at other nearby things - food and drink, etc), but the man will frequently glance around at the surroundings.
This was the observations they made. There are various tempting interpretations to put on it, of the hunter-warrior-always-on-alert-to-defend-the-clan theory, but this is a bit of a leap and not needed for today's point, which is:
It is normal for there to be a substantial difference in looking at your partner.

A positive interpretation of it is that he trusts you more than anything else, and is comfortable with you, and is naturally disposed to protect you and have you protecting him (since we are vulnerable from the back).

But, negative spin:
He's not just looking around, but turning away. That's different. The interpretation that he wants more distance between you is tempting. But even if that is correct, that doesn't mean he wants to leave. He just likes a bit more space.

Confused spin. Does he do it much more in public than in private? Interesting, not consistent with the negative theory above.

Practical suggestion: flash your boobs at him occasionally. That'll stop his eyes wandering.