"they'd spend a hell of a lot of time having to defend those self-sufficient gardens from people who don't have a garden to mind"
Why is it bad that one should spend time in self-defense?
Yeah, self-defense might take one away from the smart phone apps, 'Dancing with the Stars', and vat o' Haagen Daz, but some things are worth workin' for...
As I say: folks would have to reorganize priorities, re-assert the difference between 'need' and 'want'.
As I say: it'll never happen.