A persons gender is either male or female as determined by their chromosome structure. There are disorders which can cause one or more of the pathways to
physical appearence to not reflect that gender (usually men who end up looking like women) such as dysgenic pseudo-hermaphroditism. In the end though, you must have either XX or XY chromosomes, no wiggle room there. As for people who identify as the opposite gender, that's a psychological issue with sociological factors than biological one.
Within society and the issues you mentioned Ibram, those decisions should fall back on genetics. If someone thinks they're a woman but is a man, then they should mark 'male' on forms and such. The only exception to this rule I would think of would be the bathroom issue, WAY easier to just use whichever one people identify you as at a glance than to try to explain things to everyone and deal with the trouble everyday.
(Plus if we indoctrinated people in the other opinion I know plenty of guys who would just take that as open season on women's restrooms
)