One thing that I've always thought about Fraser is that he isn't really aware of how standoff-ish his walls make him. Sure, he can blush and cough his way out of Frannie's advances, but he even rebuffs Ray's innocent (?) attempts at friendship, without even realizing it really. That, to me, just screams baggage. Thus, this little scene of his sad realization...