Should Hate Speech Still Be Free Speech?

How far does freedom of speech go? Recently, a New York cab driver was suspended from his job because he was wearing Nazi paraphernalia. Let me be clear: I do not agree with what this man was wearing, or what he stands for. But, he shouldn't be forced to hide his, although uninformed and ignorant, deeply held political views. After all, as a New York City cab driver, he cannot legally discriminate or turn down anyone who gets in his cab. Freedom of speech was created to protect unpopular opinions. I want him to have his rights, because it means I still have mine.