(CNN) - This past week, my inbox blew up with e-mails asking whether Donald Sterling's First Amendment rights were violated in the uproar over the Los Angeles Clippers owner's racist remarks about black people. After all, he was simply expressing his views, however unpopular.

While he did have some rights violated, his First Amendment rights remain intact.

The First Amendment protects you from the government punishing you because of your speech. The NBA is a private club, and it can discipline Sterling all it wants.

What about the chorus of criticism? Are we all violating his First Amendment rights by criticizing him? We are punishing him for his speech.

Nope. The First Amendment does not insulate you from criticism. In fact, that's the First Amendment in action. That is how the marketplace of ideas works. We float our ideas in the marketplace, and we see which idea sells.

Most everyone would agree that Sterling's ideas fail in the marketplace of ideas. Nevertheless, I reluctantly stand on Sterling's side today. What happened to him may have been illegal and was morally wrong.

Start with illegal. In California, you can't record a conversation without the knowledge or consent of both parties. The recording featuring Sterling and V. Stiviano may be the result of a crime. Once she gathered this information, someone leaked it (she denies it was her) -- and it went viral. This is where I think things went morally wrong.

We all say things in private that we might not say in public. Sometimes we have ideas that are not fully developed -- we try them out with our closest friends. Consider it our test-marketplace of ideas. As our ideas develop, we consider whether to make them public. Should we not all have the freedom to make that choice on our own?

The Nevada rancher Cliven Bundy made his own stupid and bigoted statements, and he's been nationally pilloried, too -- but he chose to make those statements to the world. He deserves every ounce of obloquy heaped upon him.

But does Sterling? Think about what his public character execution means. It means that we now live in a world where if you have any views that are unpopular, you now not only need to fear saying them in public, but you need to fear saying them at all -- even to your intimate friends. They might be recording you, and then that recording may be spread across the Internet for everyone to hear.

Read more:

What happened to Sterling was morally wrong