"You're an extremist, a small minority; nobody agrees with your views!"
I would remind you that extremism in the defense of liberty is no vice! And let me remind you also that moderation in the pursuit of justice is no virtue! —Barry Goldwater
Ignoring the ad populum fallacy, "extremist" is often used as if it was some sort of negative, and as if by asserting it the one asserting has won some sort of point or argument. But translate it to what they usually mean by it—that the person "attacked" is consistently principled—and it can be seen that it is no insult at all. (DBR)
Related: Responses to charges of 'Extremism'.