I would just like to point out that fighting for justice does not automatically equal to giving someone punishment. Fighting for justice also can mean that you defend your views (or possibly even the objective truth) with clear arguments and proof - although admittedly this could feel like a punishment for the recipient of the lecture.
In this situation, 'letting it be' could translate to letting the other person spouting their generalistic/biased/prejudiced opinions while convincing yourself that you won't be able to convince *them*, except possibly to changing the topic of the conversation. Depending on the person and the encounter, you may not even have to deal with them (and their way of thinking) again; so "why even bother".