It’s like saying all white people are racist on account of the society they were born into

Yes. That’s exactly what we’re saying. The alternative is that people are inherently racist, which is worse.