It’s not a “no one asked you”, but it is an idiom. As you can probably guess, it means more “don’t help people working against your interests”.

So, you’d say something like that to a pro-ICE minority person, for example.