From what I understand, these restrictions only apply to if you have been provided the software.

Correct.

Red Hat is under no legal obligation to supply you with their software,

Correct. I am not a Red Hat customer so they do not have to provide me with anything.

nor to continue doing so if you violate their terms.

This is also correct, but here is where the problem lies. If they provide a customer with binaries and accompanying GPL-licensed sources, along with a customer agreement which says “if you distribute these sources then we will terminate your customer agreement,” then that is in violation of the GPL. RHEL contains code from many projects (the vast majority I would think) that Red Hat does not own the copyright for. Red Hat has a license to use that software under the terms of the GPL. The GPL requires that they make the sources available of any binaries they distribute without further restrictions which limit the freedoms granted by the GPL. One of those freedoms is the freedom to redistribute the source code. Their customers who receive RHEL source packages must be given that freedom for Red Hat to be in compliance with the GPL license.

I agree this makes them total scumbags, but as far as I understand the GPL they are not breaking the law.

Whether they are “breaking the law” depends on jurisdiction and laws around copyright infringement, but this would certainly be handled as a civil matter. Projects which provide software to Red Hat under the GPL would be within their rights to ask a civil court to enforce the terms of the license or to mandate that Red Hat cease using their software until they comply. See the recent Stockfish vs. Chessbase case that was settled in Germany: https://stockfishchess.org/blog/2022/chessbase-stockfish-agreement/

In that case, the infringing party was not complying with any terms of the GPL at all, but the same procedure would be relevant in the case of Red Hat vs. any other FOSS project that wants to enforce the terms of the GPL.