You have this idea in the early 20th century with eugenicists and white supremacist thinkers saying that racial intermarriage will cause the destruction of the white race. These theories go back further than that even and there is always this implicit statement that, if we allow minorities to take over they will do to us the same thing we did to them. But of course they are not going to say that explicitly, because they are also explicitly denying doing anything wrong. So this position is inconsistent, but that does not stop the far right from holding this position.