Since "they have more of a right to be here than us"