White is just a label applied to a changing cast of sorts of people deemed worthy. People of Polish, Irish, Italian etc descent would have been considered non-white. In some folks whether they could be considered by others as white comes down to skin tone.

If you consider more Americans as defined by culture and nationality have a darker skin tone then you might be a racist.