I see young children playing and there is no prejudice. Then they grow up.
I remember the first time I realized that an old couple I knew were black. I was shocked at my own realization and the difference in my perception of them. Up till that moment, I would have sworn on a bible that I wasn't prejudiced.
From the sounds of it, you WEREN'T "prejudiced", because you hadn't "PRE-judged" them, and in fact, unless I'm mistaken, you haven't "judged" them at all! Your realization that they were "different" does not qualify as "prejudice", nor does it qualify as "racist", it merely means that you realized that they were "different" than you in some respect.
That's the whole problem with liberals attempts at indoctrinating people. They refuse to acknowledge that words really do have meaning, that they are powerful things, and as such need to be used properly. Today, they would have you convinced that the simple realization that someone is different than you makes you "prejudiced" or "racist", or that if you decide that you don't like someone or something that you're somehow bad because you've "discriminated". Well of COURSE you're discriminating! EVERYONE discriminates, every day of their lives, in some respect or another, but that doesn't make it a "bad" thing, regardless of what the libs will try to tell you.
This whole thing about calling people "racist" simply because they decide that they don't like someone of a different race is patently INSANE! Now, if you decide that you don't like ANYONE of a specific race, BECAUSE of their race, then that IS "racist", but deciding that you don't like them AFTER you've taken the time to find out something about them is NOT "racist", whether it takes you 2 years, or only 2 seconds. If you say that all of "those" people are bad, or ugly, or stupid, or "elitist", or whatever buzz word you care to use, without getting to know them first, then you ARE "prejudiced", but saying that about a particular individual is NOT "prejudiced", because you've obviously taken the time to analyze them to one degree or another PRIOR to making the statement.
All of that to say this, don't succumb to allowing the "PC Police" to set the rules of the game. They don't own you, they're not your parents, you don't owe them anything, and if you refuse to GIVE them to power to run your life, they'll NEVER HAVE IT! If you DO give them that power, then they DO own you, and you'll never be your own person again, EVER.