I'll give you a personal anecdote of something that happened to me recently.
I work for a global company servicing many countries, so most of my coworkers are from all over the world. In my team of 9 I am the only white English person. There are about 100 people where I work.
One of the people I sit next to, let's call him 'G' (because I do!), comes from a small country in Africa called Benin that I had never heard of before I met him.
So one day G comes in moaning about the English weather pissing down and how he cut his hand on his umbrella. Well I thought it was hilarious. Laughing my head off I said "Of course he's cut his hand on his brolly - man's from Benin - he's never seen rain before!"
G got upset at that so I suggested we go chat somewhere private.
When we were alone I said "So how come you can say that in England it's always raining but I can't say that in Africa it never rains?"
He said to me "When you say something like that to me, I don't mind because we're friends. But when you say it in front of everyone I have to shut it down."
"Because if other people think it is ok to say stuff to me about being African, I guarantee one of them at some point will say the wrong thing and then we'll have a big problem."
So I learned something then. G's experience is so different to mine. Yes, there are some things certain people can 'get away with' (or however you want to put it) or put up with in different situations - but if we don't draw that line and say it is all totally unacceptable - people are going to get hurt.
So instead of "Oh it's ok to say that but it's not ok to say this?"
Lets cut out the confusion and just make a stand because that joke you like that you think is 'only mildly racial' - it's holding the door open for some real nasty shit.