Going to let the actual law have the last word on this:
A person who uses threatening, abusive or insulting words or behaviour, or displays any written material which is threatening, abusive or insulting, is guilty of an offence if—
(a) they intend thereby to stir up racial hatred, or
(b) having regard to all the circumstances racial hatred is likely to be stirred up thereby.
You are only focusing on (a) here. But (b) is important. It's racist speech if anybody who is NOT the speaker (because they're covered in (a)) feels like it is racist.
Again, if a black guy walks up to another black guy and uses the N word in an endearing or friendly way "hows things N..." or uses it in conversation somewhere, then he isn't guilty of either A or B, as it's near enough globally recognised as something that it fine amongst black people.
He hasn't intended to stir up racial hatred, nor do the circumstances make it likely that it will be.
Completely different to a non black person saying it.
The argument that black people shouldn't say the N word to each other sounds all a bit Ron Atkinson.