He did actually turn his head away when Firminho said the word, so his attention wasn't there. There is also the chance Firminho mouthed it and didn't actually say it, so the ref wouldn't have heard it then either.
Or Firminho hasn't actually said it.
Personally I think there's more chance of a South American saying something like Negrito (as it's not seen as derogatory, similar to the Suarez situation) than there is of Holgate lieing about it. Holgate wouldn't have known JJKs head had blocked the camera.
There are many suggestions I suppose and I guess none of us will know what actually was said, I do think Madley is a bit of a twat though, just my opinion from watching him over numerous matches.