Dramatising a novel is one thing, rewriting historical fact to pander to a particular mindset is another. None of the Seymours or Boleyns were anything other than white Caucasian in character. Yes there were people of colour in London at the time, probably some at Court, but they were not part of the dynastic manoeuverings within the Tudor palaces.
What if The Crown had cast a non-white actor as one of the Queen's relatives ?