It's fine. Lithgow is like Kelsey Grammer or John Huston - there's that kind of patrician aspect to their accent that allows them to get around an English accent. It's less obvious than say, someone like Timothy Spall, but then Spall is perhaps (oddly) the worst part of The King's Speech. It can't be too broad.
Obviously it's not the same mob making this but James Cromwell nailed playing Phil the Greek in The Queen. So there's a nice precedent.
And Jeremy Northam as an ineffectual bozo is perfect casting. As, of course, is Matt Smith as a pompous blowhard who says whatever he thinks and damns the consequences.
Harris seems like really odd casting as George VI though - someone so haggard playing someone so, well, not.