And Smith said it came up when he met Prince Harry at a polo match.
Speaking on The Today Show in the US, the star said: “I met Harry once, at polo, which sounds a bit grand, but it wasn’t that grand.
“And he rocked up to me and went ‘Granddad’.”
Smith also shared that he’d previously met King Charles III and that he had complimented the now monarch on his red shoes.
"I told him that I though he had fabulous shoes, which he told me he'd had for 30 years," he said.
"They were red, they were red brogues, they were beautiful."
But he said that the Duke of Edinburgh "definitely didn’t".
Asked how he knew that, the actor said that a friend of his had once sat next to the Duke at a dinner and had asked him if he ever tuned in to The Crown.
"Apparently he turned round and said, 'Don’t be ridiculous’," he said.
Smith, who now stars in House Of The Dragon, played Prince Philip in the first two series of the popular period drama.
Claire Foy starred alongside him as the Queen.
Watch: Claire Foy felt 'very honoured' to play Queen Elizabeth