Hugh Bonneville, who plays the character, told Standard Online the aristocrat would not approve of the Conservative MP for “slouching” while Parliament was in session as “a gentleman of class wouldn’t do that.”
His comments come after the Leader of the House of Commons was criticised for "lounging languidly" on the front bench during a debate ahead of a crunch vote on Tuesday evening.
Jim Carter, who portrays the butler Charles Carson in the film, added his character would be dismayed with the state of the world today.
Speaking ahead of the film’s release, Carter, 71, said: “Mr Carson would stick his head in the sand and moan. He wouldn’t be able to cope at all.
“The lack of honour. Lord Grantham would chase Rees-Mogg with a horse whip for lying down in Parliament.
“Carson is an old traditionalist, always looking backwards, and he would look back to a time when there was more integrity in public life, shall we say, than there seems to be now.”
Explaining why Robert Crawley – also known as Lord Grantham - would chase Mr Rees-Mogg, Bonneville, 55, quipped: “For slouching on the furniture of the House of Commons. A gentleman of class wouldn’t do that.”
Elizabeth McGovern, who plays Lord Grantham’s wife, Cora Crawley, added: “We’re living in a very stressful age and probably the characters in Downton would be thrown to the same quandaries that all of us are in today.
“When you’re reading the paper and feel there’s no leadership that you can trust anymore. It’s a very confusing world to be in.”
Mr Rees-Mogg was seen reclining in the chamber, at times with his eyes closed, as MPs fought to take control of the Commons agenda and pass legislation which would prevent the UK crashing out of the EU on October 31.
Green Party MP Caroline Lucas told the Brexiteer his body language was "so contemptuous of this House and of the people".
After the photograph of Mr Rees-Mogg went viral, Twitter users mocked his body language with a series of memes which were posted online.
The Downton stars were speaking ahead of the movie’s release on Friday.
The film – which is a continuation of the ITV hit series – shows the characters preparing for a royal visit from King George V and Queen Mary in 1927. Their visit throws the usual peace in the stately home into disarray.