Nathan Congleton/NBC/NBCU Photo Bank via Getty Images; Getty Images
The former president celebrated his 60th birthday last Saturday at his Martha's Vineyard home with an outdoor party for about 200 close friends and family. Days before, it was announced that the celebration had been scaled back amid coronavirus concerns after the lengthy guest list and other details of the party were criticized.
David, 74, told The New York Times that he got a call from Obama's assistant ahead of the event and assumed he was being asked to perform, causing him to panic at the thought of having to come up with a stand-up routine in just three days.
"I was pretty glum when I finally called back his assistant," he told the outlet in an email. "When he told me I was eighty-sixed from the party, I was so relieved, I screamed, 'Thank you! Thank you!' He must have thought I was insane."
"Then I hung up the phone, poured myself a drink and finished my crossword puzzle," the Curb Your Enthusiasm star added.
ALEX EDELMAN/AFP via Getty Barack Obama
"[Obama] decided to celebrate by throwing this huge blowout on Martha's Vineyard and everybody who was anybody was going to be there, including yours truly," said Colbert, 57. "But given the whole pandemic thing and the delta variant, a celebrity mosh pit was maybe not the wisest choice, so Obama decided to scale back the guest list for his party — he was forced to limit the invites to only his closest Beyoncé's."
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.
Hannah Hankins, a spokeswoman for the Obamas, told PEOPLE that the size of the birthday bash was reduced due to a recent rise in COVID-19 cases and the spread of the delta variant.
"This outdoor event was planned months ago in accordance with all public health guidelines and with COVID safeguards in place," she said in a statement. "Due to the new spread of the Delta variant over the past week, the President and Mrs. Obama have decided to significantly scale back the event to include only family and close friends."
"He's appreciative of others sending their birthday wishes from afar and looks forward to seeing people soon," Hankins added.
RELATED VIDEO: Why Barack & Michelle Obama Talked About "the Hard Parts" of Their Marriage: "We Came Through … Together"
PEOPLE previously reported that guests attending the party were required to undergo COVID testing. The event was also overseen by a medical professional who was intended to ensure that state and local pandemic protocols were followed along with the guidance from the Centers for Disease Control and Prevention.
Despite having to scale back, Obama still had "a great time" during the outdoor gathering, which took place under a massive tent on the grounds of his 29-acre Martha's Vineyard estate, a source close to the event told PEOPLE.
"He never stopped smiling," the source said. "Literally, every time I saw him, he had the biggest smile."
The insider added that he "danced all night." The celebrity guests who did end up attending included Gayle King, Tom Hanks, Rita Wilson, and Steven Spielberg, as well as performers John Legend, H.E.R., and Erykah Badu.