Joe Manganiello Says He 'Could Not Take His Eyes Off of' Sofia Vergara When They First Met

Joe Manganiello won’t be forgetting his first encounter with Sofia Vergara anytime soon.

During a Wednesday interview on SiriusXM’s The Jess Cagle Show, the 43-year-old actor recalled being immediately smitten with his now-wife when they first met at the White House Correspondents’ dinner party in 2014.

Manganiello was Cagle’s guest at the dinner, bringing his brother Nicholas as his plus one. “As we’re walking around, who comes swooping down in front of us for about 100 yards is Sofia Vergara,” he said. “She was wearing this dress that looked great on her, and I just could not take my eyes off of her.”

“Jess said, ‘Oh, you like that?’ I said, ‘Yes, yes, I like that a lot!'” Manganiello added with a laugh, explaining that he told Cagle he next wanted to go to whatever party the Modern Family actress, 47, was headed to. “So we just kind of followed her around.”

“It was my job to babysit the celebrity, so whatever Joe wanted to do, I was like, ‘All right, we’re gonna follow Sofia Vergara around all night and stare at her from behind,'” joked Cagle, 54.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Jason Merritt/Radarpics/Shutterstock Joe Manganiello (L) and Sofia Vergara

RELATED GALLERY: Joe & Sophie, Tom & Gisele and More Celeb Couples Who Shared Their Cute First Date Stories

Vergara’s Modern Family costar, Jesse Tyler Ferguson, must’ve sensed a connection because he came up to Manganiello later that night and told him, “My friend Sofia wants to come over and say hi.”

“I said, ‘Oh, your friend Sofia whose fiancé is also at this party?'” continued the Rampage actor, adding that Ferguson said Vergara was a “big fan” of his and that was why she wanted to talk. “I didn’t know what he was up to. I was like, ‘Is he gonna get me into a fight in my tuxedo here? What is going on?’ ”

And while his conversation with Vergara was a casual one in which they “exchange(d) pleasantries,” things changed around three weeks later after the actress announced her split from her then-fiancé, Nick Loeb.

“I hit up Jesse Tyler Ferguson and said, ‘Hey man, was she broken up with the guy at the time?’ He said, ‘Yeah, I wouldn’t have tried to pull anything if that wasn’t the case.’ I said, ‘Man, well, I don’t know what kind of emotional state she’s in, but tell her I want to take her out,'” Manganiello explained.

Sofia Vergara/Instagram Sofia Vergara (R) and Joe Manganiello

RELATED VIDEO: Joe Manganiello Stitches Sweet Secret Note for Sofia Vergara into His Star Wars Premiere Look

Joe Manganiello Stitches Sweet Secret Note for Sofia Vergara into His Star Wars Premiere Look

If you weren’t already envious enough of Joe Manganiello and Sofia Vergara’s exceptionally attractive marriage, this might do it

Ferguson gave Manganiello Vergara’s phone number later that evening, and the two chatted while Vergara was in New Orleans filming Hot Pursuit. He then flew to the Big Easy for their first date — “and the rest is history.”

“I said, ‘I’m coming to New Orleans. If you’re not available, I’m just gonna do some sightseeing, but I’m gonna be there and if you’re free I can take you out,'” he said. “She said, ‘You’re crazy.’ I said, ‘I’ll see you next week.'”

Manganiello popped the question about seven months later, in December 2014, and the pair tied the knot the following November.

The couple celebrated four years of marriage last fall, with Vergara commemorating the special occasion with a touching wedding photo that she captioned, “Happy Aniversary [sic] mi amor!!!”