Scott Foley Doesn't 'Keep in Contact' with Ex-Wife Jennifer Garner but Says They're 'Cordial'

Leave it to Andy Cohen to make Scott Foley "squirm."

During Foley's appearance on SiriusXM's Andy Cohen Live, Cohen asked the actor, 49, if he was was able to "reap the benefits" of his Felicity fame in his romantic life. Foley admitted that "there were a few" young actresses that sought after him during his time on the late '90s series, which inevitably brought the conversation to Foley's "short-lived marriage" to his first wife Jennifer Garner.

"We're speaking about this with my current wife [actress Marika Domińczyk] behind me, and she's laughing. She's really enjoying the squirming that I'm doing here," Foley said. "[Garner and I] were together for a total of about five years. I say that, I downplay that, because my wife's right behind me. ... I think we were married for three."

"This is dangerous, dangerous territory, my friend," the Scandal alum added with a laugh. "This is where Andy Cohen wants to be!"

RELATED: Scott Foley Opens Up About His Family's Big Move East — and His New Job with Ellen DeGeneres

Aaron Poole/E! Entertainment/NBCU Photo Bank/NBCUniversal via Getty; Amy Sussman/Getty

Asked whether he would avoid Garner, 49, if they were to run into each other, Foley replied, "No, you have to say hello. You're very civil."

"I mean, we do not keep in contact because she has her own family. I have mine. It wouldn't be something that we would search out," he continued. "But I've seen her a few times, especially when we were living in L.A. and Marika has been with me. And, you know, everybody says 'hi' and you're cordial and you move on. That's life, right?"

Foley and Garner met when she played his girlfriend on Felicity, which ran from 1998 to 2002 on The WB. The costars "hit it off instantly and were soon dating," Foley recalled on Andy Cohen Live.

They wed in 2000 and announced their separation in 2003. Their divorce was finalized the following year.

Paul Drinkwater/NBCU Photo Bank/NBCUniversal via Getty

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Foley went on to wed his current wife Domińczyk, 41, in 2007. The couple shares three kids: daughter Malina, 11, son Keller, 9, and son Konrad, 6.

As for Garner, the Alias alum married Ben Affleck in 2005. The exes, who divorced in 2018, share daughter Violet, 15, daughter Seraphina, 12, and son Samuel, 9.