'Avatar: The Way of Water' visual effects master, a Center High grad, discusses trailblazing work

The waves look amazing, the new sea creatures astonish, everything from the forest fauna to menacing war machines pops off the screen to dazzle the mind in "Avatar: The Way of Water."

Pioneering technology spearheaded by Beaver County native Joe Letteri has begun entrancing moviegoers as the long-awaited sequel to the largest-grossing movie ever plays in theaters globally.

One key reason it took 13 years to release this sci-fi blockbuster: New technology and techniques needed to be crafted first by Letteri and his team at Weta FX to realistically portray detailed, underwater movements for computer-generated characters and creatures. The performance capture technique Letteri helped revolutionize with the Gollum character in "The Lord of The Rings" film series ― where facial expressions of live actors are tracked, recorded, and rendered to create animations ― had never been done in scenes that so closely mix underwater and above water action, until "Avatar: The Way of Water."

In a one-on-one video chat Wednesday, Letteri, a 1975 Center High grad and five-time Academy Awards winner, talked about that new technology and his hopes and expectations for the James Cameron-directed movie that's already earned a Golden Globes nomination.

In the mildest of spoiler alerts here, a chunk of the film's action takes place in an oceanside village that looks like the Planet Pandora's version of Tahiti. Before you Google information on how to travel to this idyllic spot, you'll need to realize it's all computer-generated imagery (CGI) from Letteri's team.

Letteri: "It's all CGI. All of it was us. It starts with the whole performance capture process. We did a big water tank that had the whole performance capture volume put in there. All those things really help with the believability of what the actors can do and what we can take from that. Everything you see on screen is digital.

Q: The underwater CGI was trailblazing, so what was the secret to cracking the code?

Letteri: I think people have messed with doing that before for specific things but the secret for us was we're doing a lot of underwater shots but then also the characters popping up and doing dialogue at the water surface. So, to make that work, you need a capture volume that's in the water, you need a capture volume that's above the water, and you need the two to talk to each other, which they don't naturally because water absorbs light in different ways. Normally, it will use infrared light for a normal motion capture like on land. In this case, we had to use ultraviolet light for underwater and then put the two together in real time so that Jim could see the performance as he was going with it.

Q: The actors themselves, how did they respond and did they have to learn diving and underwater techniques?

Letteri: Yes, they did extensive free diving training. Like they would do 2-minute takes. It was amazing to watch. They had, I'm guessing, 15 to 30 seconds to get down and set and let everything clear, then they'd roll off a take for about a minute and then come back up so that they had a little bit of safety there. And there were safety divers. I mean the whole thing was about safety for everyone, but they were impressive with what they did. All of them from Trinity (Bliss), the youngest; Sigourney (Weaver). I watched them and I was like, 'I could never do that.'

Q: Kate Winslett, too?

Letteri: Kate did, yes, absolutely she had underwater scenes. They all did.

Q: What was it like for you and the team to create all these new sea creatures we've never seen before?

Letteri: That's always fun. That first time you go underwater there's that nice collection of fish that could only exist on Pandora that I thought were cool. Especially the tulkun, the big whale-like creatures, because we're trying to show this idea there's sentience, that there's intelligence, and anatomically they are very much like whales, so we had to figure out how do you give them a personality. We actually took a little bit from a seal, so you get a little bit more of a puppy dog personality. Especially with Payakan, because in age Payakan is a teenager just like Lo'ak is, and the two become friends. Mostly we had like one big eye to work with plus that little eye next to it. Really it was just all about that close-up performance to the eye and the way they worked off of each other.

Q: The ocean waves look so realistic. How did you capture that science?

Letteri: There's a number of ways. For the actors when we were shooting in the tank, that tank had a big motor-driven paddle on the side that would do waves for the surface. We would look at those and do a lot of math to try to recreate the formula for those waves. And the same thing when you're going out into the ocean. There've been a lot of studies to understand how ocean waves are formed, so we based what we did on a lot of that. But then you have to deal with what we call a physical simulation, which really comes down to math and physics. You start saying, 'OK, here's how much water is being pushed at what velocity and here's how many waves are out there' and you get that all running. Now you put a character in it or a creature or both, and these nice waves start slamming into things and breaking up and forming spray or bubbles under the water. And all of this is just running on computers for days and days until you look at it and again and hopefully you got it right. And if you didn't you go back and do it again.

Q: Besides new sea creatures, we also see some really cool war machines, like those submarines with the claws. Do you have a favorite?

Letteri: Those crab suits that you're talking about, that's what we called them. Yeah, those were good. Because rather than just being a sub, they could also articulate and crawl around and move. The more complex the behavior, usually the more interesting you can make it.

Q: You might want to get a patent for those before they start inventing them.

Letteri: (Laughs). Jim's probably just going to build one and have it at home.

Q: What's the creative process like for you? Do you see the script ahead of time and then start planning, or is there a give-and-take where you pitch an idea?

Letteri: Most or all that stuff comes from Jim. He spent a long time with the writing team coming up with what originally was going to be three sequels and that turned into four because you had so much story. And as he was writing he had an art department there and would pitch them the ideas saying 'This is what I want' and they'd start doing artwork on it. Once they were in the ballpark they'd start creating digital models to start to see what it would look like. And then the team at Lightstorm (Entertainment) would start experimenting with motion studies because you've got to make sure these things move correctly. Because you can draw something that looks really cool, but if it can't move that doesn't help you. So all that kind of happens while the scripts are being developed. Then once you know the scripts are locked down that's what we know is really a keeper; what's staying in the movie and what's falling by the wayside. And that's when we start doing the real detailed development.

Q: The first "Avatar" came out in 2009, so in what ways has the technology changed in a way moviegoers might notice?

Letteri:  A lot of things. I think a couple of the big ones were the facial system. When we were doing the first "Avatar" we were basically using the same concept of how you animate faces that we developed for Gollum, with some refinements but that's what we use really all the way through "Planet of The Apes" and "Alita: Battle Angel," but for this film, I had an idea about how we might actually understand a little bit better how faces actually work. So, we wrote basically a neural network. We did some deep learning to use that to understand what the muscles in the face do and how that relates to performance. So we could get the actor's performance and we could do an analysis to figure out what must be going on to get that performance and then transfer that over to the characters, look at it and then make adjustments from there to get the emotional feel of the performance that you want. So, hopefully, that comes through. There's a lot more detail, a lot more nuance than we were ever able to capture on the first film.

Q: The release date was delayed a couple of years like many movies. Did any new technology catch up with you so you had to go back and reshoot something you already thought was done but now could be improved?

Letteri: No, because all this new technology we were writing ourselves. That's all the stuff we created for the film. We were still going while that was going on. Any delays in the shooting didn't really affect us because we're all still working on computers, we just kind of switched to working from home and just carried on so mostly all those shots became testbeds for all this new software we were writing. Like the first sequence we did where Jake is talking with Neytiri and saying, 'We've got to leave this place' and the kids are spying on them. I mean, we had Jake and Neytiri from the first film obviously, but now they're 15 years older, we've got all this new technology for the face, we're testing it all out. Just that one scene took us a year to do just to figure everything out and then it just went from there.

Q: With WETA was their pride and pressure on this sequel? You blazed a new trail with the original "Avatar" so was there a spoken or unspoken belief where you had to up your game? 

Letteri: It's not really about upping the game so much as we know things that we'd like to do better and just keep going. Even Jim said at one time, 'Why can't we just do it the way we did last time?' It's like, because we can do it better this time (laughs).

Q: Have you attended any screenings yet?

Letteri: I saw it for the first time at the L.A. premiere the other night. It was great. I always like to wait to see the film for the first time with an audience. It's nice to just sit back and forget about the work and just watch the movie. That is why we make these things. We make movies we want to go see. So, you've got to go see them.

Letteri said he does plan a visit to Beaver County next week to see his mother around the holidays.

Family is a central theme to "Avatar: The Way of Water."

"This film is all about family. The first one was Jake and Neytiri finding each other, and in the last 13 years Jim's kids have all grown up and this one was more about, OK, what comes next?" Letteri said. "He didn't want to just go back and pick up the story where it was. He said by the time we make this it is going to be 15 years later, so what's happening? That gives it a really nice center, being about family."

Scott Tady is entertainment editor at The Times and easy to reach at stady@timesonline.com.

This article originally appeared on Beaver County Times: Letteri raises the bar again with 'Avatar: The Way of Water' effects