‘The Irishman’: How Industrial Light & Magic’s Innovative De-Aging VFX Rescued Martin Scorsese’s Mob Epic

Once again technology caught up with need when Industrial Light & Magic developed an innovative digital de-aging process for Martin Scorsese’s character-driven, non-linear mob epic, “The Irishman,” requiring Robert De Niro, Al Pacino, and Joe Pesci to look decades younger as hitman Frank Sheeran, Teamsters leader Jimmy Hoffa, and Philly crime boss Russell Bufalino. The impressive results put ILM on the Academy’s long-list this week for the VFX Oscar.

Frankly, it couldn’t have been done any other way to preserve their career-capping performances in this zigzagging meditation on friendship, betrayal, and mortality, spanning four decades.

“This was a movie about conversations and we needed a new system because the actors wanted to be themselves,” said ILM VFX supervisor Pablo Helman, who first prepared a test with De Niro from “Goodfellas” in 2015 that got the film greenlit by Netflix. The costly VFX de-aging, therefore, became the tech centerpiece, with 1,750 shots created for two and a half hours of footage, which was the equivalent of making two movies in one — but in nine months instead of 18.

“The Irishman” camera rig dubbed “The three-headed monster.”

Netflix

Popular on IndieWire

Helman and ILM undertook a two-year, NASA-like science project, developing a special camera rig (with cinematographer Rodrigo Prieto and ARRI) along with a markerless, light-based performance capture software called FLUX. And one of the many ways that FLUX made a difference in going markerless was by adding the proper reverberation to the actors’ upper and lower lips when speaking. That’s because the markers separate the movement and obscure the deformation. Not anymore.

In prep, the actors were shot separately, each performing a range of facial movements, then again on a special lightstage to capture a range of lighting conditions. That data was then used to create 3D models of each contemporary actor (all in their ’70s).

All scenes were shot on set using the proprietary camera rig (dubbed “The three-headed monster”), which consisted of two high-res ARRI witness cameras attached to and synced with the primary RED production camera. This system captured the facial performances from multiple angles with all of the necessary on-set lighting, and also threw infrared light onto the actors’ faces while remaining invisible to the production camera.

“The Irishman” Robert De Niro (Before De-Aging)

Netflix

“The problem is, the system gets confused when there are really big changes between key light and shadows, so we had to calibrate and equalize the light,” Helman said. “That’s why the left and right witness cameras are infrared, which don’t have any shadows. It doesn’t affect the lighting on set but it does affect the way the computer sees the data.

“The system analyzes the lighting and texture information and makes a renderable mesh of geometry for every frame,” Helman continued. “The system then compares it to the digital double of the contemporary actor. Then it gets re-targeted to the younger model.” The modeling, muscle and skin development were all customized for FLUX. Additionally, ILM researched and accumulated thousands of historical photographic and video references of the actors’ younger performances, and built an AI-based engine to reference them in real-time to aid in the de-aging process.

ILM created four age variation models for each actor with built in sliders so they could age up to best match the raw performance. The shapes of the neck and chin were completely different on each model. “Also, there’s the negotiation that goes on with the design, where the wrinkles are, where the eyelids are,” said Helman. “There’s a basic outline to the face that needed to be addressed: the eyebrows, eyelids, the wrinkles around the face, around the mouth, and around the nose. When you take that away, then even if you keep the performance, you have to make some concessions.

“The Irishman” Robert De Niro (De-Aged)

achiu

“To Marty, these were damaged people and he defined the characters based on having a rough life, and de-aging was skewed toward older,” Helman added. “From that point of view, you will see Frank Sheeran as a young version of the older Frank played by De Niro in his ’80s [which relied more heavily on makeup except for the hairline]. We did some 2D work on the hands and on the bodies. But we also aged up De Niro if he didn’t look old enough. For instance, when he goes shopping for a casket.”

Since Scorsese was only interested in seeing final results, Helman would deliver a wide range of complete renders once a week for the director and editor Thelma Schoomaker to evaluate during post-production. And there was no key-frame animation involved because Scorsese wanted unadulterated performances. “It was really difficult to finish one sequence and then move to another one because it’s kind of a puzzle,” Helman said. “Marty would look at it side by side and sometimes catch something that was important to him. Oddly enough, the three and a half hour cut had not changed for nearly a year. I was surprised. I thought Marty was going to have a problem with different takes and we would have to recompute everything. But he did not do that. Obviously he had a great vision of what he wanted with Thelma.

By far, De Niro’s Sheeran was the most demanding to de-age because he’s so reserved. And the most difficult to get right was the restaurant scene when he’s summoned by Harvey Keitel’s Angelo Bruno (the head of the Philly crime family) and Bufalino to explain his surreptitious arson gig, which harmed him financially. “That was a very difficult scene for all of us because we haven’t observed how ruthless Angelo Bruno can be,” Helman said. “The only reason we think that he’s ruthless is because of De Niro’s performance because he acts like a little kid that did something wrong. If you saw the original dailies, he looks like a turtle trying to hide in its shell. If you take the neck out and unwrinkle it, then you lose the body language.

“The Irishman”

Netflix

“That was a conversation that took months to get Marty and Thelma to understand and for me to go back and show them what we could do. And those were shots
that we did at the beginning and then we went back after the software was acting a lot better and faster to get the emotion right. And I think it made a difference.”

In fact, the way “The Irishman” de-aging ultimately came together as a potential industry gamechanger made Helman reflect on the very first question asked by Scorsese: “What is this going to do to my movie?” Helman’s answer: “It shouldn’t do anything to your movie. The only thing it should provide is context in the form of the narrative so that the story makes sense.”

Source: Read Full Article