A Journey into Ukraine to Volumetrically Capture President Zelenskyy

Secretly crossing borders to “capture” and “shoot” a sitting president in a war zone may sound nefarious. But that’s exactly what was done, and it was all for a good cause.

President Zelensky’s message in Zoom calls and press conferences is getting old. Unfortunately, too much of the world is losing interest. In a 24/7 news cycle, it’s hard to stay front of mind. President Zelensky and his team, including his press secretary, are incredibly savvy. They know it’s important to retain people’s attention on this war and that volumetric video of a presidential message, distributed as a mobile AR experience is unique and allows anyone around the world with a smartphone to “bring it close to home” by placing the President in their environment.

They’re also determined to not allow this war to destroy their country and everything they’ve built over the past decades. As you can hear in his AR speech, he’s focused not just on defending Ukraine, but also on rebuilding the country. In particular, Ukraine has had a strong reputation for engineering and technology talent and in the speech, the President is appealing to the broader tech community to help this rebuilding effort and is offering a plan for digital transformation in Ukraine. Delivering this message in a way that leverages advanced technology from Evercoast was key to its impact.

“The Metaverse is an expansive network of persistent, real-time rendered 3D worlds and simulations that support continuity of identity, objects, history, payments, and entitlements, and can be experienced synchronously by an effectively unlimited number of users, each with an individual sense of presence.”

How did it come together?

Thomas Hoegh of Garden Studios, Brent Hoberman and Carolyn Dawson of Founders Forum, and Martin Williams of Talesmith had started discussing the idea to try and capture President Zelensky as a hologram. Irra Ariella Khi, a Ukrainian-born entrepreneur based in London was also instrumental in bringing the idea to life.

We were introduced to the brilliant team Talesmith through common colleagues. On our first call, Martin and Mark cryptically told us about a project in a very high risk part of the world. Early discussions moved to an encrypted voice and chat platform where we could more freely communicate about the details of the project.

While busy at AWE in Santa Clara, CA, we were finalizing a go / no-go over email and on calls with London. Once it became clear there was a strong possibility of it all happening, one of the Evercoast team members had to cut AWE a touch short to fly back to NYC and pack the system for what would turn out to be a multi-week adventure.

There was no real time for pre-production. We had no idea what we were getting into in Ukraine, what resources we’d have access to, the space we’d be given, how fast the internet would be, the President’s schedule and availability, and what logistical support we’d have. It was an extremely fluid situation that changed hourly. In many ways, we were all flying by the seats of our pants.

First stop: London

We first flew to London to set up at Garden Studios. The goal there was to setup the system to capture a mix of London-based celebrities and Ukrainian entrepreneurs that the hyper-connected team of people behind this whole thing were arranging in real-time to come get captured and share a message of support. Peter Gabriel, Nile Rodgers, and Bernard-Henri Lévy were just a few of the people who came to get captured in 3D.

But at every step of the way, the entire President Zelensky part of the project was in jeopardy. While things were going smoothly at Garden Studios, it was an incredible effort to pull off getting the Evercoast system and even a 1-person operator plus a director from Talesmith into Kyiv. One moment we were at “this is 70% happening” and later that day it would be “we’re probably down to 10% at this point.” Thankfully, through some heroic efforts by a broad team of people, we were off to Ukraine!

The Trek Into Ukraine

Once the production in London wrapped, we packed up the Evercoast system in a couple hours. A diplomatic contingent from the Ukrainian embassy in London came over to Garden Studios and loaded up the cases in the back of a van. They drove across Europe, crossing through Belgium, the Netherlands, Germany, Poland and finally arriving in Kyiv 4 days later.

Meanwhile, Martin Williams, the Creative Director and Founder at Talesmith and Ben Nunez, the CEO and Co-Founder at Evercoast, were headed to London’s Heathrow airport and joined up with a former British Commando who would escort them on the trek into Ukraine. They flew from London to Krakow in Poland where another member of the security and transportation team picked them up to drive across the border into Ukraine.

It was sad to see the indiscriminate destruction of apartment buildings, hotels, cars, gas stations and other non-military targets along the drive through Ukraine. It was a bit nerve-wracking to see and prompted questions of how long ago those attacks had happened. Hoping for an answer of “months” dating back to the beginning of the war, unfortunately for stress levels, the answer was “a couple weeks ago.”

The trip to Kyiv was long and was broken up with a night in Lviv, a beautiful city on the west side of the country. Repeated air raids and the associated sirens throughout the night reminded us that we were in a war zone.

A couple of days before we got to Kyiv, the Russian military resumed attacks on the outskirts of the city. While attacks weren’t happening in the heart of the city, crossing the perimeter to get in was another source of volatility. Keeping things in perspective, we reminded ourselves that this was nothing compared to what the Ukrainians were going through and what was happening in the eastern regions near the Russian border.

The city was in an eerie state with the Ukrainian people aiming to strike a balance between trying to maintain “normal” life while staying safe, abiding by the curfew, and dealing with the stresses of being at war with air raids and sirens going off regularly.

Volumetric at the Presidential Palace in Kyiv

Bringing computer and studio equipment for a volumetric production inside the Presidential palace in a country at war isn’t easy. The logistical challenges were incessant and the security precautions being taken at every single step were enormous.

Fortunately, Evercoast’s system uses completely off-the-shelf components and can fit in just a few small Peli cases. We use 21 Intel RealSense Depth Cameras plugged into a high-powered workstation, and lots of soft, ambient, evenly dispersed and diffuse light.

We’d hired a local production crew to build a cube that’s roughly 5m x 5m. Inside that cube are lots of LED panel lights and 8 poles configured in a circle with a 3m diameter. We distribute 21 cameras across those 8 poles and mount one overhead (to get tops of heads, shoulders, etc.). All of those cameras plug-in to a single workstation running Evercoast software – which is really where all of the magic happens.

Evercoast Mavericks captures depth and color from every camera, reconstructs it in 3D in real-time for live streaming, instantly renders preview assets immediately after a take to review the performance, and then uploads and launches renders of the source data on our cloud rendering platform, Evercoast Cloudbreak, where we generate the highest resolution output, encode and compress the data, and then make it available for streaming – all in a fully automated fashion and executed within an hour. This is an entirely proprietary software stack we’ve built from scratch over a few years.

Delivering a Speech

The President nailed his speech. He did 2 takes in English and 2 takes in Ukrainian. Both were great. The President is a very sharp man, with a curiosity of advanced technology and even an awareness of some of the challenges with 3D capture. He didn’t think he could use his hands much because he had heard of challenges with capturing things like fingers in any level of detail. So on his first take, he barely moved. We let him know he could feel free to be as animated and passionate as he wanted to be and that Evercoast’s reconstruction pipeline can keep his hands, fingers and any motion intact (which you can see quite well in the final product).

Within a day of Evercoast arriving in Kyiv with just a few equipment cases and a 1-person operator, the system was fully set up, President Zelensky was filmed, his volumetric was rendered, and an AR experience was available for streaming to mobile web browsers across the world. It’s unprecedented.

The entire endeavor is a testament to the quality and breadth of the end-to-end platform Evercoast has built and the incredible team who has built it. It required only 1 operator from Evercoast plus some locally-rented lights and rigging to deliver the quality of volumetric output worthy of a President, and do so with speed, portability and ease, all while under unimaginable logistical and time constraints.

Our team of geniuses has worked hard and smart to build a suite of products and services that enables such a light pack of hardware and crew to generate such a high-quality of volumetric capture with immediate turnaround. In a warzone.