Almost everyone loves to dance, but only a few are blessed with the capability to make dance moves that are appreciated by all, for instance, Michael Jackson’s iconic Moonwalk. Now, a new AI tool – referred to as deepfakes for dancing – could make you dance like a superstar.
Deepfakes for dancing – what is it?
Artificial intelligence (AI) is growing more useful by the day. We already have AI tools that manipulate videos of people by face-swapping deepfakes. Now, thanks to the UC Berkeley researchers, we have a tool that can read any dance moves and copy them to another body.
“With our framework, we create a variety of videos, enabling untrained amateurs to spin and twirl like ballerinas, perform martial arts kicks or dance as vibrantly as pop stars,” the researchers say.
UC Berkeley researchers have developed an AI tool that copies dances moves from one body to another body. To use this deepfakes for dancing, one will need the test subject with dance moves for reference. This is will be enough to give an inexperienced dancer the look of a pro.
One algorithm in this deep learning system creates a virtual skeleton to map poses, while two algorithms work in tandem to transfer the move and also create a realistic face for subjects.
“These two networks are trained simultaneously and drive each other to improve, as the generator must learn to synthesize more realistic images to deceive the discriminator,” the researchers said in their paper.
Complex process and needs more fine-tuning
Internally, however, it is quite a complicated process involving a few steps, as described by the researchers in a paper titled Everybody Dance Now, posted on arXiV. First, the AI tool records the video of the target, and then an algorithm converts the movements as if coming from a stick figure. Around twenty minutes of good quality video at 120 frames per second is needed of the target subject. Then, another algorithm swaps the dance moves with the help of neural network synthesizing.
It sure sounds simple, but there are quite a few things at work to give the end result. One algorithm works to smooth out the movement of the stick figure so that it fits nicely, and another to ensure that the target’s face does not go off-target. Further, researchers claim that they have developed this model without using the expensive 3D or motion capture data.
This deepfakes for dancing tool, however, is still not perfect and needs some fine tuning. One can notice occasional missing body pieces and blurry details. Also, the dance moves shown in the demo video are relatively easy, suggesting the tool may struggle to replicate more complex dance moves. One more limitation of this tool is that it can’t accurately model loose fabrics. Thus, the dancers need to wear tight-fitting clothes.
As always, ethical concerns
It must be noted that the researchers have nowhere used the term “deepfake” in the paper. Rather, they prefer to call it “‘do as I do’ motion transfer.” Deepfake is a term used for an AI-based technique that superimposes part of one body onto another body. The first popular deepfake was reported in 2016 from Professor Matthias Nießner at the Visual Computing Group.
Thereafter, the tech found heavy usage in the porn industry, for instance, scandalous videos of Wonder Woman actress Gal Gadot, Scarlett Johannson and Emma Watson and so on. Even Barack Obama was deepfaked in a PSA warning about the danger of deepfakes. Deepfakes are banned on Reddit, and in the UK, one can even be prosecuted for using them inappropriately.
Also, like the face-swapping technology, this deepfakes for dancing could also face potential ethical concerns. At one time, face-swapping technology was seen as a breakthrough technology, but soon many started using it for porn. Similarly, deepfakes for dancing could prove a useful tool for parodies, but it may also be used to swap videos for making personal attacks, like faking a politician delivering a fake speech.
For now, there is no guarantee that the voice and face in a video is real. Going forward, the same would be the case with the person’s body movements as well.
