The Truth About AI-Generated Videos: Strengths and Limitations
Generating images with AI, as empowering as it sounds, can sometimes be an arduous process, especially when you have a particular idea and getting something close isn't straightforward. The same applies to generating videos, only with even greater challenges. And here are AI-generated videos' strengths and limitations.
Sora, a video generation AI model, took everyone by surprise when Open AI released it in February. It was the beginning of something new in the AI community that would transform how videos are made, as well as give more power to people with inadequate experience and equipment in professional videography or filmmaking to bring their creative ideas to life.
Shy Kids, a digital production team based in Toronto, was selected by OpenAI to produce promotional short films. They created "air head," a mind-blowing AI-generated short film that gives elaborate insights into what you can achieve with Sora. The truth behind it all, however, is that a lot more than just a well-built prompt fed to Sora was involved.
In reality, many of the things we see—the promotional videos—for these revolutionary tools we see these days often allow one to assume that "what they show us" is totally doable by anyone. What they don't show us is the actual work to bring that stunning demo to life.
Although Sora was used to generate most of what you see in "air head," professional production work is what makes it, say, "real." VFX, editing, colour correction, storyboarding, rotoscoping, and the like all result from a lack of control when working with such a tool as Sora.
Control is still the thing that is the most desirable and also the most elusive at this point. … The closest we could get was just being hyper-descriptive in our prompts. Explaining wardrobe for characters, as well as the type of balloon, was our way around consistency because shot to shot / generation to generation, there isn’t the feature set in place yet for full control over consistency. Patrick Cederberg, a post-production artist at Shy Kids, said.
When shooting videos or making films, you are able to control many parameters, such as clothing colours and the positioning of props. When generating videos, however, you don't have that much control, even though you can describe it to the AI. You mostly have to depend on the AI's imaginative capabilities to deliver what you really want, and that can be laborious as you may need to keep going back and forth.
In some instances, you may even find the AI introducing unwanted elements. In "air head," for example, a producer described how the model would routinely generate a face on the balloon that the main character has for a head, or a string hanging down the front. These had to be removed in post, another time-consuming process, if they couldn’t get the prompt to exclude them. Source
Getting the timing right can be another issue in some cases, as can getting the desired camera movements. And then one may need to do the work themselves to fix these shortcomings by coming up with workarounds in production, just as Shy Kids had to with "air head."
The shot above is an example of what Sora generated and what the end results look like after a lot of work has been done. A yellow baloon could have been requested as the head of the character, but what Sora generated was a red baloon and an actual human head. You'll also see some colour correction was done—for consistency.
At the end of the day, hundreds of videos may be generated, but only a very few could turn out useful.
Something rather interesting about Sora is how it responds to requests that can lead to potential copyright issues. For example, if you ask it to give you a video in the style of a certain artist of film, it will refuse such a request. So if you asked it to generate Spider-Man clips, it wouldn't.
More concerning, though, is how Sora knows what kind of content to not generate or how it figures out what you're trying to do when you want to wiggle your way to make it generate something it should. Does it imply that Sora was trained on such data? Surely, we may never get a direct answer to that, as AI vendors keep information about their training data close to their chest and discreet, for obvious reasons anyway.
For "air head," the Shy Kids made a small behind-the-scenes video describing some of the problems they encountered in making it.
As it stands, developing professional videos with Sora isn't exactly as easy as they make it look. A lot goes into making really good AI-generated videos, and it involves a lot of "behind-the-screen" manual work. Notwithstanding, Sora will get better, and perhaps we will have more control using it with time.
By the way, make earnings with your content on Hive via InLeo while you truly own your account. If you're new, sign up in a few minutes by clicking here! And here's a guide on navigating.
Posted Using InLeo Alpha
👏 Keep Up the good work on Hive ♦️ 👏
❤️ @george-dee suggested sagarkothari88 to upvote your post ❤️
I am not up to date with many of this AI stuff but for Sora, I believe a lot of corrections will be made and someday, it would give the exact thing that is expected.
Thanks for always bringing great information about these stuffs...
For sure! AI is bound to improve with more and more developments in it. For now, we're just going to keep playing around what it can do for us. Do you think you'll ever try it when it goes public?
For sure! AI is bound to improve with more and more developments in it. For now, we're just going to keep playing around what it can do for us. Do you think you'll ever try it when it goes public?
Very well sir, I am definitely checking it out.
Nothing is easy but with time we become used to with it. Again every technology has it's own strength and limitations. We just try to develop it or knowing that it how to use it's strength to use it in a better ways.
Yes, man. Nothing's perfect. We can only come close, and Sora isn't close yet. However, in the little ways that it can be useful, it'll do a lot to make things easier.
Yes, man. Nothing's perfect. We can only come close, and Sora isn't close yet. However, in the little ways that it can be useful, it'll do a lot to make things easier.
Yes, man. Nothing's perfect. We can only come close, and Sora isn't close yet. However, in the little ways that it can be useful, it'll do a lot to make things easier.
AI tools still need human expertise to produce the best outputs. This is the truth we need to know about AI.
Very true! As much as they can learn on their own, they just need humans to carry them by the hand sometimes.
New technologies always excite me. I hope they improve on Sora, giving us more control and consistency in its video generation.
When Sora finally goes public, I imagine a whole new world of creativity will be opened, man. People will use it in surprising ways, that's for sure.
I agree 100 percent
Congratulations @olujay! You received a personal badge!
Wait until the end of Power Up Day to find out the size of your Power-Bee.
May the Hive Power be with you!
You can view your badges on your board and compare yourself to others in the Ranking
Check out our last posts:
I see the Open AI posting demo videos on TikTok, asking people what video they'd like them to generatae next with Sora, and then generating SOME of them. And every time I see them generate a video according to someone's request, my head just goes "I wonder how many other people's requests they tried to generate with Sora that didn't quite fit the request before they eventually got this person's request right".
The process of AI Video generation generally is very entropic. You can be as specific as you want, but the AI will only generate what it understands, and what it has been fed enough data to be able to imagine and create from your prompt. I always think about how many tries it took to get the perfect 10-second clip every time I see them. And people talk about how AI will replace the movie industry in the very near future. It's not impossible, but it by all means obviously will take a long time before we get to where AI gets good enough to spin an entire movie from imagination. Until then, we'll enjoy the make believe OpenAI is selling us. Lol
Bullseye, my man. I imagine that they do a lot of background work to refine the prompt, if necessary, and do a lot of tweaks to fit requests that they get on X. It's not always straightforward with these AI tools. For now, anyway, we'll keep enjoying what they let us see. And I agree that AI really isn't going to replace people. Rather, it'll augment us and make us perform much better in respective fields. I know you know this very well.