“If you are going through hell, keep going!” - Winston Churchill
In the Bible, the one story that cracks me up every time is in the book of Jonah. In this story, a series of events happen that seem directly inspired by a 1980s Leslie Neilson comedy. I’m surprised that Seth Rogan hasn’t directed a comedy of this story in a style similar to Pineapple Express. The story of Jonah begins with our reluctant hero being given a task from God to preach in Ninevah, the capital city of the Assyrian empire. Jonah refuses because he hates the king. He also hates God for forcing him to preach there. Instead of heading east to Nineveh, he boards a ship heading west to Tarshish. The ship is filled with Pagan sailors, therefore Jonah believes that he is free from God’s instructions. While Jonah sleeps on the ship, God sends a terrifying storm to assault the ship that Jonah is sailing on. Somehow the Pagans figure out that Jonah is the reason that this storm is happening, and they interrogate him. Jonah reveals to the sailors that he is a Prophet of God, but God wants him to do something that he rather die than do. Therefore, Jonah suggests, to the sailors, to kill him by throwing him overboard. The sailors reluctantly do as he says, but repent while doing it, because, after all, this is a prophet of God. The storm passes and Jonah sinks to the bottom of the ocean, thinking that he’s foiled God’s plan. Suddenly, a Whale swallows him whole. Now stuck in the belly of this fish, Jonah begins to pray and fast, Jonah asks for forgiveness and says, “Fine, I’ll do it.” The whale subsequently vomits Jonah onto dry land, and he marches to Ninevah to get it over with. Once in Nineveh, Jonah gives the most bare minimum, low-effort sermon of all time. “Forty more days and Ninevah shall be overturned.” That’s it, that was the whole prophecy. Then like a tornado hit, the King of Nineveh, all its citizens, and all of their farm animals repented and begged for forgiveness. God forgives the Ninevites and does not bring destruction to the city. Fulfilling the prophecy because the city was overturned, their hearts turned from being away from God but towards God. Jonah is pissed! He desperately wants God to destroy the city, and Jonah begins to mock God for being so merciful and demands that God kills him on the spot. Then the book ends with God asking Jonah, “Why you so mad, bro?” Remember to like, comment, and subscribe to help Medallion XLN create the new internet.
Sometimes God calls us to an uncomfortable path
I’m not patient enough to wait for Seth Rogan to make a film about this Bible story. I figured that I would use AI to do it and write an article documenting the process. Last year, many AI experts were predicting the rise of Prompt Engineers. People who have such a deep understanding of how to get the correct results from a Large Language Model that they do it for a living. What I discovered is that the LLM itself, is the best Prompt Engineer for how to talk to image generators. I like to use a variety of LLMs and image generators to get the desired results. I try to use a combination of ChatGPT, Claude 2, and BARD, then stick to whichever generates the best text-to-image prompts. I like using Claude 2 the most, it accounts for side effects that I would never think of. What I like about Claude & ChatGPT is that they can track the progress of the story. First I prompt the LLMs to generate the first scene in 500 B.C., with volumetric lighting and advanced texturing techniques. As I continue with each new text prompt, the LLM remembers to include the previous constraints from each of the text-to-image prompts that it generated. I simply tell it to continue the text prompt with the next part of the story.
Claude 2 is low key the GOAT
My priority is maintaining the art style between generations to keep a consistent flow between images. For this, I like to use a variety of Text-To-Image generators as well. I like Mid-Journey’s generations but Stable Diffusion XL has come a long way and as long you prompt it correct, you can generate amazing images similar to Mid-Journey. I also find that Mid-Journey is too opinionated, it ignores whole swats of my prompt while nitpicking only what it wants to generate. I don’t have this problem with Stable Diffusion XL. For the short film, I used a combination of both and used image to image until I got the image that came closest to the one I had in my mind. My favorite Stable Diffusion XL website is called PlaygroundAi.com, which distinguishes itself by providing access to different Low-Rank Adaptations (LoRa), which is a technique used to fine-tune images to match different styles of art. PlaygroundAi allows 51 free image generations per day. I also like to use NightCafe which allows me to pick different image generation algorithms such as Style Transfer, CLIP Guided diffusion, and VQGAN + CLIP. The last part of my image generation process is upscaling. For videos, the upscaling that Mid-Journey does is too small for a good-quality image-to-video process. PlaygroundAi provides a default upscaling of 3000x2000 for landscape images that would be used in videos.
Playgroundai.com. is a great starter text-to-image generator.
Finally, we get to the video creation process. For this stage, I like to use Kaiber and RunwayML. Working with image to video generators can sometimes be frustrating. Sometimes the generator does something completely random that has nothing to do with the initial image. Kaiber uses a credit system, while RunwayML provides how many seconds of video can be generated. I like RunwayML's remaining seconds system more than Kaiber’s credit system. Kaiber does two forms of video generation, which are flip book and consistent motion. The flip book is a collection of images played one after the other. The consistent motion cycles through each image smoothly. My problem with Kaiber is how stiff their generations are, and it discards all the advanced volumetric lighting and textures. Instead, I prefer RunwayML for its dynamic image to video generations. With RunwayML I don’t like how I have to rely on the interpretation of the algorithm. It does a good job sometimes, but I want more control over the output video. Halfway through the short film, I ran out of seconds in RunwayML to generate more image-to-video clips. I discovered the PikaLabs discord and was surprised at how good the generations were. Pikalabs even solved some of the issues I was having with RunwayML generating something completely random. As these AI tools continue to improve, I look forward to the rise of multi-modal apps. I don’t want to jump from generating text-to-image prompts then finding a text-image generator, finally searching for an image to video generator. I want a text-to-video generator that has a UI interface like a video editor, and each row gives me the option to use text to image for a variable amount of seconds. Then I should be able to drag an object in the image for a certain amount of key frames to optimize for precise output. Most artists are religious because they recognize the talent that they were blessed with came from the divine. After all, God created man in his image.
RunwayML has some great AI tools for video generation. PikaLabs is a great free alternative on discord.
Join Medallion XLN’s mission to build the next era of technology. XR, Blockchain, AI, and decentralization will reclaim our digital independence.