Artificial Intelligence in image production

By Manuel Ruiz Dupont: real-time development process consultant and trainer at Pixelacademia

This article discusses the disruption that AI will bring to the field of image production. Even if you are not in the business, you can easily understand it because I illustrate my point with many visual examples (and I advise you to first look at the video of Mr. Villani’s speechin the Senate, in which he discusses the limits of AI ).

What is AI?

There are several types of AI (ANI, AGI, ASI…), each with its own definition[1], but I will only deal with those that use a database managed by complex algorithms. There are also AIs without databases already constituted: these are created at the moment “T” when the order is launched, or are created gradually (Machine Learning).

What do we do with AI?

Today, there are already many applications that work well with AI.

Creation of very high quality images, created from text or a simple sketch

Image made with Google Colab (left) and Nvidia (right)
Image made with Google Colab (left) and Nvidia (right)

Ability to create music with a very simple interface or from text

Aiva interface, music creation software with AI
Aiva interface, music creation software with AI

Automatic creation of a story (tales, in particular)

Possibility of having a game play “premonitory” actions of the player, in the field of video games

The “Neural State Machine” presented at the Siggraph show in 2019, is able to learn and “to predict” the interactions between the character (avatar) and the scene from real-time motion capture data.

A number of fun applications that use AI exclusively are also already on the market. You can also find them for free on the internet (Google Colab). They are usually programmed in Python, but you can use them without knowing how to program.

See your face instead of an actor in a movie scene

Example of a deepfake with actor Tom Cruise (right)
Example of a deepfake with actor Tom Cruise (right)

Possibility of seeing your face in 50 years or if you were a woman or a man or of another origin

Image generated from the AI ​​software Artbreeder
Image generated from the AI ​​software Artbreeder

Added to this are the applications which are still only at the embryonic state of research

Generation of 3D characters, with their animation from a text

Generation of 3D characters, with their animation from a text
HumanML3D precursor of animation “textual

Creation of 3D volumes from a single photo

And, of course, there are all those applications that we would like AI to make possible today, such as the automatic muscle rig, or even the creation of hyper-realistic clothes that can be animated according to a pattern or a simple photo, but we’ll have to wait a little longer.

AI in image production

Current AI software is not positioned as solutions to development issues. Unreal Engine took more than 6 years – despite its enormous means – to create a relatively acceptable transversality with 3DSMax, Maya and Houdini, but still very weak for Nuke. Consequently, if we arbitrarily consider the year 2022 as the year zero of AI, we will still have to wait before AI is truly applied in the image production chain, and this will necessarily go through the activation of plugins on production software and – at the same time – by the birth of AI software that takes into account the needs of production software: AI will really impose itself when it is accessible on the production software already installed.

Until now, if you worked in the field of video games, you only had to master a few technical concepts (UV, Bones, normals, polygons, shader) to develop, but the increase in the power of machines has created bridges with other sectors of activity (cinema in particular) which manage other concepts (fluids, hair, renderings, etc.). I think that AI will initially make it possible to create tools with very powerful and easy-to-use technical concepts thanks to simple interfaces, and which will be used both for cinema and for real-time professions.

Secondly, the AI ​​will also make it possible to quickly produce visual effects or tricks that required a lot of development time. These include the creation of graphic styles or the famous deepfake (it is already possible, but the result is not precise or qualitative enough to be implemented in a production).

Portland, in the style of Van Gogh's Starry Night
Using various mathematical models for artistic style transfer, here the skyline of the city of Portland, in the style of Van Gogh’s Starry Night

Is this the end of certain professions? Nobody thinks it, but Quixel, Kitbash3d, or even the texture banks have considerably reduced the human needs of a production, but at the same time, the visual production has exploded and the human needs are only increasing.

For example, I tried to create a 3D animation with as little human intervention as possible.

Here’s how I did it:

  • I created a text with an AI software that generates poems
  • I created the image with an AI image creation software from text (I copied the text created by the poem creation software)

  • I created the “2D volume” with AI software
2D volume

  • I created the 3D volume with photogrammetry software
  • I exported the volume in Maya then created a cinematic with camera movement
  • I exported the rendering in an AI software to make the animation more homogeneous

What is the future of AI?

Currently, the biggest obstacle to AI is the time required to create the database and process it through algorithms.

If we manage to reduce this time, we can imagine that AI will help us in real time – throughout development – to sublimate our creations or correct our errors. If we combine this real-time AI with other techniques (asset library, avatar creation, etc.) which will also progress enormously, we can image the creation of films without too much difficulty (with dialogues and music ) or cartoons, made according to a more or less detailed story board, where the AI ​​would simply fill in the gaps according to its databases. The creator of this film would only have to modify certain details of the result, according to his vision, using a fairly simple interface. I know this might sound like science fiction, right now…

Of course, many steps will have to be taken before arriving at a viable solution, but the closer we get to a spontaneous creation with very few human needs, the more the problem of the originality of the work will emerge, because AI needs databases. However, at the present time, many of them are created without the agreement of the authors.

Can we already imagine the creation of new companies which would market databases of images of which they would be the authors? And if, in the end, one of the possible futures, for the companies of development, was simply the creation of images intended to constitute databases?

When to switch to AI?

It reminds me of that period, 4 or 5 years ago, when real time started to be used in sectors other than video games. It was initially seen as the ultimate solution to many development issues. In some cases it was true, but in others it wasn’t. And often real time has simply offered more creative comfort. For example, real time gives real added value to video mapping, and in the field of cinema it offers greater staging comfort. In the field of animation, it makes it possible to create a more flexible development process, but the final rendering loses in quality.

Today, AI is already present in the pre-production phases (it is used by concept artists). In the production phase, it is clearly not developed or remains anecdotal, but I think it will soon be present in the last phase (post production), in particular for the colorimetric adjustments (DVinchi), because there are already thousands of databases that allow you to use it quickly.

So when to switch to AI? Should we wait until the software is finally ready, knowing that the next version will always be better? In the end, if you are careful and dedicate some time to R&D and technology watch, there is no bad choice. But in certain sectors of activity, the implementation of AI will be faster, in particular in the field of photo or video restoration (and surely, already, in that of music).

Photo restoration
Photo restoration made with Photoshop and AI software (stable diffusion).

How to teach it and why?

Teaching AI to a graphic designer like they are taught Maya or Blender seems impossible to me because, at the end of the day, it’s all about math. But we are not going to ask a graphic designer to program an AI. On the other hand, we can explain to him the technical concepts that the AI ​​uses and which are very far from UV or fluids.

What is certain is that the following trend has been emerging for several years: companies are looking for profiles who master aesthetics and tech.

New professions will appear, and therefore new profiles that will have other qualities than aesthetics or tech and rather a strong disposition to abstraction in order to be able to manipulate the tech concepts of AI, without being virtuosos of math.

We must already teach AI in schools and each school must identify how to do it because, in the short term, companies will ask for profiles comfortable with AI concepts.

to summarize

I believe that in the medium term, AI will be a real vector of upheaval within our production chains, because the development processes will evolve with its implementation in the software we use. The teaching of AI will take a full place in video game schools and we will see it appear in new courses.

Of course, new professions and companies will appear. In the very short term, the real obstacle lies at the level of ethics and legal aspects. Indeed, AI allows the creation of very high quality content that may be ethically reprehensible. Moreover, at the present time, databases are probably created without the agreement of the authors, which poses a legal problem.

If you are interested in the legal and ethical aspect of AI, I advise you to read theinterview that Mr. Emad Mostaquefounder of Stable Diffusion (AI software) granted to The Times.

Manuel Ruiz Dupont
Consultant in real-time development process and trainer at Pixelacademia

We wish to give thanks to the writer of this short article for this incredible web content

Artificial Intelligence in image production

Explore our social media profiles and also other related pages