What benefits can bring the neural network for film, video games and virtual reality

With the development of technologies of neural networks and machine learning, and expanding their scope. If the earlier neural networks were used exclusively to perform complex mathematical, medical, physical, biological, calculations and forecasting, but now these technologies are gaining wide popularity in more "mundane" environments – in the field of entertainment. Making only the first steps in this direction, they are able to demonstrate surprising and sometimes even outstanding results. Today let us examine a few illustrative examples.

The Process of remastering the video is so complex and time-consuming that many of the masterpieces of world classics we may never have seen with new, modern, clear and juicy picture. However, the world is full of intelligent fans and enthusiasts who are well versed in new technologies, in particular technologies of neural networks and machine learning, with the help of which you can achieve amazing results even at home. For example, a YouTube user Stefan rumen alias CaptRobau decided to demonstrate some of the capabilities of neural networks in processing video of the old sci-Fi series.

Earlier his work is Remako Mod — "HD-remake" of the classic and very popular Japanese RPG Final Fantasy VII called. For this he used the AI-algorithm AI Gigapixel, which was able to scale the image of the original image 4 times, moving it in HD resolution without any significant changes to the original art design. Thus, while you will have to wait another decade to the time when Japanese developer and publisher of computer games Square Enix will officially release the remaster, which is probably one of the best parts of this game series, you can try a mod Stefan rumen yourself by downloading it .

By the way, recently the technology of neural networks for remastering old games and bring them to a more relevant and modern without changing the overall original concept has become a real trend among the various modders. For example, not so long ago we told you about ESRGAN technology (Enhanced Super Resolution Generative Adversarial Networks), which have implemented technologies for scaling images with a 2-8-fold increase in quality. Algorithm "feed" the original image with low resolution, then it not only increases the native resolution of the latest, but also increases the image quality, Podrezova lifelike detail and making the texture "more natural".

Comparing the quality of the textures: on the left the original texture from the game Morrowind, to the right is processed by the neural network

Character from Doom (the left — it right —)

background Processing in the game Resident Evil 3

To Read more and see examples you can follow .

Anyway, in between the remastered "Seventh Finals" Stefan rumen decided to do another project using the same technology of machine learning, but for handling frames classic sci-Fi series of the 90-ies. As a subject for experiments, rumen selected ": deep space nine."

Scaling the "living image" of a TV series on complexity very different from the scaling in advance ofrendering image Final Fantasy VII, the author notes, so the final result looks much better source material in low resolution, but this picture is still far from the ideal that you could dream of ever since the appearance on the market first Blu-ray players. Occasionally appear on the screen a small "artifacts". But, again, overall it looks more than worthy. And, in General, look for yourself.

For this project rumen also used the algorithm AI Gigapixel, which was trained to edit images based on real photos. The author notes that the new image was obtained in 1080p and 4k, but since I currently have no TV or monitor with native 4K resolution, to adequately assess the 4K option, he can not.

Sorry to see the show in Full HD is impossible. Processing of all source material would take a very long time, so the rumen for demonstration only used the individual frames of different series. According to him, he started this project for one reason – to show that it is indeed possible. According to him, a whole team of professionals working for a major television company and which have a more suitable and powerful such computer equipment will be able to cope with this task much better.

Using neural networks to simplify the processing of old images from video games and movies are not the only areas where such technologies are able to display their talents. In the modern world, where increasingly popular panoramic camera, capable of producing a 360 degree angle, and headset of virtual reality, developers began to actively explore the potential of panoramic photography.

One recent development in this direction is the neural network capable of voice panoramic staticimage. The authors are experts in the field of machine learning from mit, Columbia University and George Mason University.

This algorithm specifies the type of environment and objects in pictures and then selects and arranges sounds from the used database in accordance with a spatial calculation of the distance to their sources in this image. Thanks to this panoramic image appear realistic and three-dimensional sound, allowing a completely new way to evaluate the presented picture.

According to the developers of this neural network, the technology could find interest among developers of VR content (movies and games). The latter in this case do not have to put all the sounds on a panoramic image manually, the neural network will be able to do it alone.

Subscribe to our . There you can find materials that are not published on the website.