Deep Reader NÜ WA: Visual Synthesis Pre training for Neural vis Ual World cre Ation

, machinelearning, deeplearning, NÜWA, paperoverview, visualworldcreation, visualsynthesis Paper Code Abstract This paper presents a unified multimodal pretrained model called NÜWA that can generate new or manipulate existing visual data images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoderdecoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate NÜWA on 8 downstream tasks. Compared to several strong baselines, NÜWA achieves stateoftheart results on texttoimage generation, texttovideo generation, video prediction, etc. Furthermore, it also sho