Implicit Neural Representation Networks for Fitting Signals, Derivatives, and Integrals скачать в хорошем качестве

Implicit Neural Representation Networks for Fitting Signals, Derivatives, and Integrals

Julien Martel, Postdoctoral Research Fellow at Stanford University in the Computational Imaging Lab David B. Lindell, Postdoctoral Scholar at Stanford University and incoming Assistant Professor in the Dept. of CS at University of Toronto For more details: https://www.meetup.com/SV-SIGGRAPH/ev... Abstract Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm, offering many possible benefits over conventional representations and new capabilities in neural rendering and view synthesis. However, conventional network architectures for such implicit neural representations are incapable of modeling signals at scale with fine detail and fail to represent derivatives and integrals of signals. In this talk, we describe three recent approaches to solve these challenging problems. First, we introduce sinusoidal representation networks or SIREN, which are ideally suited for representing complex natural signals and their derivatives. Using SIREN, we can represent images, wavefields, video, sound, and their derivatives, allowing us to solve differential equations using this type of neural network. Second, we introduce a new framework for solving integral equations using implicit neural representation networks. Our automatic integration framework, AutoInt, enables the calculation of any definite integral with two evaluations of a neural network. This allows fast inference and rendering when applied to neural rendering techniques based on volume rendering. Finally, we introduce a new architecture and method for scaling up implicit representations, called Adaptive Coordinate Networks (ACORN). The approach relies on a hybrid implicit–explicit representation and a learned, online multiscale decomposition of the target signal. We use ACORN to demonstrate the first experiments that fit gigapixel images to nearly 40 dB peak signal-to-noise ratio (an 1000x increase in scale over previous experiments), and we reduce training times for 3D shape fitting from days to hours or minutes while improving memory requirements by over an order of magnitude. Bios Julien Martel (http://www.jmartel.net/) is a Postdoctoral Research Fellow at Stanford University in the Computational Imaging Lab led by Gordon Wetzstein. His research interests are in unconventional visual sensing and processing. More specifically, his current topics of research include the co-design of hardware and algorithms for visual sensing, the design of methods for vision sensors with in-pixel computing capabilities, and the use of novel representations for visual data such as neural implicit representations. David B. Lindell (https://davidlindell.com) is a postdoctoral scholar at Stanford University and an incoming Assistant Professor in the Department of Computer Science at University of Toronto. He received a PhD in Electrical Engineering from Stanford University and is the recipient of the ACM SIGGRAPH 2021 Outstanding Dissertation Honorable Mention Award. His research spans the areas of computational imaging, computer vision, and machine learning with a focus on new methods for active 3D imaging and physics-based machine learning. #NeuralRendering Joint Event with SF Bay ACM Chapter https://www.meetup.com/SF-Bay-ACM/ 0:00 ACM Chapters Introductions 2:28 Speaker Introductions 3:26 Presentation 4:15 Signals: Images, Shapes, Audio 6:34 Differential Equations / Derivatives 7:21 SIREN: Sinusoidal Representation Networks 9:25 Related Work 10:28 SIREN Examples 11:20 Images 15:00 Audio 21:27 Videos 24:29 Poisson equation 27:15 Eikonal equation 31:13 Helmholtz equation 32:49 Wave equation 33:34 Learning priors over the space of SIREN functions 34:43 Like discrete grid or point clouds, SIREN is a data representation 35:19 With a number of benefits 37:18 Challenges Towards Large Scale Neural Representations 38:59 Challenges: Explicit / Implicit vs Efficiency / Multiscale / Pruning Chart 48:55 ACORN: an hybrid implicit-explicit architecture 49:56 Image Fitting Example (16 MP) 50:39 Scaling Up (64 MP) 50:56 ACORN Architecture 57:17 Gigapixel Image Fitting 59:06 Large-Scale 3D Shapes 1:00:36 Key question: How to operate on signals represented with coordinate-based networks? 1:02:05 AutoInt: Automatic Integration for Coordinate-based Networks 1:05:12 Volume Rendering Integration 1:06:01 Coordinate-based networks 1:06:34 Differentiation vs Integration 1:07:35 Numerical Integration Techniques 1:08:08 Automatic integration 1:08:31 Integral Network 1:10:06 AutoInt steps 1:10:45 Implementation 1:11:05 Implementation in a “compiler” 1:11:53 Example 1:12:38 Example Tomography 1:16:37 Volume Rendering Equation (VRE) 1:18:59 Approximation of the VRE 1:19:22 AutoInt Examples 1:21:45 AutoInt: open questions 1:23:47 Summary 1:26:00 Contacts, Publications, References, Collaborators 1:26:43 More Q&A