UNRAVELING THE OPTIMIZATION LANDSCAPE AND TRAINING DYNAMICS OF NEURAL NETWORK SOLVERS FOR PARTIAL DIFFERENTIAL EQUATIONS
Keywords:
Neural Network Solvers, Partial Differential Equations, Physics-Informed Neural Networks (PINNs), Deep Ritz Method, Deep Galerkin Method, Stochastic Gradient Descent (SGD), Adam Optimization, Optimization Landscape, Training Dynamics, Loss Surface, Hyperparameters, Learning Rate, Batch Size, Weight Initialization, Convergence, Stability, Nonlinear PDEs, Machine Learning, Computational Science, Fluid Dynamics, Heat TransferAbstract
This research investigates how neural network solvers are used in Partial Differential Equations (PDEs), particularly the insight the optimization landscape can provide into the training of these solvers. We explore the capabilities of several neural network-based techniques for solving different benchmark PDEs. These techniques include, but are not limited to, the Physics-Informed Neural Network (PINN), Deep Ritz, the Deep Galerkin Method, and so on. The findings show how algorithms affect training optimization. Through optimization algorithms like SGD and Adam, the objective of this paper is to take a closer look at the convergence rate and solution accuracy and stability. We will examine the loss surface of a neural network solver through theoretical analysis and experimental work. We will look at the influence of local minima, saddle points, sharp areas, etc. on these solvers. We also investigate how unstable hyperparameters (like learning rate, batch size, and weight initialization) impact solvers. Overall, the results show that Sasha and Adam outperform SGD in speed and accuracy and can be chosen accordingly. This study can improve the efficiency of neural network tools to solve PDEs It gives a look at how work in the future can help improve these tools especially for more challenging, high-dimensional and multi-physics problems
