\( \mathbb{R}\) denotes the spaces of real numbers. \( \mathbb{R}_{\geq 0}\) (reps. \( \mathbb{R}_{>0}\) ) is the set of (strictly) positive reals. For \( u \in \mathbb{R}^n\) , \( u \leq 0\) is to be understood element-wise and \( | u |^2 = u^{\top} u\) . Operators are written with fraktur letters (e.g. \( \mathfrak{F}, \mathfrak{L}\) ...). \( \mathcal{L}^2(A, B)\) is the space of square integrable functions from \( A\) to \( B\) . For \( U\) a subset of \( \mathbb{R}^n\) , \( \partial U\) represents its boundary. For a discrete set \( D\) , \( |D|\) refers to its cardinality. The bold letters in optimization problems refer to decision variables. For a random variable \( X\) , we denote as \( \mathbb{E}(X)\) its (empirical) expected value and as \( \mathbb{V}(X)\) its (empirical) variance.

Let \( \mathfrak{F} : \mathcal{H} \to \mathcal{L}^2(U, \mathbb{R}^n)\) be a possibly nonlinear operator where \( U\) is an open bounded subset of \( \mathbb{R}^p\) . PINNs are approximating solutions of the ODE/PDE problem defined as

where \( v: U \to \mathbb{R}^n\) belongs to a Sobolev space \( \mathcal{H} \subseteq \mathcal{L}^2(U, \mathbb{R}^n)\) , \( \Gamma\) is a subset of the boundary of \( U\) and \( v_{\Gamma}: \Gamma \to \mathbb{R}^n\) is prescribed as the initial condition and the boundary conditions. This describes a conventional PDE system as in [18, Chap. 3], therefore the problem can be:

1. under-determined: there is at least one solution;
2. well-posed: there is a unique solution;
3. over-determined: there is possibly no solution.

We make the following working assumptions.

Assessing if a PDE problem is well-posed is an active area of applied mathematics [18] that goes far beyond the scope of this article.

This technical assumption will be justified later on but relates the smoothness of the solution \( v\) to (1).

The general problem in PINNs is to approximate the solution \( v\) to (1) using a dense feed-forward neural network \( v_{\Theta}\) with \( L\) layers and a linear output defined by

where \( H_i(u) = \phi(W_i u + b_i)\) , \( \phi \in \mathcal{C}^{\infty}(\mathbb{R}, \mathbb{R})\) being an element-wise activation function, \( b_i\) and \( W_i\) are real matrices of appropriate dimensions. The number of neurons in each layer is denoted \( N \in \mathbb{N}\) . The function \( v_{\Theta}\) is parametrized by the tensor \( \Theta = \{b_1, W_1, \dots, b_L, W_L\}\) .

The vanilla PINN framework proposes to discretize the set \( \Gamma\) into \( D_{\Gamma}\) and then create the data cost as the mean squared error between the measurement points and the neural network \( v_{\Theta}\) evaluated at the same locations:

where \( \left| D_{\Gamma} \right|\) is the cardinal of \( D_{\Gamma}\) .

The original PDE problem (1) then becomes a learning problem as defined in [2, Section 5.1]. However, \( D_{\Gamma}\) is not dense in \( U\) since it is a subset of its boundary. Consequently, the problem, as defined, does not meet the criteria of a well-posed machine learning problem [2]. Therefore, there is little chance that a learning algorithm can find a tensor \( \Theta\) such that \( v_{\Theta}\) is close to \( v\) on the whole domain \( U\) .

The main idea in [1] was to use automatic differentiation to calculate the derivative of \( v_{\Theta}\) with respect to its input. Consequently, it is possible to compute \( \mathfrak{F}[v_{\Theta}]\) . This new information that \( \mathfrak{F}[v_{\Theta}]\) should be zero on any sampling \( D_U\) of \( U\) is added as a regularization agent to help learning. This leads to the extended cost:

where \( \lambda > 0\) and

which is the mean squared error between \( \mathfrak{F}[v_{\Theta}]\) and zero on the discrete set \( D_U\) . With that addition \( D_{\Gamma} \cup D_{U}\) should cover the set \( U\) well, making the following machine learning problem well-posed:

It is proven in [8] for a specific class of PDEs that the loss \( \mathcal{L}_{\lambda}\) has \( 0\) as a global optimum. In the more general case, the following proposition holds.

The proof is available in A.

The following subsection formulates the problem.

Learning algorithms compute a path from an initial state \( \Theta_0\) to a final state \( \Theta_*\) such that the total loss \( \mathcal{L}_{\lambda}\) decreases. The algorithm is convergent if \( \nabla_{\Theta} \mathcal{L}_{\lambda}(\Theta_*) = 0\) , which means that we have reached a local minimum. The learning algorithm is then defined as the mapping \( \mathcal{T}\) from \( \Theta_0\) to \( \Theta_*\) .

The initial state \( \Theta_0\) is a random variable. Some strategies have been developed to improve the convergence of \( \mathcal{T}\) , based on the use of a specific distribution for \( \Theta_0\) , such as the celebrated Glorot initialization (\( \Theta_0 \sim G\) ) for deep neural networks [3] with symmetric activation functions, otherwise, it is preferable to use the He initialization [19]. These two initialization strategies focus on improving the gradient propagation in the network, but other works have been done in the same area considering other metrics such as convergence time [20] or variance-preserving [21]. Other strategies rely on pretraining algorithms to get a warm start [7, 22].

Once convergence is obtained, in the case of PINNs, we are interested in the generalization error defined as:

This is the \( \mathcal{L}^2\) -error between the approximated solution and the real one on the considered domain \( U\) . The efficiency of the learning algorithm is then related to the random variable

Therefore, we define the following quantities.

These definitions can be understood in a logical context from Figure 1. The accuracy metric is the traditional metric used to evaluate PINNs learning algorithm [24] but we want to highlight the stochasticity of the random variable \( E\) which is often neglected. The robustness metric comes from stochastic control theory [25, Chapter 6] where it is used to measure the size of the confidence interval and therefore the robustness with respect to the external parameters of the algorithm. To the best of the authors’ knowledge, this metric has not been properly studied despite its central role when characterizing the efficiency of a learning algorithm. In short, it is important to measure the accuracy of the algorithm as an indicator of how the final approximation \( v_{\Theta_*}\) is close to the real solution \( v\) and to show the approximation capabilities of the neural network. The robustness of the algorithm measures the impact of the initial condition \( \Theta_0\) on the generalization error and, consequently, shows reproducibility. A small robustness value \( R\) means that the generalization error will remain close to the accuracy, while a large value indicates that the generalization error could differ significantly for different values of \( \Theta_0\) .

For PINNs, it has been shown in [8, Theorem 7.1] and in [26] that the global minimum of \( \mathcal{L}_{\lambda}\) can be as close to \( 0\) as desired provided a sufficient number of neurons \( N\) in \( v_{\Theta}\) and Lipschitz continuity of the operator \( \mathfrak{F}\) . Moreover, in that case, the generalization error can be as close to \( 0\) as desired [8]. From the work [27], it seems that the number of local optimums increases with the number of neurons \( N\) but the local optimums tend to converge to the global optimum. In other words, these works strongly suggest that \( A(\mathcal{T})\) goes to \( 0\) with an infinite number of neurons. Since \( E = \mathcal{E}\left(\mathcal{T}(\Theta_0) \right)\) is a positive random variable, a low accuracy \( A(\mathcal{T})\) implies a small robustness metric \( R(\mathcal{T})\) . Consequently, there is evidence that indicates that \( R(\mathcal{T})\) goes to \( 0\) with the number of neurons, and then the performance of the learning algorithm would not depend on the initial condition \( \Theta_0\) .

The previous remarks explain why there has been so little interest in the robustness metric since an accurate training algorithm would naturally induce high robustness. However, in practice, the training algorithm might enter some early stopping conditions due, for instance, to saddle points, exploding or vanishing gradients [2, Section 4.3] or other PINN-specific pathologies [24]. Noise in the measurements or multi-objective problems are usually going in the opposite direction of increased accuracy. In addition, there is not an infinite number of neurons and the ecological impact of machine learning tends to justify a shallow neural network. Consequently, \( A(\mathcal{T})\) is usually strictly positive and sometimes not very small, and robustness must also be evaluated.

The last metric would relate to computational complexity. Indeed, some methods might be accurate and robust, but they require so much computational power that it is impossible to consider them in an engineering application. A rigorous evaluation of the computational complexity is tedious and, in general, very generic. An empirical measurement is the computational time and it is the one chosen in this paper.

Note that it is usually not possible to compute \( A(\mathcal{T})\) or \( R(\mathcal{T})\) and we must use estimators. However, the empirical mean and variance are very sensitive to outliers, especially with little data. Therefore, in numerical examples, we instead recommend computing the median value of \( \mathcal{E}(\mathcal{T}(\Theta_0))\) as its accuracy and its interquartile range (IQR) as its robustness measure.

As explained in [15, 24], multiple attempts have been made to derive algorithms that are more accurate for PINNs using strategies such as reformulation [8, 9], novel architectures [10, 11], new learning paradigms [12, 13] and loss reweighting [14, 15, 16]. In this paper, we focus on the latter class which consists of changing the weight \( \lambda\) during the training of (3) to improve the robustness and accuracy of the learning algorithm. That leads to the following problem.

We also aim to propose a new strategy that is naturally supposed to ensure robustness. We decided to focus on loss reweighting because this method has much potential and is easily generalizable for rapid adoption. Therefore, we will evaluate the accuracy and robustness of loss reweighting strategies and derive a primal-dual learning algorithm that can be interpreted using traditional optimization or renowned learning techniques.

Reweighting strategies in machine learning have been investigated in multiple contexts such as classical mechanics [28], regularization [29], and multiobjective optimization [30], to cite a few. This methodology provides an interesting approach to the problem since it improves the accuracy significantly with minor modifications of the training algorithm. In the specific case of PINNs, changing the weighting during training has led to some improvements in the balancing between data and physics [15, 31]. However, a fair comparison between the different strategies with metrics is missing. In this section, we explore some of the most famous contributions made in this field.

2.3.1 The annealing method

2.3.2 The inverse Dirichlet weighting

2.3.3 The neural tangent kernel method

2.3.4 Other methods

Approximating a weak solution \( v\) to the original problem (1) is equivalent to solving the following one on \( v_{\Theta}\) :

The Lagrangian [38] related to the previous system is defined as

with \( \lambda > 0\) . Following the Lagrangian theory, the constrained optimization problem defined in (13) can be relaxed to the following unconstrained \( \min\) -\( \max\) problem:

The primal-dual approach consists in solving these two problems:

and

Instead of solving a sequence of primal-dual problems as proposed in [38], the main idea is to alternate between solving the primal and the dual to get close to a local optimum of the original constrained problem [44]. This methodology has been investigated in other settings by the authors of [45, 46] who considered solving these two problems in parallel to decrease training time.

Basically, problems (16)-(17) are not numerically tractable since they contain integrals. An approach consists of using Monte-Carlo approximations as follows:

where \( D_{\Gamma} \subset \Gamma\) and \( D_{U} \subset U\) are finite sets. If we denote by \( N_{\Gamma}\) and \( N_U\) the number of points in \( D_\Gamma\) and \( D_U\) respectively, the law of large numbers ensures that if the sets \( D_{\Gamma}\) and \( D_{U}\) are uniformly sampled from \( \Gamma\) and \( U\) , both with volume \( 1\) , then

A numerically tractable relaxed version becomes

and

Since \( \nabla_{\lambda} \mathcal{L}_{\lambda}(\Theta) = g(\Theta)\) , the gradient-ascent from equation (20) translates to the instruction in line \( 13\) of Algorithm 1. Consequently, the primal-dual method applied to (19) and (20) leads to Algorithm 1.

The previous subsection justified the use of Algorithm 1 and Proposition 2 ensured convergence in the limit case. We propose here to take other perspectives and justify Algorithm 1 using robustness arguments.

3.2.1 Adversarial learning perspective

3.2.2 Curriculum learning perspective

Even if Algorithm 1 works well in theory, it is, however, slow and tends to oscillate around the local optimum (as any first-order optimization scheme [2, Section 8.5]). A second-order solver is usually faster and does not oscillate as much at the end of the training [2, Section 8.6.1]. However, changing the cost during the training is not straightforward, implying that it is more sensitive to initialization and ill-conditioning.

An idea is to combine the advantages of both solvers by using a first-order method as a pretraining with a limited number of epochs. Since gradient descent algorithms have a convergence rate of \( 1/k\) with \( k\) the number of epochs [52], we get that \( e_k\) is decreasing fast in the first epochs. Bad local minima around the initialization are thus avoided. Using a second-order solver with the warm startup \( \Theta_k\) obtained from the first-order solver enables faster convergence, with increased robustness.

4.1.1 Case study 1: one-dimensional quasi-linear hyperbolic equation

To show the ability to solve time-dependent problems, the first example comes from the traffic state estimation problem [54], which is the simplest macroscopic model of traffic flow and considers the following viscous Burgers’ equation [55]:

\[ \begin{equation} \frac{\partial \rho}{\partial t} + V_f ( 1 - 2 \rho) \frac{\partial \rho}{\partial x} = \gamma^2 \frac{\partial^2 \rho}{\partial x^2} \text{ on } U = [0, 2] \times [0, 5]. \end{equation} \]

(21)

where \( \rho \in [0, 1]\) is the normalized density of vehicles, and \( V_f > 0\) is the free flow velocity. We are interested in the case where we approximate the solution to the hyperbolic problem (\( \gamma = 0\) ), and we therefore select a very small viscosity \( \gamma\) .

The simulation results are obtained with the following initial and boundary conditions:

\[ \begin{equation} \begin{array}{l} \forall t \in [0, 2], ~ \rho(t, 0) = \rho_1, \ \rho(t, 5) = \rho_3, \\ \forall x \in [0, 5], ~ \rho(0, x) = \left\{ \begin{array}{ll} \rho_3 & \text{ if } x > 3, \\ \rho_2 & \text{ if } x \in \left[ 2, 3 \right], \\ \rho_1 & \text{ if } x < 2, \end{array} \right. \end{array} \end{equation} \]

(22)

where \( \rho_1, \rho_2, \rho_3 \in [0, 1]\) are drawn randomly between \( 0\) and \( 1\) . These conditions simulate all kinds of Riemann problems: a shock (discontinuity) or a rarefaction (smooth transition). Since the initial condition is a simple function with three components, all kinds of phenomena might appear and also interfere, leading to a more complex shape. In the case \( \gamma = 0\) , there might be multiple solutions. However, only one is meaningful and is called the entropic solution. It corresponds to the limit when \( \gamma \to 0\) of (21). With this setup, we want to evaluate if the algorithm manages to extract the entropic solution or if it will converge inaccurately to the non-entropic one.

To compute the accuracy of the method, the solution to (21) is approximated using a finite difference numerical scheme. The stability and consistency of the numerical scheme are ensured through the CFL condition [56]. For this problem, the simplicity of the analytical solution suggests sampling with \( N_\Gamma = 2520\) and \( N_U = 3000\) , and using a simple neural network such as a 2-hidden-layer fully connected network with \( 10\) and \( 5\) neurons per layer.

We ran the seven described algorithms with and without pretraining (when meaningful) on this model. To get statistically significant results, we ran each algorithm \( 30\) times with different initial tensor parameters \( \Theta_0\) . To enable a fair comparison, all algorithms are using the same random initial and boundary conditions (22). The results are reported in Figure 5. For a better understanding of the Primal-Dual algorithm, the final loss landscape is plotted in Figure 8. In this figure, the blue line represents the trajectories of the loss during the training.

4.1.2 Case study 2:

The second example, the Helmholtz equation, demonstrates the ability to solve time-independent problems, which has been widely studied in many areas of physics such as acoustics and electromagnetics [57]. We consider the following two-dimension Helmholtz equation:

\[ \begin{equation} \begin{array}{l} \Delta u(x,y) + k^2u(x,y) = q(x,y), ~ (x,y) \in U = [-1,1]^2, \\ u(x,y) = h(x,y), ~ (x,y) \in \partial U, \end{array} \end{equation} \]

(23)

given a source term of the form:

\[ \begin{equation} q(x,y) = -\pi^2 \sin{\pi x}\sin{4\pi y} - (4\pi)^2\sin{\pi x}\sin{4\pi y} + k^2\sin{\pi x}\sin{4\pi y}. \end{equation} \]

(24)

One can easily derive the exact solution to this equation as: \( u(x,y) = \sin{\pi x} \cdot \sin{4\pi y}\) . This example is taken from [4] since it has been shown rather difficult to learn because of the source term. We considered a \( 3\) -hidden-layer neural network with \( 50\) , \( 50\) , and \( 20\) neurons per layer for this problem. The training data are sampled with \( N_\Gamma = 400\) and \( N_U = 500\) points.

The seven described algorithms with and without pretraining were employed to solve this problem. Similar to the first example, we executed each algorithm \( 20\) times with different initial tensor parameters \( \Theta_0\) and sampling seeds to ensure the significance of the statistical results. The results are shown in Figure 9. Meanwhile, we monitored the evolution of the weight \( \lambda\) and the generalization error during the training process for each algorithm. The results are displayed in Figure 12 and Figure 18 respectively. We also gathered results from all simulations and normalized them to get the normalized histogram of the random variable \( E\) (generalization error) from equation (6) in Figure 23.

4.1.3 Case study 3:

The 1D Poisson equation is a fundamental partial differential equation that describes the distribution of a scalar quantity, such as potential or temperature, influenced by a source term in a one-dimensional domain. The general form of the equation is given as:

\[ \Delta u(x) = f(x), ~ x \in U := [-10,10] \]

where \( \Delta\) is the Laplace operator, \( u\) is the unknown solution and \( f\) represents a known source term that introduces excitation. In this case study, we examine the following 1D Poisson’s equation:

\[ \begin{equation} \Delta u(x) = -0.49\sin(0.7x) - 2.25\cos(1.5x), ~ x \in U, ~ u(0) = 1 \end{equation} \]

(25)

where the solution is \( u(x) = \sin{0.7x} + \cos{1.5x} - 0.1x\) . We sampled \( N_\Gamma = 2\) and \( N_U = 400\) points randomly in \( U\) and the network contains \( 2\) hidden layers with \( 30\) neurons per layer. The comparison between the algorithms is available in Figure 24.

4.2.1 Observation

4.2.2 Interpretation

4.2.3 Conclusion

[1] M. Raissi and P. Perdikaris and G. E. Karniadakis Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations Journal of Computational Physics 2019 378 686–707

[2] I. Goodfellow and Y. Bengio and A. Courville Deep Learning MIT Press 2016

[3] X. Glorot and Y. Bengio Understanding the difficulty of training deep feedforward neural networks Proceedings of the thirteenth international conference on artificial intelligence and statistics 2010 249–256 JMLR Workshop and Conference Proceedings

[4] S. Wang and Y. Teng and P. Perdikaris Understanding and mitigating gradient pathologies in physics-informed neural networks SIAM Journal on Scientific Computing 2021 43 5 A3055-A3081

[5] P. Rathore and W. Lei and Z. Frangella and L. Lu and M. Udell Challenges in training PINNs: A loss landscape perspective arXiv preprint arXiv:2402.01868 2024

[6] J. Guo and Y. Yao and H. Wang and T. Gu Pre-training strategy for solving evolution equations based on physics-informed neural networks Journal of Computational Physics 2023 489 112258

[7] S. Li and F. Bragone and M. Barreau and K. Morozovska MILP arXiv preprint arXiv:2501.16153 2025

[8] J. Sirignano and K. Spiliopoulos DGM Journal of computational physics 2018 375 1339–1364

[9] H. Son and J. W. Jang and W. J. Han and H. J. Hwang Sobolev training for physics informed neural networks Communications in Mathematical Sciences 2023 21 6

[10] K. Shukla and A. D. Jagtap and G. E. Karniadakis Parallel physics-informed neural networks via domain decomposition Journal of Computational Physics 2021 447 110683

[11] A. D. Jagtap and K. Kawaguchi and G. E. Karniadakis Adaptive activation functions accelerate convergence in deep and physics-informed neural networks Journal of Computational Physics 2020 404 109136

[12] A. F. Psaros and K. Kawaguchi and G. E. Karniadakis Meta-learning PINN loss functions Journal of computational physics 2022 458 111121

[13] Q. Liu and M. Chu and N. Thuerey ConFIG arXiv preprint arXiv:2408.11104 2024

[14] Lu Lu and Raphael Pestourie and Wenjie Yao and Zhicheng Wang and Francesc Verdugo and Steven G Johnson Physics-informed neural networks with hard constraints for inverse design arXiv preprint arXiv:2102.04626 2021

[15] Z. Hao and S. Liu and Y. Zhang and C. Ying and Y. Feng and H. Su and J. Zhu Physics-informed machine learning: A survey on problems, methods, and applications arXiv preprint arXiv:2211.08064 2022

[16] S. Wang and S. Sankaran and P. Perdikaris Respecting causality is all you need for training physics-informed neural networks Computer Methods in Applied Mechanics and Engineering 2024 421 116813

[17] G. E. Karniadakis and I. G. Kevrekidis and L. Lu and P. Perdikaris and S. Wang and L. Yang Physics-informed machine learning Nature Reviews Physics 2021 1–19

[18] L.C. Evans Partial Differential Equations American Mathematical Society 1998 Graduate studies in mathematics

[19] K. He and X. Zhang and S. Ren and J. Sun Delving deep into rectifiers: Surpassing human-level performance on imagenet classification Proceedings of the IEEE international conference on computer vision 2015 1026–1034

[20] A. M. Saxe and J. L. McClelland and S. Ganguli Exact solutions to the nonlinear dynamics of learning in deep linear neural networks arXiv preprint arXiv:1312.6120 2013

[21] LeCun, Y. A. and Bottou, L. and Orr, G. B. and Müller, K.-R. Efficient BackProp Springer Berlin Heidelberg 2012

[22] D. Hendrycks and K. Lee and M. Mazeika Using Pre-Training Can Improve Model Robustness and Uncertainty Proceedings of the 36th International Conference on Machine Learning 2019 Chaudhuri, Kamalika and Salakhutdinov, Ruslan 97 Proceedings of Machine Learning Research 2712–2721 PMLR

[23] P. Simard and B. Victorri and Y. LeCun and J. Denker Tangent Prop - A formalism for specifying selected invariances in an adaptive network Advances in Neural Information Processing Systems 1991 Moody, J. and Hanson, S. and Lippmann, R. P. 4 895–903 Morgan-Kaufmann

[24] S. Wang and S. Sankaran and H. Wang and P. Perdikaris An expert's guide to training physics-informed neural networks arXiv preprint arXiv:2308.08468 2023

[25] Å Introduction to stochastic control theory Courier Corporation 2012

[26] Y. Shin and J. Darbon and G. E. Karniadakis On the convergence and generalization of physics informed neural networks arXiv preprint arXiv:2004.01806 2020

[27] A. Choromanska and M. Henaff and M. Mathieu and G. B. Arous and Y. LeCun The loss surfaces of multilayer networks Artificial intelligence and statistics 2015 192–204 PMLR

[28] D. Ocampo and D. Posso and R. Namakian and W. Gao Adaptive loss weighting for machine learning interatomic potentials Computational Materials Science 2024 244 113155

[29] H. Zhao and Y.-H. H. Tsai and R. R. Salakhutdinov and G. J. Gordon Learning neural networks with adaptive regularization Advances in Neural Information Processing Systems 2019 32

[30] O. Sener and V. Koltun Multi-task learning as multi-objective optimization Advances in neural information processing systems 2018 31

[31] F. M. Rohrhofer and S. Posch and C. Gößnitzer and B. C. Geiger On the apparent Pareto front of physics-informed neural networks IEEE Access 2023

[32] S. Maddu and D. Sturm and C. L. Müller and I. F. Sbalzarini Inverse Dirichlet weighting enables reliable training of physics-informed neural networks Machine Learning: Science and Technology 2022 3 1 015026

[33] S. Wang and X. Yu and P. Perdikaris When and why PINNs fail to train: A neural tangent kernel perspective Journal of Computational Physics 2022 449 110768

[34] H. Chen and G. E. C. Flores and C. Li Physics-informed neural networks with hard linear equality constraints Computers & Chemical Engineering 2024 189 108764

[35] W. Karush Minima of functions of several variables with inequalities as side constraints M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago 1939

[36] H. W. Kuhn and A. W. Tucker Nonlinear programming Traces and emergence of nonlinear programming Springer 2014 247–258

[37] H. Son and S. W. Cho and H. J. Hwang Enhanced physics-informed neural networks with augmented Lagrangian relaxation method (AL-PINNs) Neurocomputing 2023 548 126424

[38] Dimitri P Bertsekas Constrained optimization and Lagrange multiplier methods Academic press 2014

[39] Z. Xiang and W. Peng and X. Liu and W. Yao Self-adaptive loss balanced physics-informed neural networks Neurocomputing 2022 496 11–34

[40] D. C. Liu and J. Nocedal On the limited memory BFGS method for large scale optimization Mathematical programming 1989 45 1 503–528

[41] P. Wolfe Convergence conditions for ascent methods SIAM review 1969 11 2 226–235

[42] C. Liu and L. Zhu and M. Belkin Loss landscapes and optimization in over-parameterized non-linear systems and neural networks Applied and Computational Harmonic Analysis 2022 59 85–116

[43] C. Liu and D. Drusvyatskiy and M. Belkin and D. Davis and Y. Ma Aiming towards the minimizers: fast convergence of SGD for overparametrized problems Advances in neural information processing systems 2024 36

[44] M. X. Goemans and D. P. Williamson The primal-dual method for approximation algorithms and its application to network design problems Approximation algorithms for NP-hard problems 1997 144–191

[45] Fioretto, F. and Van Hentenryck, P. and Mak, T. W. K. and Tran, C. and Baldo, F. and Lombardi, M. Lagrangian Duality for Constrained Deep Learning Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track 2021 118–135 Springer International Publishing

[46] Y. Nandwani and A. Pathak and P. Singla A Primal Dual Formulation For Deep Learning With Constraints Advances in Neural Information Processing Systems (NeurIPS) 2019 Wallach, H. M. and Larochelle, H. and Beygelzimer, A. and d'Alché-Buc, F. and Fox, E. B. and Garnett, R. 12157–12168

[47] I. Goodfellow and J. Pouget-Abadie and M. Mirza and B. Xu and D. Warde-Farley and S. Ozair and A. Courville and Y. Bengio Generative adversarial nets Advances in neural information processing systems 2014 27

[48] Szegedy Intriguing properties of neural networks 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings 2014 Yoshua Bengio and Yann LeCun

[49] Y. Bengio and J. Louradour and R. Collobert and J. Weston Curriculum learning Proceedings of the 26th annual international conference on machine learning 2009 41–48

[50] P. Chaudhari and A. Choromanska and S. Soatto and Y. LeCun and C. Baldassi and C. Borgs and J. Chayes and L. Sagun and R. Zecchina Entropy-SGD: Biasing gradient descent into wide valleys Journal of Statistical Mechanics: Theory and Experiment 2019 2019 12 124018

[51] H. Li and Z. Xu and G. Taylor and C. Studer and T. Goldstein Visualizing the loss landscape of neural nets Advances in neural information processing systems 2018 31

[52] Y. Nesterov Introductory lectures on convex optimization: A basic course Springer Science & Business Media 2013 87

[53] D. P. Kingma and J. Ba Adam: A Method for Stochastic Optimization CoRR 2014 abs/1412.6980

[54] M. Barreau and M. Aguiar and J. Liu and K. H. Johansson Physics-informed Learning for Identification and State Reconstruction of Traffic Density Conference on Decision and Control (CDC) 2021 2653–2658

[55] J. M. Burgers The nonlinear diffusion equation: asymptotic solutions and statistical problems Springer Science & Business Media 2013

[56] R. Courant and K. Friedrichs and H. Lewy On the partial difference equations of mathematical physics IBM journal of Research and Development 1967 11 2 215–234

[57] J.-Claude. Nédélec Acoustic and electromagnetic equations: integral representations for harmonic problems Springer 2001 144

[58] M. Barreau and J. Liu and K. H. Johansson Learning-based State Reconstruction for a Scalar Hyperbolic PDE under noisy Lagrangian Sensing Learning for Dynamics and Control 2021 144 1-15 PMLR

[59] K. Hornik Approximation capabilities of multilayer feedforward networks Neural networks 1991 4 2 251–257

[60] C. Daskalakis and I. Panageas The limit points of (optimistic) gradient descent in min-max optimization Advances in neural information processing systems 2018 31