Documents Live, a web authoring and publishing system
If you see this, something is wrong
Table of contents
First published on Thursday, Jul 2, 2026 and last modified on Thursday, Jul 2, 2026 by François Chaplais.
School of Mathematics and Statistics, Beijing Institute of Technology, China and Department of Computer Science & UCL AI Centre, University College London, UK. Work done during visiting University College London
School of Professional Education and Executive Development The Hong Kong Polytechnic University, China
School of Mathematics and Statistics, Beijing Institute of Technology, China
Department of Computer Science & UCL AI Centre, University College London, UK. Corresponding author. Email
Abstract
Physics-informed neural networks (PINNs) are powerful surrogates for differential equations but are notoriously difficult to train due to spectral bias, stiffness, and poor accuracy on high-frequency or multiscale solutions. Adversarial training based on generative adversarial networks (GANs) has recently gained surprisingly strong empirical results in improving the training, but the underlying mechanisms remain elusive. To this end, we propose a new analysis framework for adversarially trained PINNs, based on the key observation of how the discriminator in GANs can influence the training dynamics of PINNs. The framework first provides a much needed theoretical grounding to why and when adversarial training is effective in PINNs, then presents a unified analysis of GANs variants in such training, and finally leads to a new, practical, efficient training algorithm for PINNs. Empirical results demonstrate that our method can significantly reduce the pathology of PINNs training, thereby providing better models with superior performance, often several magnitudes more accurate than alternative methods.
1 Introduction
Partial Differential Equations (PDEs) are foundational modeling tools in science and engineering, but designing stable and efficient solvers for them is challenging, particularly for complex systems characterized by strong nonlinearity, stiffness, or multiscale structures. Neural networks recently emerged as a promising alternative for solving PDEs, where physics-informed neural networks (PINNs) have become widely used, achieving remarkable success across a broad range of applications. However, PINNs are notoriously hard to train, especially when applied to PDEs with solutions exhibiting high-frequency or multiscale features [1, 2, 3]. Recent studies have systematically analyzed these failures and attributed the pathological behavior of PINNs to multiscale interactions and spectral bias among different loss components [4, 5]. In practice, this manifests itself as a stagnation of the training loss near stiff regions of the PDE where the network prediction deviates significantly from the ground truth. A recent attempt to mitigate this pathology is to leverage adversarial training via generative adversarial networks (GANs), i.e. adversarial PINNs training [6, 7, 8, 9, 10].
GANs consist of a generator and a discriminator, trained via a minmax game [11]. When applied to PINNs, the prediction errors, e.g. PDE residual, boundary, or/and initial condition errors, are modeled as generated samples, while the ideal zero error is treated as the real target. In other words, training amounts to driving the empirical distribution of the prediction error toward a Dirac measure centered at zero. A growing body of work has demonstrated the empirical effectiveness of this idea, including employing vanilla GANs with hard constraints [6], Wasserstein GANs with penalty-based methods [7] for PDEs and Stochastic Differential Equations [8]. More recent research also investigates using separate discriminators for different loss terms [9] and balancing physics constraints and control objectives in PDE-constrained optimal control [10].
Despite empirical successes, the theoretical grounding of adversarial PINNs training is missing, especially in why GANs succeed or fail in some settings. This has led to inconsistent or even contradictory observations and analysis. For example, Ciftci et al. [7] and Yang et al. [8] argued that vanilla GAN training is essentially Jensen-Shannon divergence minimization under suitable assumptions, and therefore should be avoided in PINN training due to its well-known instability, but Bullwinkel et al. [6] showed that vanilla GANs can in practice drive PINNs to extremely accurate solutions. More broadly, the generated distribution of adversarially trained PINNs is often a finite Dirac mixture supported on the PDE residual samples, whereas the target distribution is a Dirac measure with singular support at zero. Under this support mismatch, the classical divergence-based theory [12, 13] provides little useful guidance and may even suggest degenerate training behaviors, which is clearly inconsistent with empirical evidence. Overall, the above discrepancies show that existing analysis which is mainly based on static optimal-discriminator behaviors is insufficient to explain the adversarial PINNs training. A dynamical perspective is required.
We draw inspiration from the gradient-flow analysis of neural networks, specifically through the lens of Neural Tangent Kernel (NTK) [14], to characterize the dynamics of adversarial PINNs training. NTKs have been employed successfully in GANs [15] to show that the study of the alternating updates between the generator and the discriminator is essential to understand the training dynamics. We generalize this analysis to adversarial PINNs training to reveal: (1) how discriminator training shapes the dynamics of PINNs, (2) how different GAN objectives induce different sample-wise weighting, and (3) how these effects interact with the generator training dynamics. These phenomena jointly explain the convergence (or the lack of) of adversarial PINNs training.
Overall, our main contributions include:
- A new unified NTK framework for analyzing adversarially trained PINNs.
- A formal analysis of adversarial PINNs training under different GAN variants.
- New findings in how the discriminator shapes the spectral dynamics of PINNs training.
- A new training algorithm with improved optimization stability and convergence behavior.
2 Preliminaries
2.1 Generative Adversarial Networks
GAN consists of a generator and a discriminator. We denote the discriminator by \( f(\cdot;\phi):\mathcal X\to\mathbb R \), where \(\phi\) denotes the discriminator parameters. The generator \( g(z;\theta)\) is parameterized by \(\theta\), where \( z \sim p_z\) is a latent variable. Let \(\alpha_r\) and \(\alpha_g^{\theta}\) denote the real and generated data distributions on \(\mathcal X\), respectively. GAN training can be seen as a minmax game:
(1)
(2)
where \(P,Q,R:\mathbb R\to\mathbb R\) are scalar functions. In particular, we consider three GAN variants:
(3)
(4)
and
(5)
2.2 Hard-constrained PINNs
Let \(\Omega\subset\mathbb R^d\) be the space–time domain and consider the PDE
(6)
where \(\mathcal N\) is the differential operator, \(a(x)\) is the source term, and \(\mathcal B\) denotes the boundary and/or initial constraint operator. Through a proper hard-constraint parameterization [17, 18], the neural solution \(u_\theta\) satisfies \(\mathcal B(u_\theta)(x)=b(x)\) exactly. Therefore, training reduces to minimizing the interior residual \( \mathcal R(x;\theta):=\mathcal N[u_\theta](x)-a(x).\) The hard-constrained PINN loss is then given by
(7)
where \(\mu_\Omega\) denotes the sampling distribution over the interior collocation domain.
2.3 Adversarially trained PINNs
Given collocation points \(\{x_i\}_{i=1}^{N_r}\subset\Omega\), define residual \(r_i:=\mathcal R(x_i;\theta)\) and \(\mathbf r:= (r_1,\dots,r_{N_r})^\top\in\mathbb R^{N_r}\). Furthermore, we define \(\hat\mu_\Omega:=\frac{1}{N_r}\sum_{i=1}^{N_r}\delta_{x_i}\) to be the empirical collocation measure, where \(\delta_{x_i}\) is the Dirac measure at \(x_i\). Then the generated residual distribution is the pushforward empirical measure \( \hat\mu_\theta^{\,r} := (\mathcal R_\theta)_\#\hat\mu_\Omega = \frac{1}{N_r}\sum_{i=1}^{N_r}\delta_{r_i}, \) where \(\mathcal R_\theta:x\mapsto \mathcal R(x;\theta)\). The ideal target residual distribution is the Dirac measure at zero, \(\mu_{\mathrm{real}}=\delta_0\). Equivalently, we define the target residual map \( \mathcal T(x_i):=0, i=1,\dots,N_r, \) which represents the desired zero-residual state (real data). Therefore, the adversarial PINNs training objectives become:
(8)
and
(9)
Here \(P,Q,R\) are the same as in Equation 1, Equation 2. The objectives in Equation 8, Equation 9 will be the starting point of our analysis.
2.4 Neural tangent kernel regime
Our analysis is carried out in the neural tangent kernel (NTK) regime. At a high level, this regime describes sufficiently wide neural networks trained in a lazy-training manner, so that the network output can be approximated by its first-order expansion around initialization, while the associated kernel remains nearly constant during training.
We adopt the following standard assumptions:
Assumption 1 ([14])
(i) The network is a standard architecture, such as a fully connected, convolutional, or residual network, with standard nonlinearities. (ii) The activation function is sufficiently smooth, and the network output is differentiable with respect to the parameters. (iii) Linear layers contain nonzero bias terms. (iv) The network parameters are randomly initialized with a standard width-dependent scaling. (v) The hidden widths tend to infinity.
Under these assumptions, if \(g(x;\theta)\) denotes the network output, then around initialization \(\theta_0\),
(10)
and the NTK is defined by
(11)
For a finite set of training points, the output dynamics under gradient flow are then approximated by
(12)
where \(K\) is the NTK Gram matrix. Therefore, in the NTK regime, training can be analyzed in function space through an approximately fixed kernel rather than only through the original nonlinear parameter dynamics. A more detailed discussion of the related works and their connections to the present framework is provided in Appendix B.
3 The Unreasonable Successes in Adversarial PINNs Training
In vanilla GAN, it is known that the generator’s objective is, up to constants, minimizing the Jensen–Shannon divergence between the generated \( \hat{\mu}_\theta^{\,r}\) and the target \( \delta_0\) , under proper assumptions. Similar conclusions also arise from other analysis based on e.g. KL-type [12] and \( \chi^2\) -type [13] criteria. All analyses point to one phenomenon: when the supports of the generated and target distributions are nearly disjoint and the discriminator becomes nearly optimal, the generator receives a degenerate training signal. This issue is particularly severe in adversarial PINNs. At the beginning of training, the residual samples \(\{r_i\}_{i=1}^{N_r}\) are typically far from zero, so the support of \(\hat{\mu}_\theta^{\,r}\) is almost disjoint from that of \(\delta_0\). Theoretically, one would expect the training signal for the generator to saturate or become uninformative. In reality, existing adversarial PINNs methods often perform well. This discrepancy indicates that the classical divergence-minimization framing does not directly apply to PINNs, as it assumes the generator is always trained against an optimal discriminator, whereas practical adversarial PINNs training uses a sub-optimal discriminator. As a result, the generator is driven not by the gradient of a fixed divergence alone, but also by the gradient field induced by a transient discriminator state.
As a motivating example, we show how the generator–discriminator update scheme can significantly influence the training, using the two-dimensional Laplace equation:
(13)
with boundary conditions
(14)
and a hard-constrained PINNs. We vary the generator–discriminator update ratio by:
and show the training dynamics in Fig. 1.
Fig. 1 shows that both over-trained discriminators and over-trained generators can harm adversarial PINNs optimization. When \(D\) is updated too many times relative to \(G\), it may become nearly optimal for the current residual distribution, causing the learning signal passed to \(G\) to become saturated, weak, or poorly conditioned. Conversely, when \(G\) is updated too many times relative to \(D\), the discriminator remains under-trained and cannot provide a reliable gradient. In both cases, the imbalance leads to deteriorated training and validation errors. Therefore, the U-shaped curve in Fig. 1 provides direct evidence that the success of adversarial PINNs is governed by finite-step alternating dynamics between \(G\) and \(D\), rather than by the static optimal-discriminator alone. The best performance appears in an intermediate regime where \(D\) is informative but not over-trained, and \(G\) is updated sufficiently but does not outrun the discriminator.
These observations naturally raise two questions: how can we formally characterize such finite-step alternating dynamics, and how can we use this characterization to choose or control the generator–discriminator update schedule? Addressing these questions is the main goal of the following analysis.
4 NTK Analysis of Adversarially Trained PINNs
We establish a dynamical analysis of the alternating optimization of adversarially trained PINNs. To this end, we first formulate the discriminator-induced residual-space gradient field, then show how it influences the training of the generator.
4.1 Discriminator-induced gradient field in the NTK regime
Given a fixed generator with \( \theta\) and the objective in Equation 9, the discriminator gradient flow is:
(15)
If we define the discriminator NTK as:
(16)
then NTK theory implies that \( k_t^D(x,y)\approx k^D(x,y)\) during training. Therefore, for any residual-space input \( x\) , the discriminator admits the function-space evolution:
(17)
NTK regularity implies that \( k^D(\cdot,y)\) is differentiable with respect to its first argument, so \( f(\cdot;\phi(t))\) is differentiable w.r.t. the input; see Appendix C.
Lemma 1 (Input-gradient representation of the discriminator [15])
Under Assumptions 1 the discriminator input gradient satisfies
(18)
(19)
where \( f_0(x):=f(x;\phi(0))\) .
Remark 1
Lemma 1 describes the evolution of the discriminator gradient field w.r.t. the input during training. It shows that \(\partial_r f(r;\phi(t))\) is governed by the derivatives of the discriminator objective and the derivatives of the discriminator NTK kernel \(k^D\) w.r.t. the input. Therefore, the residual-space gradient field produced by the discriminator is not arbitrary, but is structured by the discriminator architecture, objective, and tangent kernel. This discriminator gradient field will later enter the generator dynamics through the sample-wise weighting (see Appendix C). The corresponding NTK-based discriminator dynamics for vanilla GAN, LSGAN, and IPM-type GANs are given in Appendix C.1–C.3.
4.2 Generator residual dynamics and objective-dependent weighting
For the generator, given the generator objective in Equation 8, the generator gradient flow is:
(20)
We further define the generator residual NTK:
(21)
and a discriminator-induced weighting vector
(22)
where
(23)
Proposition 1 (Residual dynamics under adversarial training)
Under the generator gradient flow (20), the residual vector evolves according to
(24)
Remark 2
Equation (24) shows that adversarial PINNs training takes the form of a preconditioned residual descent. The generator residual NTK \( K_{rr}^G(t)\) determines how a weighted residual signal is propagated across collocation points, whereas the discriminator enters through the sample-wise weighting vector \( \boldsymbol{\gamma}(t)\) . In other words, the discriminator reshapes the residual descent field through its input gradient rather than through its scalar output alone. Detailed derivations of the generator–discriminator update dynamics, together with the corresponding propositions and theorems, are provided in Appendix C.4.
Remark 3
The NTK-based generator dynamics above are derived for the hard-constrained setting with the residual itself as the discriminator input. When the discriminator input is changed from the residual to squared residual, the dynamics become fundamentally different and take a closed-loop multiplicative form. The detailed derivation of the dynamics based on squared residual input and the corresponding residual-energy law under hard constraints is provided in Appendix E.
Remark 4
Beyond the hard-constrained setting, Appendix F extends the analysis to soft-constrained adversarial PINNs, where we derive the coupled residual and energy dynamics for interior, boundary, and initial-condition violations, and further motivate our research.
Remark 5
More broadly, we emphasize that the present framework is not limited to standard GAN variants. Existing adversarially trained PINNs such as SA-PINN [19], LA-PINN[9], and WAN[20] are all special cases of our analysis with the discriminator optimization restricted to structured function classes; the corresponding derivations are given in Appendix G. This perspective also reveals a much broader and still underexplored design space of adversarial PINNs methods, which we summarize in Appendix H.
4.3 Residual-energy Law, Spectral Interpretation, and Practical Optimization
To characterize the optimization effect of the discriminator-induced weighting \( \boldsymbol{\gamma}(t)\) (Equation 22), we define a residual energy \( E(t):=\frac12\|\mathbf r(t)\|^2\) .
Theorem 1 (Residual-energy law under adversarial training)
Under the residual dynamics in Equation 24, the residual energy satisfies
(25)
where \( S(t)\) is the instantaneous residual-energy variation rate induced by the current discriminator state. In particular,
(26)
Theorem 1 shows that the first-order effect of adversarial interaction on the generator optimization is fully characterized by the scalar quantity \( S(t)\) . To interpret this quantity from a modal viewpoint, we consider the constant-NTK regime \( K_{rr}^G(t)\approx K_{rr}^G\) , decompose it as
(27)
and define \( \widetilde{\mathbf r}(t):=U^\top \mathbf r(t), \widetilde{\boldsymbol{\gamma}}(t):=U^\top \boldsymbol{\gamma}(t).\) Then we obtain
(28)
Remark 6 (Energy decay rate and spectral analysis)
By Equation 25, Equation 28, the decay of \(E(t)\) is jointly determined by the generator spectrum \(\{\lambda_j\}\), the residual amplitudes \(\widetilde r_j(t)\), and the discriminator-induced weights \(\widetilde\gamma_j(t)\). As training proceeds, modes with large \( \lambda_j\) s decay fast, then the energy decay increasingly depends on whether \( \widetilde\gamma_j(t)\) remains aligned with the dominant modes. If it aligns with already decayed modes thereby underweighting the others, the energy decay slows down.
Remark 7 (Representative failures)
The analysis explains several representative failures in adversarial PINNs training: (i) energy increase, where \(S(t)>0\); (ii) vanishing discriminator gradients, where \(\boldsymbol{\gamma}(t)\approx 0\); and (iii) noisy or misaligned weighting, where \( \widetilde\gamma_j(t)\) becomes unstable or ineffective.
More detailed analysis and justifications for Remark 6 and 7 are in Appendix D.
To systematically address the above possible failures, one could explicitly control individual modes, where modes would need to be monitored during training and the discriminator feedback would need to be adjusted. However, a straightforward solution from Equation 25 is to ensure the continuous decrease of \( E(t)\) , i.e. \( S(t) < 0\) . Therefore, we propose a practical approach below.
We denote by \( \theta^m\) the generator parameters at the beginning of the \( m\) -th iteration, and define residual
(29)
After a discriminator substep, we denote the frozen discriminator by \( f^{m+\frac12}\) , and define
(30)
We also let \( K_{rr}^{G,m}\) denote the NTK at \( \theta^m\) .
Proposition 2 (One-step discrete residual-energy variation)
A first-order generator update with frozen discriminator yields
Consequently, if \( E^m:=\frac12\|\mathbf r^m\|^2\) , then
Algorithm 1 Adaptive Alternating Adversarial PINNs training with Rollback
From Proposition 2, we derive Algorithm 1, which is essentially a dynamic update scheme for both the generator and the discriminator. Its key principle is that the discriminator should be retained only when it can induce a residual weighting that is able to improve the generator’s ability to reduce the residual. At the same time, the generator update should be conducted only when the residual energy actually decreases. In this way, the discriminator acts not as an always accepted adversary but as an adaptive controller to balance the residual modes for continuous residual-energy decay.
5 Experiments
5.1 Experimental setup
We compare DEQGAN [6], GAN [11], LSGAN [13], and WGAN-GP [16] with their variants enhanced versions GAN-RB, LSGAN-RB, and WGAN-GP-RB, where “RB” denotes our rollback strategy. The benchmark suite includes the Poisson, Laplace, viscous Burgers, Reaction–Diffusion, and Klein–Gordon equations, covering linear and nonlinear, stationary and time-dependent PDEs.
For fairness, all methods share the same generator architecture, while the discriminator architecture is chosen according to the adversarial objective. Except for DEQGAN, which follows its original tuning-dependent protocol, all methods are trained under the same default setting. We report training and validation errors as the main accuracy metrics, and additionally monitor the residual energy and the first-order quantity \(r^\top K_{rr}^{G}\gamma\). Full implementation details are deferred to Appendix I.1.
5.2 Mechanistic validation: when and why adversarial training improves PINNs
| \(G:D\) | Positive ratio | Positive mean | Max value | \(L^1\) violation | \(L^2\) violation |
| \(1:1\) | 0.0220 | 18.1644 | 322.8066 | 1180.6849 | 190919.5174 |
| \(10:1\) | 0.0075 | 0.1429 | 1.8219 | 3.1449 | 4.0658 |
| \(100:1\) | 0.0661 | 1.0747 | 75.9468 | 209.5676 | 10192.1785 |
We revisit DEQGAN under different generator–discriminator update ratios (Fig. 1) to clarify a central question of this paper: when adversarial training improves PINNs, and why. Rather than merely comparing schedules, we use the first-order quantity \(r^\top K_{rr}^{G}\gamma\) to distinguish successful alternating regimes from failed ones.
A clear pattern emerges from Fig. 2. Figure 2 Left shows three ratios \(G:D=100:\!1\), \(10:\!1\), and \(1:\!1\) which are successful regimes because their \(r^\top K_{rr}^{G}\gamma\) curves remain predominantly negative. This indicates that, on average, the discriminator-induced signal is aligned with residual descent after modulation by the generator kernel. However, the behaviors of the three are not identical. Table 1 reports aggregated statistics of the positive part of \(r^\top K_{rr}^{G}\gamma\), including the positive ratio, positive mean, maximum value, and \(L^1/L^2\) violation measures. \(G:D=10:\!1\) yields the smallest positive ratio and the weakest violation of the energy descent requirement, indicating the best overall performance. By contrast, although \(G:D=1:\!1\) has a smaller positive ratio than \(100:\!1\), its other indicators are much larger. So, successful adversarial training depends not only on whether \(r^\top K_{rr}^{G}\gamma\) is mostly negative, but also on how strongly it deviates from the energy descent regime when violations occur.
Figure 2 Right illustrates three failure modes described in Remark 7. \(G:D=1000:\!1\) corresponds to persistent misalignment: \(r^\top K_{rr}^{G}\gamma\) remains almost always positive between \(10^2\) and \(10^3\), indicating that the discriminator is updated too slowly to provide an effective gradient. Also, \(G:D=1:\!1000\) corresponds to discriminator overtraining: the curve rapidly collapses to zero after roughly 200 steps, showing that many residual samples receive nearly zero gradients, consistent with the classical analysis of vanishing generator gradients under an overly strong discriminator. Next, the intermediate discriminator-dominant ratios \(G:D=1:\!10\) and \(1:\!100\) correspond to unstable weighting: \(r^\top K_{rr}^{G}\gamma\) oscillates repeatedly across positive and negative regions, indicating that the effective alignment direction after modulation by \(K_{rr}^{G}\) is unstable.
Overall, these results support our main claims. Adversarial training can improve PINNs but only when the alternating generator–discriminator dynamics induce a weighting field that is sufficiently aligned with the residual descent. \(r^\top K_{rr}^{G}\gamma\) is useful not only as a strict per-step certificate, but also as an aggregated indicator that separates successful regimes from qualitatively different failure modes.
5.3 Comparison on Diverse PDEs
We next compare the proposed rollback strategy with existing adversarial PINNs baselines on the benchmark suite above. The main results are summarized in Table 2, while detailed per-equation curves are in Appendix I.2.
Four observations are most important. First, LSGAN is already a strong adversarial PINNs baseline, and even the classical GAN performs well under practical alternating optimization, which again suggests that the standard optimal-discriminator divergence picture is insufficient. Second, rollback is particularly effective for GAN and LSGAN: GAN-RB and LSGAN-RB consistently outperform their non-RB counterparts and achieve the best overall results on this benchmark suite. Third, the benefit is not uniform across objectives: rollback is weaker for WGAN-GP, and can even hurt performance on some problems, likely because the gradient penalty in WGAN-GP already regularizes the discriminator and the witness-style objective is less aligned with the proposed first-order criterion. Finally, LSGAN has rarely been explored in adversarial PINNs training so far but appears particularly promising, suggesting substantial room to revisit classical adversarial objectives from the perspective of practical alternating dynamics rather than nominal divergence form alone.
| Equation | DEQGAN | WGAN-GP | WGAN-GP + RB | LSGAN | LSGAN + RB | GAN | GAN + RB |
| Poisson | 9.78e-13 / 9.75e-13 | 1.33e-04 / 1.33e-04 | 7.20e-05 / 7.18e-05 | 1.02e-08 / 1.01e-08 | 5.86e-13 / 5.84e-13 | 4.80e-07 / 4.79e-07 | 8.43e-12 / 8.47e-12 |
| Laplace | 3.65e-11 / 2.79e-11 | 9.04e-06 / 6.30e-06 | 6.51e-05 / 4.29e-05 | 5.86e-10 / 4.25e-10 | 3.49e-12 / 2.74e-12 | 8.12e-06 / 5.58e-06 | 2.39e-11 / 1.69e-11 |
| Viscous-Burgers | 6.49e-06 / 1.84e-04 | 7.36e-02 / 7.38e-02 | 1.84e-01 / 1.84e-01 | 2.35e-02 / 2.37e-02 | 4.34e-05 / 3.49e-04 | 6.48e-04 / 7.78e-04 | 2.57e-06 / 3.87e-04 |
| Reaction-Diffusion | 2.92e-09 / 2.45e-09 | 2.13e-02 / 1.79e-02 | 7.15e-02 / 5.96e-02 | 8.16e-07 / 7.09e-07 | 1.18e-09 / 9.24e-10 | 9.81e-08 / 8.19e-08 | 1.40e-13 / 1.08e-13 |
| Klein-Gordon | 3.79e-06 / 3.16e-06 | 1.62e-02 / 1.35e-02 | 2.39e-04 / 1.78e-04 | 3.91e-03 / 3.23e-03 | 5.45e-05 / 4.36e-05 | 1.65e-02 / 1.40e-02 | 4.75e-10 / 4.02e-10 |
An important advantage of our method is that it avoids expensive hyperparameter search. Except for DEQGAN, all methods are trained with the same default learning rate \(10^{-3}\) and default optimizer settings. By contrast, DEQGAN relies on exhaustively tuning both generator/discriminator hyperparameters. Therefore, our method provides a more efficient route to strong PINNs performance. In addition, the rollback strategy can be used as a plugin in existing adversarial PINNs methods, also demonstrated by additional ablations in the appendix I.3: (i) robustness under modified boundary conditions; (ii) compatibility with DEQGAN, and (iii) controlled comparisons under a fixed \(G:D=20:\!20\) update budget. These results indicate that the observed gains come from the rollback mechanism itself rather than merely from extra update steps.
6 Related work
[5] studied PINNs training from the NTK perspective and explained optimization difficulties such as spectral bias and unbalanced convergence. However, their analysis is restricted to gradient descent on a fixed PINNs loss. By contrast, we study adversarial PINNs training, where the generator residual dynamics are continuously modulated by discriminator feedback. Classical GAN theory explains training through min–max optimization and divergence minimization [13, 12], but this static picture does not explain why adversarial PINNs can still train successfully when the generated residual distribution and the zero target are initially far apart. [15] later analyzed GANs from an NTK viewpoint, but their setting is standard generative modeling rather than PDE residual learning. Our work instead studies adversarial PINNs from a residual-dynamics viewpoint. [7] proposed a physics-informed GAN framework and reported strong empirical performance, while [6] showed that adversarial objectives can improve differential equation solvers. However, these works mainly demonstrate that adversarial training can work well in practice, rather than explaining when and why it works. In contrast, we develop a residual-dynamics framework for adversarial PINNs and, based on it, derive a practical first-order rollback strategy. To the best of our knowledge, this is the first work to provide a unified theoretical framework for analyzing when and why adversarial training improves PINNs.
7 Limitations, Conclusions, and Future Work
There are limitations to our analysis. The NTK regime we employ is limited in that it assumes a fixed NTK. In reality, the kernel can evolve during training depending on the PDEs and the networks have finite widths. Also, training can still oscillate under some PDEs and settings even with our rollback strategy. This is likely to be caused by the NTK assumptions, which needs further investigation.
In summary, we have proposed a new analysis framework to explain the success of adversarially trained PINNs. The framework is rigorous in providing the much needed theoretical grounding, general in unifying the analysis of existing PINNs training, and practical by presenting effective and efficient training guidance. It reconciles the conflicts between the theories in GANs analysis and their empirically successful applications in training PINNs. In future work, we will move away from the constant-NTK approximation and derive a more refined update strategy based on an evolving NTK spectrum. We include detailed discussions and limitations in Appendix J.
Appendix Contents
B Preliminary Knowledge
B.1 Generative adversarial networks
Generative Adversarial Networks (GANs) provide a powerful framework for learning probability distributions through an adversarial game between a generator and a discriminator. In this work, we adopt a unified functional perspective that allows several popular GAN variants to be represented under a common formulation.
Let \( \alpha_r\) denote the real data distribution and \( \alpha_g^{\theta}\) denote the generated distribution induced by the generator \( g(z;\theta)\) , where \( z \sim p_z\) is a latent variable. The discriminator is represented by a function \( f(\cdot ; \phi): \mathcal{X} \rightarrow \mathbb{R}\) . Here, \( \theta\) and \( \phi\) denote the trainable parameters of the generator and discriminator networks, respectively.
We consider the following general adversarial objective:
(31)
(32)
where \( P(\cdot)\) , \( Q(\cdot)\) and \( R(\cdot)\) are scalar-valued functions that determine the specific adversarial divergence or discrepancy being minimized between \( \alpha_r\) and \( \alpha_g^{\theta}\) . This formulation encompasses a broad family of GAN objectives depending on the choice of \( P\) , \( Q\) and \( R\) .
B.1.1 Vanilla GAN.
The vanilla GAN objective corresponds to the minimization of the Jensenv–Shannon divergence between the real and generated distributions. This is recovered by choosing
(33)
where \( \sigma(\cdot)\) denotes the sigmoid function.
B.1.2 Least-Squares GAN (LSGAN).
The least-squares GAN replaces the logistic loss with a quadratic penalty, which improves gradient stability and alleviates saturation. This corresponds to choosing
(34)
B.1.3 Integral Probability Metric (IPM) GANs.
IPM-based GANs measure distribution discrepancy via a supremum over a constrained function class \( \mathcal{F}\) . In this setting, the adversarial objective becomes
(35)
This formulation is recovered from 31-32 by choosing
(36)
together with structural constraints on \( f\) , such as Lipschitz continuity. Prominent examples include Wasserstein GANs (WGAN) and Wasserstein GANs with gradient penalty (WGAN-GP). At this point, \( f\) is often referred to as a critic in the literature. For consistency and ease of understanding, we will uniformly refer to it as the discriminator throughout this paper.
B.2 Physics-informed neural network
Physics-Informed Neural Networks (PINNs) provide a framework for solving partial differential equations (PDEs) by embedding governing physical laws into neural network training. Instead of relying solely on labeled data, PINNs incorporate PDE constraints through automatic differentiation.
Let \( \Omega \subset \mathbb{R}^d\) denote a spatial–temporal domain with boundary \( \partial \Omega\) . We consider a general partial differential equation (PDE) expressed in operator form
(37)
where \( u:\Omega \rightarrow \mathbb{R}\) represents the unknown solution field, \( \mathcal{N}\) denotes a differential operator involving spatial and/or temporal derivatives of \( u\) , and \( a(x)\) is a prescribed source or forcing term defined on \( \Omega\) . The governing equation is complemented by boundary and/or initial conditions given by
(38)
where \( \mathcal{B}\) is a boundary or initial operator specifying constraints on the solution, such as Dirichlet, Neumann, or Robin conditions, and \( b(x)\) denotes the corresponding prescribed data on the boundary or initial manifold. In physics-informed neural networks (PINNs), the unknown solution \( u(x)\) is approximated by a neural network parameterized by \( \theta\) , namely \( u_\theta(x) \approx u(x)\) , allowing the differential operators \( \mathcal{N}\) and \( \mathcal{B}\) to be evaluated through automatic differentiation during the training process.
B.2.1 Physics-informed loss construction.
Let \( \mu_\Omega\) denote a sampling measure over the interior domain \( \Omega\) , and \( \mu_{\partial \Omega}\) denote a sampling measure over the boundary \( \partial \Omega\) . The PINNs objective can be written as
(39)
The expectations are approximated in practice via Monte Carlo sampling over collocation and boundary points.
B.2.2 Hard constraint enforcement.
Besides penalty-based enforcement, physical constraints can also be incorporated directly into the network architecture through hard-constrained parameterizations. Instead of penalizing boundary or initial condition violations, the solution ansatz is parameterized as
(40)
where \( \mathcal{G}(x)\) satisfies the prescribed boundary or initial conditions, and \( \mathcal{H}(x)\) is a function that vanishes on \( \partial \Omega\) . The neural network \( \tilde{u}_\theta(x)\) represents the unconstrained component of the solution.
By construction, the resulting approximation \( u_\theta(x)\) satisfies the constraints exactly for all parameter values \( \theta\) . Consequently, the training objective reduces to minimizing the interior PDE residual
(41)
Hard constraint formulations often improve training stability and reduce sensitivity to loss weighting parameters, although they may introduce additional complexity in constructing suitable constraint functions. For clarity and conciseness in the theoretical derivation and exposition, we focus primarily on the hard-constrained formulation of PINNs (41) in the subsequent analysis.
B.3 Solving Partial Differential Equations via Adversarially Trained PINNs
To mitigate imbalance issues in physics-informed learning, we introduce an adversarial training framework for PINNs based on the vanilla GAN formulation.
Let the PDE residual evaluated at collocation point \( \{x_i\}_{i=1}^{N_r}\) be defined as \( \mathcal{R}(x_i;\theta) := \mathcal{N}[u_\theta](x_i)-a(x_i)\) and \( \mathcal{T}(x_i) := 0.\) Here, \( \mathcal{R}(\cdot;\theta)\) denotes the PDE residual function induced by the generator parameterized by \( \theta\) , and is interpreted as the source of generated data in the adversarial framework. In contrast, \(\mathcal T\) denotes the target residual map corresponding to the ideal zero-residual state. The collocation points \( \{x_i\}_{i=1}^{N_r}\subset\Omega\) are sampled from the interior sampling measure \( \mu_\Omega\) , where \( N_r\) is the number of residual collocation points and \( \Omega\) is the interior domain of the PDE. For notational convenience, define
(42)
and collect these residual values into the residual vector
(43)
Let
(44)
denote the empirical measure associated with the collocation points, where \( \delta_{x_i}\) is the Dirac measure centered at \( x_i\) . Then the generated residual distribution is given by the pushforward empirical measure
(45)
where \( \mathcal{R}_\theta : x \mapsto \mathcal{R}(x;\theta)\) denotes the residual mapping, and \( (\mathcal{R}_\theta)_\# \hat{\mu}_\Omega\) denotes the pushforward of the empirical collocation measure under this map. Therefore, the generated data distribution can be viewed as a finite Dirac mixture supported on the residual samples \( \{r_i\}_{i=1}^{N_r}\) , which serves as a discrete approximation of the pushforward measure \( (\mathcal{R}_\theta)_\#\mu_\Omega\) . In contrast, the real data distribution associated with the ideal PDE-satisfying state is concentrated at zero and is thus represented by the Dirac measure
(46)
B.3.1 Adversarial objective.
We formulate the adversarial training between the solution network \( u_\theta\) and the discriminator network \( f(\cdot;\phi)\) within a general adversarial learning framework. Let \( \mathcal{R}(x;\theta)\) denote the generated PDE residual induced by the solution network and \( \mathcal{T}(x)\) represent the target residual distribution. The generator (solution network) is trained to minimize
(47)
while the discriminator aims to distinguish target samples from generated residuals by maximizing
(48)
where \( P(\cdot)\) , \( Q(\cdot)\) , and \( R(\cdot)\) are scalar loss functions defined in (33), (34) and (36). Different choices of these functions recover various GAN formulations.
In the general setting, the real data correspond to the zero-residual target. For the original GAN and LSGAN formulations, the discriminator is trained to output 1 when the input equals the real data (i.e., \( \mathcal{T}(x_i)=0\) ) and 0 when the input corresponds to generated residual samples (\( \mathcal{R}(x_i;\theta)\) ). In contrast, under the WGAN formulation, the discriminator (critic) is trained to assign larger values to the real data (zero residual) and smaller values to generated residual samples, without restricting its output to the interval \( [0,1]\) . Meanwhile, the generator seeks to reduce the PDE residual magnitude so that the generated samples become indistinguishable from the zero-residual target, thereby enforcing the governing equation.
B.4 Background on the neural tangent kernel regime
This appendix recalls the standard assumptions and conclusions of the neural tangent kernel (NTK) regime used in our analysis. The goal is not to rederive NTK theory from scratch, but to state clearly the conditions under which later kernel-based generator and discriminator dynamics are justified. The presentation follows the standard NTK viewpoint established for wide neural networks, where the network output becomes approximately linearized around initialization and the corresponding kernel remains nearly constant during training [14].
Let \(g(x;\theta)\) denote a neural network with input \(x\in\mathcal X\) and parameters \(\theta\in\mathbb R^P\). We are interested in the regime where gradient-based training of \(g\) can be described directly at the level of its output function, rather than only through the nonlinear parameter trajectory.
Assumption 2 (Network architecture)
The network is a standard architecture, such as a fully connected, convolutional, or residual network, with standard hidden layers and affine transformations.
Assumption 3 (Activation regularity)
The activation function is standard and sufficiently smooth. The activation can be any standard function: tanh, softplus, sigmoid, Gaussian, etc.
Assumption 4 (Nondegenerate bias)
Linear layers contain nonzero bias terms, which is standard in NTK formulations and helps avoid degenerate kernels in certain settings.
Assumption 5 (Initialization)
The network parameters are randomly initialized with a standard width-dependent scaling. In the usual NTK parameterization, the preactivations remain of order one as the hidden widths grow, which ensures a nontrivial infinite-width limit. Under such initialization, the network output at initialization converges in law to a Gaussian process.
Assumption 6 (Infinite-width limit)
The hidden widths \(n_1,\dots,n_{L-1}\) tend to infinity. In this regime, fluctuations due to finite width vanish, and both the network output at initialization and the NTK admit deterministic limiting descriptions.
Under these assumptions, the network output admits the first-order approximation
(49)
This approximation is the basis of the NTK viewpoint: the nonlinear network is replaced by a linearized model around initialization, while the training dynamics are encoded by a kernel.
The corresponding neural tangent kernel is defined by
(50)
For a finite set of points \(\{x_i\}_{i=1}^N\), the associated NTK Gram matrix is
(51)
The central NTK conclusions can then be summarized as follows.
Theorem 2 (Standard NTK regime [14])
Under Assumptions 2–6, the following hold:
- Gaussian-process initialization. The random function \(g(\cdot;\theta_0)\) converges in law to a Gaussian process.
- Deterministic limiting NTK. The random NTK \(k(x,x';\theta_0)\) converges to a deterministic limiting kernel \(k_\infty(x,x')\).
-
Kernel stability during training. Along training, the NTK remains approximately constant:
\[ \begin{equation} k(x,x';\theta(t)) \approx k(x,x';\theta_0) \approx k_\infty(x,x'). \end{equation} \](52)
-
Output dynamics in function space. If \(\mathbf g(t)=[g(x_1;\theta(t)),\dots,g(x_N;\theta(t))]^\top\) denotes the vector of network outputs on training points and \(\mathcal L(\mathbf g)\) is the loss viewed as a function of the outputs, then under gradient flow one has
\[ \begin{equation} \dot{\mathbf g}(t) \approx -K\,\nabla_{\mathbf g}\mathcal L(\mathbf g(t)), \end{equation} \](53)
where \(K\) is the limiting NTK Gram matrix.
B.4.1 Interpretation.
The theorem states that, in the wide-network regime, training can be analyzed directly in function space. The nonlinear parameter dynamics are replaced by a kernel-preconditioned gradient flow on the outputs. This is precisely why the NTK framework is useful for studying residual dynamics, modewise convergence, and kernel-induced weighting mechanisms.
Hence the convergence rate is governed by the spectrum of the kernel matrix \(K\): components associated with large eigenvalues decay faster, while components associated with small eigenvalues decay more slowly. This spectral viewpoint is one of the key motivations for using NTK analysis in training-dynamics problems.
B.4.2 Scope of the approximation.
The NTK regime is an asymptotic and mechanism-level approximation. Its justification relies on wide hidden layers, standard smooth parameterization, and small parameter drift. Therefore, it is not a complete description of all finite-width training phenomena. Nevertheless, it provides a mathematically tractable and often accurate description of the early and intermediate training dynamics of overparameterized neural networks, which is why it serves as the theoretical basis for the kernel-based analysis in this work.
B.4.3 Relevance for the present paper.
In the sequel, this NTK viewpoint will be applied separately to the generator and the discriminator. On the generator side, it yields a kernel description of residual evolution. On the discriminator side, it motivates a function-space flow under alternating training. The resulting interaction between these two kernel-induced dynamics is the basis of our later residual-space analysis.
C Detailed derivations of discriminator and generator dynamics
In this appendix, we provide the detailed derivations of the discriminator and generator dynamics used in the main text. Throughout, we follow the notation introduced in Section 2.3. Recall that the adversarial objective is written in the unified form
(54)
for the discriminator, and
(55)
for the generator.
Under alternating optimization, gradient ascent is applied to \( \phi\) and gradient descent is applied to \( \theta\) . The corresponding gradients are
(56)
and
(57)
Hence the discrete parameter updates are
(58)
(59)
where \( a(n)\) and \( b(n)\) denote the generator and discriminator learning rates, respectively. Passing formally to the continuous-time limit yields the gradient-flow system
(60)
(61)
C.1 Discriminator dynamics in parameter space
We first derive the discriminator evolution. For a fixed generator state \( \theta\) , we have defined the residual samples
(62)
Then (60) becomes
(63)
This is the discriminator gradient flow in parameter space.
Let
(64)
denote the discriminator NTK. Under the standard Assumptions 2-6, this kernel remains approximately constant during training:
(65)
For any input \( x\) in the discriminator input space, the time derivative of the discriminator output is
(66)
Under the constant-NTK approximation, this reduces to
(67)
In particular, evaluating at \( x=r_j\) yields
(68)
Integrating (67) from \( 0\) to \( t\) , we obtain the representation
(69)
where \( f_0(x):=f(x;\phi(0))\) .
C.2 Functional-space notation for discriminator training
Before deriving the discriminator dynamics in function space, we briefly introduce the notation that will be used in the following subsection. The purpose of this step is to rewrite the discriminator update not only at the parameter level \(\phi\), but also at the level of the discriminator function \(f(\cdot;\phi)\) acting on the residual domain.
C.2.1 Residual-domain training measure.
For a fixed generator state \(\theta\), the discriminator is trained on two types of samples in the residual domain:
- the real target samples \(\mathcal T(x_i)=0\),
- the generated residual samples \(\mathcal R(x_i;\theta)=r_i\).
Accordingly, we introduce the empirical training measure on the residual domain,
(70)
Thus, \(\hat\gamma_\theta^{\,r}\) is the empirical measure supported on the real zero state and the generated residual samples. It plays the role of the discriminator-side training distribution in residual space.
C.2.2 Discriminator as a function on residual space.
Instead of only viewing the discriminator through its parameter vector \(\phi\), we may also regard it as a time-dependent function
(71)
defined on the residual variable \(r\in\mathbb R\). In this viewpoint, the discriminator training flow (60) induces an evolution equation for the function \(f_t\) itself.
C.2.3 Functional gradient.
Let \(\mathcal L_D(f)\) denote the discriminator objective viewed as a functional of the discriminator function \(f\). Its gradient with respect to the empirical measure \(\hat\gamma_\theta^{\,r}\), denoted by
is the function on the support of \(\hat\gamma_\theta^{\,r}\) such that, for every perturbation \(\psi\in L^2(\hat\gamma_\theta^{\,r})\),
(72)
In other words, \(\nabla_{\hat\gamma_\theta^{\,r}}\mathcal L_D(f)\) is the discriminator-side functional gradient evaluated on the empirical residual training measure.
C.2.4 Discriminator NTK and RKHS.
Assume now that the discriminator is in the infinite-width or constant-NTK regime, and let \(k^D(r,r')\) denote its Neural Tangent Kernel on the residual domain. Associated with \(k^D\) and the empirical measure \(\hat\gamma_\theta^{\,r}\), one defines the reproducing kernel Hilbert space (RKHS), denoted by
which is the function space generated by the kernel sections \(k^D(r,\cdot)\) over points \(r\) in the support of \(\hat\gamma_\theta^{\,r}\).
C.2.5 Kernel integral operator.
The corresponding kernel integral operator
is defined by
(73)
Since \(\hat\gamma_\theta^{\,r}\) is empirical, this becomes
(74)
Therefore, \(T_{k^D,\hat\gamma_\theta^{\,r}}\) maps a function defined only on the discrete training support into a kernel-smoothed function on the whole residual domain.
C.2.6 Interpretation.
The key point is that the discriminator loss is only evaluated on the empirical support \(\{0,r_1,\dots,r_{N_r}\}\), whereas the generator requires the input derivative \(\partial_r f_t(r)\) of the trained discriminator as a genuine function of the residual variable. The kernel operator \(T_{k^D,\hat\gamma_\theta^{\,r}}\) provides precisely this extension: it turns the empirical discriminator training signal into a smooth function on residual space, shaped by the architecture-dependent NTK \(k^D\).
With these notations in place, the next subsection rewrites discriminator training as a functional gradient flow in residual space.
C.3 Discriminator analysis under alternating training: an NTK interpretation in residual space
We adapt the discriminator-side NTK viewpoint of alternating GAN training to our residual-space formulation of adversarial PINNs. Concretely, the general NTK function-space flow of the discriminator is adapted from the discriminator-side NTK framework of Franceschi et al. [15], while our contribution here is to rewrite and interpret it in the present residual setting, where the generated distribution is the empirical residual measure \(\hat\mu_\theta^{\,r}\) and the real distribution is the zero state \(\delta_0\). This residual-space reinterpretation is what allows the discriminator flow to be connected directly to the generator-side residual dynamics studied in the main text. With the functional-space notation introduced above, the discriminator training dynamics under a fixed generator state can be written as
(75)
This equation shows that, under alternating optimization, the discriminator should not be viewed as an arbitrary optimizer on the empirical residual samples, but rather as a finite-time function-space flow shaped jointly by the empirical residual training measure \(\hat\gamma_\theta^{\,r}\) and the discriminator NTK \(k^D\). In particular, the discriminator architecture enters generator training through the kernel operator in (75), and therefore directly affects the gradient field induced on residual space.
A key consequence of (75) is that the trained discriminator is uniquely determined by its initial state and admits the representation
(76)
Hence the increment \(f_t-f_0\) lies in the RKHS generated by \(k^D\), which means that discriminator training amounts to adding to the initial network a kernel-smoothed correction determined by the empirical residual samples and the real zero state. This observation is important in our setting because the discriminator loss is only evaluated on the discrete support \(\{0,r_1,\dots,r_{N_r}\}\), whereas the generator dynamics require the input derivative \(\partial_r f_t(r)\) as a function defined on the whole residual domain.
Under standard architectural assumptions, the discriminator NTK is differentiable, and therefore so is the trained discriminator \(f_t\). As a result, the derivative field \(\partial_r f_t(r)\) appearing in the generator-side residual dynamics is well-defined. This resolves the gradient-indeterminacy issue that would arise if one treated the empirical discriminator as an unconstrained optimum only defined on a discrete support. In our framework, the discriminator derivative is instead a finite-time, NTK-smoothed quantity determined by both the sample distribution and the discriminator architecture.
This interpretation fits directly into the generator dynamics developed in the main text (24). Once the discriminator is frozen after its substep, the residual evolution of the generator takes the form
(77)
where the diagonal entries of \(\Gamma^D\) depend on (23):
Therefore, the discriminator-induced weighting in adversarial PINNs is not produced by an abstract optimal critic, but by the finite-time residual-space vector field generated by (75). From this viewpoint, the discriminator plays the role of an architecture-dependent smoothing and weighting operator acting on the empirical residual distribution.
C.3.1 The IPM case: kernel witness function on residual samples
We now reinterpret the general discriminator-side NTK framework of [15] in our residual-space setting. The key point here is that the discriminator behaves as a kernel witness function measuring the discrepancy between the empirical residual distribution and the zero target. This makes the resulting generator-side weighting especially explicit.
Proposition 3 (IPM discriminator flow as a kernel witness function on residual space)
Fix the generated residual distribution \(\hat\mu_\theta^{\,r}\) and the target \(\delta_0\). In the IPM case, let \(k^D\) denote the discriminator kernel. Then:
-
The discriminator evolves linearly in function space as
\[ \begin{equation} f_t = f_0+t\,f_\theta^\ast, \end{equation} \](78)
where \(f_\theta^\ast\) is the unnormalized MMD witness function associated with \(k^D\).
-
In residual-space notation, the witness function is given by
\[ \begin{equation} f_\theta^\ast(r) = -\mathbb E_{\tilde r\sim \hat\mu_\theta^{\,r}}\big[k^D(\tilde r,r)\big] + k^D(0,r), \end{equation} \](79)
or equivalently
\[ \begin{equation} f_\theta^\ast(r) = -\frac{1}{N_r}\sum_{i=1}^{N_r}k^D(r_i,r)+k^D(0,r). \end{equation} \](80)
-
The induced generator-side sample weighting is
\[ \begin{equation} \gamma_i^{\mathrm{IPM}} = \frac{1}{N_r}\,\partial_r f_t(r_i) = \frac{1}{N_r}\,\partial_r f_0(r_i) + \frac{t}{N_r}\,\partial_r f_\theta^\ast(r_i). \end{equation} \](81)
Proof
In the IPM setting, the discriminator-side NTK dynamics admit an explicit linear form in function space. For fixed \(\hat\mu_\theta^{\,r}\) and target \(\delta_0\), the discriminator moves in the direction of the kernel witness function associated with the discrepancy between the generated residual distribution and the zero target. Therefore,
(82)
which proves (78).
By definition, the corresponding unnormalized witness function is
(83)
which proves (79). Since
the empirical representation (80) follows immediately:
(84)
Finally, in the IPM-GAN case, the generator-side weighting is given by
(85)
Substituting \(f_t=f_0+t\,f_\theta^\ast\) yields
(86)
which proves (81). \( \square\)
Interpretation
Proposition 3 shows that, in the IPM case, the discriminator is most naturally interpreted as a kernel witness function on residual space. Unlike the LSGAN discriminator, which relaxes toward a regression target, or the vanilla GAN discriminator, which behaves as a nonlinear kernel classifier, the IPM discriminator directly represents the discrepancy between the empirical residual distribution and the zero target through the kernel \(k^D\).
This makes the generator-side effect especially transparent: the discriminator induces a residual-space gradient field through \(\partial_r f_t(r_i)\), and this field is precisely the derivative of the witness function. Therefore, the IPM discriminator contributes not merely a notion of sample realism, but an explicit sample-dependent preconditioning field for residual descent. In the language of this paper, this is the clearest example of how discriminator training reshapes generator dynamics through residual-space weighting.
C.3.2 The LSGAN case: kernel-smoothed regression target for our choice of labels
We next specialize the discriminator-side NTK framework to our LSGAN formulation in residual space. The main point here is not to introduce a new NTK tool, but to reinterpret the discriminator dynamics under our labeling convention: generated residual samples are assigned target value \(0\), while the real zero state is assigned target value \(1\). Under this choice, the discriminator becomes a kernel-smoothed regression field on residual space, whose derivative then induces the generator-side residual weighting.
Proposition 4 (LSGAN discriminator flow as kernel-smoothed regression on residual space)
Consider the discriminator objective
(87)
Let
(88)
and define the regression target
(89)
Then the following statements hold:
-
The \(L^2(\hat\gamma_\theta^{\,r})\) functional gradient of \(\mathcal L_D^{\mathrm{LS}}\) is
\[ \begin{equation} \nabla_{\hat\gamma_\theta^{\,r}} \mathcal L_D^{\mathrm{LS}}(f) = 2(\rho_\theta-f). \end{equation} \](90)
-
Consequently, the discriminator NTK flow is
\[ \begin{equation} \partial_t f_t = 2\,T_{k^D,\hat\gamma_\theta^{\,r}}(\rho_\theta-f_t). \end{equation} \](91)
-
The resulting function-space dynamics are linear, and the solution is
\[ \begin{equation} f_t = \exp\big(-2t\,T_{k^D,\hat\gamma_\theta^{\,r}}\big)(f_0-\rho_\theta) + \rho_\theta. \end{equation} \](92)
Equivalently, if
\[ \begin{equation} \varphi_t(x):=e^{-2tx}-1, \end{equation} \](93)
then
\[ \begin{equation} f_t = f_0 + \varphi_t\big(T_{k^D,\hat\gamma_\theta^{\,r}}\big)(f_0-\rho_\theta). \end{equation} \](94)
-
The induced generator-side sample weighting is
\[ \begin{equation} \gamma_i^{\mathrm{LSGAN}} = -\frac{1}{N_r} \big(f_t(r_i)-1\big)\, \partial_r f_t(r_i). \end{equation} \](95)
C.3.2.1 Proof.
Starting from (87), we rewrite the discriminator loss with respect to the empirical measure \(\hat\gamma_\theta^{\,r}\). Since
we may express \(\mathcal L_D^{\mathrm{LS}}\) as
(96)
Taking the functional derivative with respect to \(f\) gives
(97)
Since
we have
Therefore,
(98)
which proves (90).
Substituting this gradient into the general discriminator function-space flow yields
(99)
which proves (91).
Since this is a linear differential equation in function space, its solution is
(100)
which proves (92). The equivalent representation (94) follows immediately from the definition \( \varphi_t(x)=e^{-2tx}-1. \)
Finally, under our generator-side convention, the induced sample-wise weighting is
(101)
which proves (95). \( \square\)
C.3.2.2 Interpretation.
Proposition 4 shows that, under our labeling convention, the LSGAN discriminator is best understood not as an abstract optimal critic, but as a kernel-smoothed regression function on residual space. The target \(\rho_\theta\) encodes the desired regression rule induced by the real zero state and the generated residual samples. When the supports of \(\delta_0\) and \(\hat\mu_\theta^{\,r}\) are disjoint, this rule reduces to the usual binary labeling convention: the real zero state is assigned value \(1\), while generated residual samples are assigned value \(0\). The NTK flow then drives the discriminator toward this regression target in function space, up to the null space of the kernel operator.
This interpretation is useful for adversarial PINNs because the generator does not only need discriminator values on the empirical support, but also requires the derivative field \(\partial_r f_t(r)\). The kernel operator \(T_{k^D,\hat\gamma_\theta^{\,r}}\) extends the target regression signal into a smooth function on the whole residual domain, so that \(\partial_r f_t(r_i)\) is well-defined at finite training time. In this sense, the discriminator acts as a kernel-smoothed regression field whose derivative modulates the generator residual dynamics.
Compared with the IPM case, where the discriminator behaves as a witness function, the LSGAN discriminator instead behaves like a regression map relaxing toward a target field. Therefore, the essential difference is not only the nominal loss form, but the type of residual-space structure induced on the generator: witness-type geometry in the IPM case versus regression-type weighting in the LSGAN case.
Remark 8 (On the target density and convergence)
The target \(\rho_\theta=d\delta_0/d(\delta_0+\hat\mu_\theta^{\,r})\) should be interpreted as a Radon–Nikodym density with respect to the combined residual measure, rather than always as a pointwise binary label. In particular, if some generated residual samples satisfy \(r_i=0\), then the supports of \(\delta_0\) and \(\hat\mu_\theta^{\,r}\) overlap, and the value of \(\rho_\theta\) at zero reflects the relative mass of the real zero state and the generated residual samples located at zero. Thus, the simple \(1/0\) labeling interpretation is exact only in the support-disjoint case.
Moreover, the formula
should be understood as relaxation toward \(\rho_\theta\) modulo the null space of the kernel operator. If \(T_{k^D,\hat\gamma_\theta^{\,r}}\) is injective on the empirical support, then the discriminator converges exponentially to \(\rho_\theta\) on that support. Otherwise, components of \(f_0-\rho_\theta\) lying in the null space of \(T_{k^D,\hat\gamma_\theta^{\,r}}\) are not dissipated by the flow.
C.3.3 The vanilla GAN case: kernel-smoothed nonlinear classifier in residual space
We finally rewrite the vanilla GAN discriminator flow in the same residual-space language, so that it can be compared directly with the IPM and LSGAN cases from the viewpoint of generator-side residual weighting. In contrast to the linear regression structure of LSGAN, the key new feature here is the sigmoid nonlinearity, which turns the discriminator dynamics into a nonlinear kernel classification flow.
Proposition 5 (Vanilla GAN discriminator flow on residual space)
Let the discriminator output be
(102)
where \(f_t(r)\) is the pre-sigmoid score. Consider the discriminator objective
(103)
Let
(104)
and define
(105)
Then:
-
The \(L^2(\hat\gamma_\theta^{\,r})\) functional gradient of \(\mathcal L_D^{\mathrm{GAN}}\) is
\[ \begin{equation} \nabla_{\hat\gamma_\theta^{\,r}}\mathcal L_D^{\mathrm{GAN}}(f) = 2\big(\rho_\theta-\sigma(f)\big). \end{equation} \](106)
-
Consequently, the discriminator NTK flow is
\[ \begin{equation} \partial_t f_t = 2\,T_{k^D,\hat\gamma_\theta^{\,r}} \big(\rho_\theta-\sigma(f_t)\big). \end{equation} \](107)
-
If \(f_\infty\) is a stationary point of (107), then
\[ \begin{equation} T_{k^D,\hat\gamma_\theta^{\,r}} \big(\rho_\theta-\sigma(f_\infty)\big) = 0. \end{equation} \](108)
In particular, if the kernel operator is injective on the empirical support, then
\[ \begin{equation} \sigma(f_\infty)=\rho_\theta ~~ \text{on } \mathrm{supp}(\hat\gamma_\theta^{\,r}). \end{equation} \](109)
-
Near a stationary point \(f_\infty\), writing \(f_t=f_\infty+h_t\) with \(\|h_t\|\ll 1\), one has the first-order approximation
\[ \begin{equation} \partial_t h_t = -2\,T_{k^D,\hat\gamma_\theta^{\,r}} \big(\sigma'(f_\infty)h_t\big) + \mathcal O(\|h_t\|^2). \end{equation} \](110)
-
The induced generator-side sample weighting is
\[ \begin{equation} \gamma_i^{\mathrm{GAN}} = \frac{1}{N_r}\, \frac{1}{1-D_t(r_i)}\,\partial_r D_t(r_i) = \frac{1}{N_r}\, D_t(r_i)\,\partial_r f_t(r_i). \end{equation} \](111)
C.3.3.1 Proof.
Rewriting (103) over the empirical measure \(\hat\gamma_\theta^{\,r}\), we have
(112)
Differentiating with respect to \(f\) yields
(113)
Since
we obtain
(114)
which proves (106). Substituting this into the general discriminator NTK flow gives (107).
The equilibrium condition follows immediately by setting \(\partial_t f_t=0\) in (107), which gives
If the kernel operator is injective on the empirical support, then (109) follows.
For the local linearization, write \(f_t=f_\infty+h_t\) and use the Taylor expansion
Substituting this into (107) and using the equilibrium relation yields (110).
Finally, for the minimax generator objective,
(115)
Equivalently, writing the discriminator output directly as \(D_t(r)=\sigma(f_t(r))\), one obtains
(116)
Using the chain rule
this becomes
(117)
which proves (111). \( \square\)
C.3.3.2 Interpretation.
Proposition 5 shows that, in residual space, the vanilla GAN discriminator is best understood as a kernel-smoothed nonlinear classifier rather than as a linear regression field or a witness function. The target \(\rho_\theta\) still separates the real zero residual from the generated residual samples, but the sigmoid nonlinearity changes the discriminator dynamics qualitatively. Formally, if a finite stationary point exists, it satisfies the equilibrium condition in Proposition 5. In the separable case, however, the limiting relation \(\sigma(f_\infty)=\rho_\theta\) may correspond to saturation of the logits rather than convergence to a finite discriminator.
This saturation behavior directly affects the generator-side weighting. Although the logit-level discriminator flow is driven by the classification residual \(\rho_\theta-\sigma(f_t)\), the probability-level input derivative satisfies
Hence the residual-space derivative of the discriminator output is suppressed in saturated regions where \(D_t(r)\approx 0\) or \(D_t(r)\approx 1\). Compared with IPM and LSGAN, vanilla GAN also introduces an additional confidence factor \(D_t(r_i)\) into the sample-wise weighting, so the discriminator acts through both its residual-space score gradient \(\partial_r f_t(r_i)\) and its current classification confidence. In this sense, the essential difference is not only the nominal adversarial loss, but also the way the sigmoid nonlinearity reshapes the discriminator-induced residual weighting.
Remark 9
The purpose of the present appendix is not to claim a new NTK theory of GAN discriminators in itself. Rather, we use the existing discriminator-side NTK framework as an analysis tool and reinterpret it in residual space so that it becomes directly applicable to adversarial PINNs. What is specific to the present setting is that the “generated samples” are residual values, the “real samples” correspond to the zero residual state, and the resulting discriminator flow can therefore be linked explicitly to the generator-side residual weighting and residual-energy dynamics developed in the main text.
C.4 Generator residual dynamics
We next derive the generator-side dynamics in residual space. Unlike the discriminator, whose evolution is naturally described in function space, the generator affects training through the evolution of the residual values themselves. It is therefore convenient to rewrite the parameter-space gradient flow of the generator as a closed dynamics for the residual vector. This leads directly to the kernel-preconditioned residual evolution used throughout the main text.
Proposition 6 (Generator residual dynamics in residual space)
Let the generator gradient flow be
(118)
For each collocation point \(x_j\), define the residual
(119)
Let
(120)
be the residual vector, and define the generator residual NTK matrix
(121)
Further define the diagonal matrix
(122)
with entries
(123)
Then the generator-induced residual dynamics satisfy
(124)
or equivalently, in vector form,
(125)
where \(\mathbf 1\in\mathbb R^{N_r}\) denotes the all-one vector.
Equivalently, if one introduces the weighting vector
(126)
with
(127)
then the residual dynamics admit the compact form
(128)
\paragraph{Proof.} Starting from the generator gradient flow,
(129)
and substituting (56), we obtain
(130)
For each collocation point \(x_j\), apply the chain rule:
(131)
Since \(r_i(t)=\mathcal R(x_i;\theta(t))\), this becomes
(132)
Defining the generator residual NTK matrix \(K_{rr}^G(t)\) and the diagonal weighting matrix \(\Gamma(t)\) by (123), we obtain the componentwise representation
(133)
which yields the vector form
(134)
Finally, introducing \(\boldsymbol{\gamma}(t)=\Gamma(t)\mathbf 1\) gives
(135)
This proves (125) and (128). \( \square\)
\paragraph{Interpretation.} Proposition 6 shows that the generator does not evolve in residual space by an ordinary gradient descent with a scalar loss coefficient. Instead, its residual dynamics are preconditioned by the generator residual NTK \(K_{rr}^G(t)\) and modulated samplewise by the discriminator-induced weighting \(\boldsymbol{\gamma}(t)\).
This decomposition is central to the viewpoint of the main text. The matrix \(K_{rr}^G(t)\) encodes the geometry of residual propagation through the generator, while \(\boldsymbol{\gamma}(t)\) contains the discriminator-side signal through the factor
Hence, different adversarial objectives do not simply change a scalar divergence value; rather, they reshape the residual dynamics through different sample-dependent weighting mechanisms. In this sense, adversarial PINNs training should be understood as a discriminator-modulated, kernel-preconditioned residual descent process.
C.4.1 Specialization to common adversarial objectives
We finally record the specialization of \( \Gamma(t)\) for several standard adversarial objectives.
C.4.1.1 Vanilla GAN.
If
(136)
then
(137)
Hence, by (127),
(138)
C.4.1.2 Least-Squares GAN (LSGAN).
If
(139)
then
(140)
Hence
(141)
C.4.1.3 Integral Probability Metric GAN (IPM-GAN).
If
(142)
then
(143)
Therefore,
(144)
Equations above show that, once the discriminator state is fixed, different adversarial objectives enter the generator residual dynamics through different sample-wise weighting rules, while the structural form of the residual dynamics remains the same.
C.4.2 Interpretation in our framework
The discriminator-side NTK analysis can be summarized in the language of this paper as follows. First, for a fixed generator, discriminator training against \(\hat\mu_\theta^{\,r}\) and \(\delta_0\) produces a finite-time, architecture-dependent residual-space function, rather than an arbitrary empirical optimum. Second, the derivative \(\partial_r f_t(r)\) entering the generator dynamics is therefore well-defined and inherits the smoothing and inductive bias of the discriminator NTK. Third, in the IPM case, this derivative is induced by the NTK-MMD witness function comparing \(\hat\mu_\theta^{\,r}\) and \(\delta_0\); in the LSGAN case, it is induced by a kernel-smoothed regression solution; and in the vanilla GAN case, it is induced by a kernel-smoothed nonlinear classification flow shaped by the sigmoid nonlinearity.
From our perspective, the essential role of the discriminator in adversarial PINNs is therefore not to realize an abstract optimal divergence critic, but to generate, after finite-time NTK-constrained training, a differentiable residual-space weighting field whose derivative directly modulates the generator residual dynamics. This is precisely the mechanism captured by our framework.
D Analysis of the residual-energy decay
D.1 Residual-energy law and constant-kernel approximation
We now turn from the residual dynamics themselves to the evolution of the residual energy. This provides a more compact scalar description of whether adversarial training is effectively driving the residual toward zero. Starting from the generator-side residual dynamics derived in the previous appendix, we show that the energy dissipation rate is determined by the interaction between the generator residual NTK and the discriminator-induced weighting vector. This identity will serve as the basis for the later decay estimates, failure-mode analysis, and spectral interpretations.
Theorem 3 (Residual-energy law under adversarial residual dynamics)
Let
(145)
be the residual energy, where the residual vector evolves according to
(146)
with generator residual NTK \(K_{rr}^G(t)\in\mathbb R^{N_r\times N_r}\) and discriminator-induced weighting vector
(147)
Then the residual energy satisfies
(148)
In particular, the sign and magnitude of \(\dot E(t)\) are determined by how the discriminator-induced weighting \(\boldsymbol{\gamma}(t)\) aligns with the residual vector through the kernel geometry imposed by \(K_{rr}^G(t)\).
D.1.1 Proof.
By definition,
(149)
Differentiating with respect to time yields
(150)
Substituting the residual dynamics (146), we obtain
(151)
which proves (148). \( \square\)
D.1.2 Interpretation.
Theorem 3 shows that the energy decay in adversarially trained PINNs is not controlled by the generator kernel alone, nor by the discriminator weighting alone, but by their interaction. The matrix \(K_{rr}^G(t)\) determines how a local discriminator signal propagates across residual samples, while \(\boldsymbol{\gamma}(t)\) determines whether that propagated signal is aligned with residual descent. Therefore, the key quantity is not simply the size of the discriminator gradient, but the signed bilinear form
which acts as the instantaneous energy-dissipation rate.
To obtain explicit convergence rates and more refined decay mechanisms, it is convenient to further work under the constant-NTK approximation
(152)
Under this approximation, the energy law admits a clearer spectral interpretation, which allows us to distinguish exponential decay, slower algebraic decay, and several representative failure modes.
Remark 10 (On the constant-NTK approximation)
The approximation (152) is best understood as a mechanism-level simplification rather than a universally exact property. Under Assumptions 2–6, for some relatively simple or weakly nonlinear problems—especially linear PDEs and certain regular regimes—the generator NTK is known or empirically observed to remain nearly constant during training[5]. By contrast, for strongly nonlinear, multiscale, or more complicated equations, the NTK may evolve nontrivially [21, 22], so the constant-kernel assumption need not hold exactly. Nevertheless, even in such cases, the approximation remains useful for revealing the dominant spectral mechanisms behind residual-energy decay and failure modes.
D.2 Spectral factorization of the aligned dynamics
We first consider the favorable case in which the discriminator-induced weighting is linearly aligned with the negative residual direction. Instead of treating \(K_{rr}^G A\) as a symmetric operator, we use a spectral factorization that converts the effective operator into the symmetric positive semidefinite matrix \(K_{rr}^{G,1/2} A K_{rr}^{G,1/2}\).
Assume throughout this subsection that the generator residual NTK \(K_{rr}^G\) is symmetric positive definite on the active residual subspace, and that the discriminator-induced weighting takes the linear alignment form
(153)
where \(A\in\mathbb R^{N_r\times N_r}\) is symmetric positive semidefinite. Then the residual dynamics become
(154)
Theorem 4 (Spectral factorization and convergence rate under linear alignment)
Let
(155)
Assume that \(K\) is symmetric positive definite and that \(A\) is symmetric positive semidefinite. Then:
-
\(M\) is similar to \(H\), namely
\[ \begin{equation} K^{-1/2}MK^{1/2} = K^{-1/2}(KA)K^{1/2} = K^{1/2}AK^{1/2} = H. \end{equation} \](156)
Hence \(M\) and \(H\) have the same eigenvalues.
-
Since \(H\) is symmetric positive semidefinite, all eigenvalues of \(M\) are real and nonnegative. Moreover, if
\[ \begin{equation} \lambda_1(A)\ge \lambda_2(A)\ge \cdots \ge \lambda_{N_r}(A)\ge 0 \end{equation} \](157)
denote the eigenvalues of \(A\), then for each \(j=1,\dots,N_r\),
\[ \begin{equation} \lambda_{\min}(K)\,\lambda_j(A) \;\le\; \lambda_j(M) = \lambda_j(H) \;\le\; \lambda_{\max}(K)\,\lambda_j(A). \end{equation} \](158)
-
Introducing the transformed variable
\[ \begin{equation} \mathbf z(t):=K^{-1/2}\mathbf r(t), \end{equation} \](159)
the dynamics (154) become
\[ \begin{equation} \dot{\mathbf z}(t)=-H\mathbf z(t). \end{equation} \](160)
Therefore,
\[ \begin{equation} \|\mathbf z(t)\|^2 \le e^{-2\lambda_{\min}(H)t}\|\mathbf z(0)\|^2. \end{equation} \](161)
Since
\[ \begin{equation} \lambda_{\min}(K)\|\mathbf z\|^2 \le \|\mathbf r\|^2 \le \lambda_{\max}(K)\|\mathbf z\|^2, \end{equation} \](162)
the residual energy satisfies
\[ \begin{equation} E(t) \le \kappa(K)\,E(0)\,e^{-2\lambda_{\min}(H)t}, ~~ \kappa(K):=\frac{\lambda_{\max}(K)}{\lambda_{\min}(K)}. \end{equation} \](163)
In particular, using (158),
\[ \begin{equation} E(t) \le \kappa(K)\,E(0)\, e^{-2\lambda_{\min}(K)\lambda_{\min}(A)t}. \end{equation} \](164)
D.2.1 Proof.
The similarity relation follows directly from
Since \(H=K^{1/2}AK^{1/2}\) is symmetric positive semidefinite, its spectrum is real and nonnegative, and therefore the same is true for \(M\). The eigenvalue bound (158) is the standard congruence bound for symmetric positive semidefinite matrices. Now define \(\mathbf z=K^{-1/2}\mathbf r\). Then
which gives (160). Since \(H\succeq 0\),
Applying Gr
specialChar{34}onwall’s inequality yields
Finally, using the norm equivalence induced by \(K\),
we obtain (163). The lower bound
then gives (164). \( \square\)
D.2.2 Interpretation.
Theorem 4 shows that, under linear alignment, the effective decay rate is controlled not by the generator residual NTK \(K_{rr}^G\) alone, but by the coupled operator \(K_{rr}^G A\), or equivalently by the symmetric matrix \(K_{rr}^{G,1/2} A K_{rr}^{G,1/2}\). Therefore, adversarially induced weighting may accelerate or slow down convergence through its interaction with the spectrum of the generator kernel. In particular, small eigenvalues of \(A\) may lead to slow-decaying directions even when \(K_{rr}^G\) itself is well-conditioned, while large eigenvalues of \(A\) may selectively amplify useful descent directions.
Remark 11 (Connection to standard PINN-NTK dynamics)
The standard non-adversarial PINN-NTK law is recovered as the special case
that is, \(A=I\). In this case the aligned dynamics reduce to
and the residual energy satisfies
Thus ordinary PINN training corresponds to the reference case in which the residual is fed back without adversarial distortion.
D.3 Modal view and slower decay under weaker alignment
We now turn to the complementary situation where the residual energy still decreases, but the convergence is no longer fast. In this regime, the discriminator-induced weighting remains descent-oriented, yet its alignment with the residual is weaker than in the strongly aligned case considered above. The resulting decay is therefore slower, and its mechanism is best understood in the eigenbasis of the generator residual NTK.
Theorem 5 (Modal decay and slower convergence under weaker alignment)
Assume the constant-NTK approximation
(165)
and let
(166)
be the eigendecomposition of the generator residual NTK. Define
(167)
Then:
-
The residual dynamics in modal coordinates are
\[ \begin{equation} \dot{\widetilde{\mathbf r}}(t)=\Lambda\,\widetilde{\boldsymbol{\gamma}}(t), \end{equation} \](168)
that is,
\[ \begin{equation} \dot{\widetilde r}_j(t)=\lambda_j\,\widetilde\gamma_j(t), ~~ j=1,\dots,N_r. \end{equation} \](169)
-
If the discriminator-induced weighting is modewise aligned in the sense that
\[ \begin{equation} \widetilde\gamma_j(t)=-a_j(t)\widetilde r_j(t), ~~ a_j(t)\ge a_*>0, \end{equation} \](170)
then each mode satisfies
\[ \begin{equation} \dot{\widetilde r}_j(t)=-\lambda_j a_j(t)\widetilde r_j(t), \end{equation} \](171)
and therefore
\[ \begin{equation} |\widetilde r_j(t)| \le |\widetilde r_j(0)|e^{-\lambda_j a_* t}. \end{equation} \](172)
Consequently, the residual energy satisfies
\[ \begin{equation} E(t) \le \frac12\sum_{j=1}^{N_r}\widetilde r_j(0)^2e^{-2\lambda_j a_* t}. \end{equation} \](173)
-
More generally, if the feedback remains descent-oriented but no longer scales linearly with the residual, and one only has
\[ \begin{equation} \mathbf r(t)^\top K_{rr}^G\boldsymbol{\gamma}(t) \le -c\|\mathbf r(t)\|^{2\alpha}, ~~ \alpha>1, \end{equation} \](174)
then the residual energy obeys the algebraic decay estimate
\[ \begin{equation} E(t)\le C(1+t)^{-1/(\alpha-1)} \end{equation} \](175)
for some constant \(C>0\).
D.3.1 Proof.
Since
(176)
left-multiplying by \(U^\top\) and using \(K_{rr}^G=U\Lambda U^\top\) gives
(177)
which proves (168).
Now assume the modewise alignment condition (170). Then each mode satisfies
(178)
Since \(a_j(t)\ge a_*>0\), Gr
specialChar{34}onwall’s inequality yields
(179)
which proves (172). Summing over all modes gives
(180)
which proves (173).
For the weaker descent-oriented case, starting from (174) and using \(\|\mathbf r(t)\|^2=2E(t)\), we obtain
(181)
Integrating the associated scalar differential inequality yields
(182)
for some constant \(C>0\), which proves (175). \( \square\)
D.3.2 Interpretation.
Theorem 5 makes clear that residual decay is inherently spectral. Under strong modewise alignment, each mode decays at a rate controlled by \(\lambda_j a_j(t)\), so modes associated with larger effective eigenvalues decay first. As a result, long-time training is dominated by the slower unresolved modes. This explains why spectral imbalance can be harmful: once the fast modes have already become small, maintaining a negative overall energy rate requires the discriminator-induced weighting to remain aligned with the slow modes that still dominate the residual energy.
The last part of the proposition shows that if the discriminator feedback remains descent-oriented but is no longer linearly aligned with the residual, then exponential decay is typically lost, and only algebraic decay can be guaranteed. This provides a natural explanation for the long optimization tails often observed in adversarial PINNs training: even when training does not fail, insufficiently strong alignment with the unresolved modes can substantially slow convergence.
D.4 Failure modes: ascent and stagnation
The previous three regimes describe successful but quantitatively different decay. The opposite cases are equally important.
Proposition 7 (Ascent, stagnation, and unstable oscillation regimes)
Under the residual-energy law (148):
-
If
\[ \begin{equation} \mathbf r(t)^\top K_{rr}^G(t)\boldsymbol{\gamma}(t)\ge 0 \end{equation} \](183)
on some time interval, then \(E(t)\) fails to decrease monotonically on that interval. In the strictly positive case, the residual energy increases.
-
If
\[ \begin{equation} \|\boldsymbol{\gamma}(t)\|\to 0 \end{equation} \](184)
while \(\|K_{rr}^G(t)\|\) remains bounded, then
\[ \begin{equation} \|\dot{\mathbf r}(t)\| \le \|K_{rr}^G(t)\|\,\|\boldsymbol{\gamma}(t)\| \to 0. \end{equation} \](185)
Thus the first-order residual dynamics stall. If this happens before \(\mathbf r(t)\) becomes small, training enters a plateau regime.
-
Let
\[ \begin{equation} s(t):=\mathbf r(t)^\top K_{rr}^G(t)\boldsymbol{\gamma}(t). \end{equation} \](186)
If \(s(t)\) changes sign repeatedly on a time interval, namely if there exist sequences
\[ \begin{equation} t_1<t_2<t_3<\cdots \end{equation} \](187)
such that
\[ \begin{equation} s(t_{2m-1})>0, ~~ s(t_{2m})<0, \end{equation} \](188)
for all admissible \(m\), then the residual-energy slope alternates between ascent-oriented and descent-oriented phases. In this case, \(E(t)\) does not exhibit a stable monotone decay trend, but instead undergoes oscillatory evolution. Consequently, training becomes dynamically unstable in the sense that the discriminator-induced update direction repeatedly switches between alignment and misalignment with residual descent.
Remark 12
Proposition 7 identifies three representative failure modes for adversarially trained PINNs. First, if the discriminator-induced weighting is misaligned with the residual after modulation by the generator residual NTK, so that \( \mathbf r(t)^\top K_{rr}^G(t)\boldsymbol{\gamma}(t)>0, \) then the residual energy increases rather than decreases. Second, if the discriminator-induced gradient vanishes, namely \( \|\boldsymbol{\gamma}(t)\|\to 0, \) then the first-order residual dynamics stall; this is especially likely to occur when the discriminator becomes overly strong and produces near-vanishing generator-side gradients, as can happen in GAN-type or LSGAN-type training. Third, even when the discriminator gradient does not vanish, it may fail to remain consistently descent-oriented, so that the induced weighting alternates between favorable and unfavorable directions. In this case, the first-order indicator changes sign repeatedly, leading to oscillatory and unstable training behavior. These three phenomena are also clearly illustrated in the right panel of Fig. 2 in the main text.
D.5 Discrete-time interpretation for the alternating algorithm
The same logic carries over to the practical alternating algorithm. Recall the first-order discrete expansion
(189)
Thus \(S^m\) is the discrete analogue of the continuous energy-dissipation rate.
D.5.1 Contractive regime.
If
(190)
uniformly and \(\eta_G\) is sufficiently small, then
(191)
which gives exponential decay in discrete time.
D.5.2 Slow-decay regime.
More generally, if \(S^m\) scales like a higher power of \(E^m\), then one expects a slower algebraic-type decay, consistent with the continuous-time estimate (175).
D.5.3 Failure regime.
If \(S^m\) becomes positive or nearly vanishes, then the discrete dynamics exhibit ascent or stagnation, respectively.
D.5.4 Implicit spectral balancing effect of the acceptance rule.
Although the proposed alternating algorithm does not explicitly enforce mode-by-mode spectral balancing, the score-based acceptance rule can still provide an indirect benefit from the spectral viewpoint. Indeed, in the eigenbasis of the generator residual NTK,
(192)
Thus, accepting a discriminator update only when \(S^m<0\) tends to favor discriminator states whose induced weighting yields a negative net contribution across the unresolved dominant modes. In particular, when fast modes have already decayed and their amplitudes \(\widetilde r_j^{\,m}\) become small, maintaining \(S^m<0\) increasingly requires useful correction on the slow modes that still dominate the residual energy. Consequently, the acceptance rule may suppress discriminator states that overemphasize already-resolved modes while underweighting unresolved ones.
From this perspective, the proposed method does not explicitly balance the spectrum, but it may exert an implicit spectral regularization effect on the alternating dynamics by filtering out discriminator states that produce poor net modal descent.
Overall, the continuous energy analysis provides a direct interpretation of the practical score used in the alternating algorithm. The convergence rate of the residual energy in adversarial PINNs is governed not by the generator residual NTK alone, but by the interaction between the generator-side kernel geometry, the discriminator-induced weighting, and the evolving residual itself.
E Different Dynamics Induced by Residual Input and Squared-Residual Input
In most existing adversarial PINNs formulations, the discriminator takes the residual itself as input,
By contrast, standard PINNs training is fundamentally driven by the squared residual, since the canonical physics loss is proportional to
This raises a natural question: what changes if the discriminator input is replaced by \(r_i^2\)? As we show below, this is not a minor reparameterization. It changes the generator dynamics from an externally forced preconditioned residual descent
into a state-dependent multiplicative feedback law
Consequently, both the residual evolution law and the energy dissipation mechanism become fundamentally different.
E.1 Generator dynamics with squared-residual input
We now consider the alternative design in which the discriminator no longer takes the residual \(r_i\) itself as input, but instead takes the squared residual \(r_i^2\). This seemingly simple modification changes the structure of the generator dynamics in an essential way. In the residual-input case, the discriminator induces an additive forcing term on the residual dynamics. By contrast, under squared-residual input, the chain rule introduces an additional factor of \(r_i\), so that the discriminator enters multiplicatively through the current residual state. The following proposition makes this distinction precise.
Proposition 8 (Generator dynamics with squared-residual discriminator input)
Let
(193)
and define
(194)
Consider the generator loss
(195)
Then:
-
The generator gradient is
\[ \begin{equation} \nabla_\theta \mathcal L_G^{(X^2)} = \frac{2}{N_r}\sum_{i=1}^{N_r} r_i\, R'\big(f_i^{(X^2)}\big)\, \partial_s f_i^{(X^2)}\, \nabla_\theta r_i, \end{equation} \](196)
where
\[ f_i^{(X^2)}:=f(s_i;\phi), ~~ \partial_s f_i^{(X^2)}:=\partial_s f(s_i;\phi). \] -
The generator gradient flow becomes
\[ \begin{equation} \dot{\theta} = -\frac{2}{N_r}\sum_{i=1}^{N_r} r_i\, R'\big(f_i^{(X^2)}\big)\, \partial_s f_i^{(X^2)}\, \nabla_\theta r_i. \end{equation} \](197)
-
Let \(J_r\) denote the residual Jacobian and \(K_{rr}^G(\theta)=J_rJ_r^\top\) the generator residual NTK. Then the residual dynamics satisfy
\[ \begin{equation} \dot{\mathbf r} = -\,K_{rr}^G(\theta)\,\boldsymbol{\gamma}^{(X^2)}, \end{equation} \](198)
where
\[ \begin{equation} \gamma_i^{(X^2)} = \frac{2}{N_r}\, r_i\, R'\big(f(r_i^2;\phi)\big)\, \partial_s f(r_i^2;\phi). \end{equation} \](199)
-
Defining
\[ \begin{equation} \widetilde{\gamma}_i := -\frac{2}{N_r} R'\big(f(r_i^2;\phi)\big)\, \partial_s f(r_i^2;\phi), \end{equation} \](200)
and
\[ \begin{equation} \widetilde{\Gamma} = \mathrm{diag}(\widetilde{\gamma}_1,\dots,\widetilde{\gamma}_{N_r}), \end{equation} \](201)
one has
\[ \begin{equation} \boldsymbol{\gamma}^{(X^2)}=-\widetilde{\Gamma}\mathbf r, \end{equation} \](202)
and therefore
\[ \begin{equation} \dot{\mathbf r} = K_{rr}^G(\theta)\,\widetilde{\Gamma}\,\mathbf r. \end{equation} \](203)
-
The corresponding residual-energy law is
\[ \begin{equation} \dot E = \mathbf r^\top K_{rr}^G(\theta)\,\widetilde{\Gamma}\,\mathbf r. \end{equation} \](204)
E.1.1 Proof.
Since \(s_i=r_i^2\), we have
Therefore,
(205)
which proves the first statement. The gradient-flow equation (197) follows immediately.
Next, multiplying by the residual Jacobian \(J_r\), we obtain
(206)
where
(207)
Defining
we have
and hence
(208)
which proves (203). Finally,
(209)
which proves (204). \( \square\)
E.1.2 Interpretation.
Proposition 8 shows that changing the discriminator input from \(r\) to \(r^2\) does not merely rescale the adversarial signal: it changes the structure of the residual dynamics. In the residual-input case, the discriminator acts through an additive forcing term of the form
where the weighting is independent of the current residual amplitudes. By contrast, under squared-residual input, the dynamics take the multiplicative form
so that the discriminator enters through the current residual state itself.
This difference is fundamental. The squared-residual input removes sign information and makes the discriminator sensitive mainly to residual magnitude, while the additional factor \(\mathbf r\) turns the dynamics into a genuine multiplicative feedback system. As a result, both the residual evolution law and the energy law become quadratic in the residual state. This is why squared-residual adversarial training is structurally different from the standard residual-input formulation, and why its later spectral analysis takes a substantially different form.
E.2 Spectral interpretation
Under the constant-kernel approximation
(210)
we analyze the residual-input and squared-residual-input dynamics in the eigenbasis of the generator residual NTK. The main purpose is to clarify how the two input choices differ at the level of modal evolution, and why the squared-residual-input case leads to a genuinely coupled multiplicative system.
Let
(211)
where \(U=[u_1,\dots,u_{N_r}]\) is orthonormal, and expand
(212)
Proposition 9 (Modal dynamics under residual and squared-residual inputs)
Under the above notation:
-
If the discriminator input is the residual itself, so that
\[ \begin{equation} \dot{\mathbf r}=K\Gamma\mathbf 1, \end{equation} \](213)
then the modal coefficients satisfy
\[ \begin{equation} \dot c_j = u_j^\top \dot{\mathbf r} = \lambda_j\,u_j^\top \Gamma\mathbf 1. \end{equation} \](214)
-
If the discriminator input is the squared residual, so that
\[ \begin{equation} \dot{\mathbf r}=K\widetilde{\Gamma}\mathbf r, \end{equation} \](215)
then the modal coefficients satisfy
\[ \begin{equation} \dot c_j = u_j^\top \dot{\mathbf r} = \lambda_j\sum_{k=1}^{N_r} c_k\,u_j^\top \widetilde{\Gamma}u_k. \end{equation} \](216)
E.2.1 Proof.
For the residual-input dynamics,
we compute
(217)
which proves (214).
For the squared-residual-input dynamics,
we similarly obtain
(218)
which proves (216). \( \square\)
E.2.2 Interpretation.
Proposition 9 reveals the structural difference between the two input choices. In the residual-input case, each mode is driven by an external discriminator-induced forcing term \(u_j^\top\Gamma\mathbf 1\). In the squared-residual-input case, by contrast, the evolution of mode \(j\) depends on all modal amplitudes \(\{c_k\}\), so the residual spectrum is fed back into its own dynamics. This is the first indication that squared-residual input leads to a genuinely multiplicative feedback system.
To understand this multiplicative structure more systematically, it is useful to introduce the coupled operator \(K\widetilde{\Gamma}\) and its symmetric counterpart. The same spectral factorization used for the linearly aligned residual-input dynamics applies here, with \(A\) replaced by \(\widetilde{\Gamma}\). The crucial difference is the sign convention: in the residual-input aligned case, decay corresponds to \(A\succeq0\) in \(\dot{\mathbf r}=-KA\mathbf r\), whereas in the squared-residual-input case, decay corresponds to \(\widetilde{\Gamma}\preceq0\) in \(\dot{\mathbf r}=K\widetilde{\Gamma}\mathbf r\).
Proposition 10 (Spectral factorization of the squared-residual-input dynamics)
Define
(219)
Assume that \(K\) is symmetric positive definite on the active residual subspace, and that \(\widetilde{\Gamma}\in\mathbb R^{N_r\times N_r}\) is symmetric. Then:
- \(M\) and \(H\) are similar. \(M\) and \(H\) have the same eigenvalues.
- Since \(H\) is symmetric, all eigenvalues of \(M\) are real.
- Since \(H\) is congruent to \(\widetilde{\Gamma}\), Sylvester’s law of inertia implies that \(H\) and \(\widetilde{\Gamma}\) have the same inertia, and therefore so does \(M\).
- If \(\widetilde{\Gamma}\preceq 0\), then all eigenvalues of \(M\) are nonpositive; if \(\widetilde{\Gamma}\succeq 0\), then all eigenvalues of \(M\) are nonnegative; if \(\widetilde{\Gamma}\) is indefinite, then \(M\) has both positive and negative eigenvalues.
E.2.3 Proof.
The similarity relation follows directly from
(220)
Hence \(M\) and \(H\) have the same eigenvalues. Since \(H\) is symmetric whenever \(\widetilde{\Gamma}\) is symmetric, its spectrum is real, and therefore so is the spectrum of \(M\). Moreover, \(H\) is congruent to \(\widetilde{\Gamma}\), so Sylvester’s law of inertia implies that they have the same inertia; similarity then transfers the same inertia to \(M\). The sign-definite and indefinite cases follow immediately. \( \square\)
E.2.4 Interpretation.
Proposition 10 shows that the squared-residual-input dynamics are governed not by \(K\) or \(\widetilde{\Gamma}\) separately, but by their coupled spectrum. The symmetric matrix \(H=K^{1/2}\widetilde{\Gamma}K^{1/2}\) provides the most transparent representation of this coupling, and its sign structure determines whether the active modes are contractive, expansive, or mixed.
This also gives a direct way to compare the spectrum of the coupled operator with those of \(K\) and \(\widetilde{\Gamma}\).
Proposition 11 (Spectral bounds and energy behavior)
Under the assumptions of Proposition 10:
-
If \(\widetilde{\Gamma}\succeq 0\), then the extreme eigenvalues of \(M\) satisfy
\[ \begin{equation} \lambda_{\min}(M) = \lambda_{\min}(H) \ge \lambda_{\min}(K)\lambda_{\min}(\widetilde{\Gamma}), ~~ \lambda_{\max}(M) = \lambda_{\max}(H) \le \lambda_{\max}(K)\lambda_{\max}(\widetilde{\Gamma}). \end{equation} \](221)
-
Introduce
\[ \begin{equation} \mathbf z:=K^{-1/2}\mathbf r. \end{equation} \](222)
Then the squared-residual-input dynamics become
\[ \begin{equation} \dot{\mathbf z}=H\mathbf z, \end{equation} \](223)
and the residual energy satisfies
\[ \begin{equation} \lambda_{\min}(K)\|\mathbf z\|^2 \le \|\mathbf r\|^2 \le \lambda_{\max}(K)\|\mathbf z\|^2. \end{equation} \](224)
-
If \(\widetilde{\Gamma}\preceq 0\), then \(H\preceq 0\). In the strictly negative definite case \(\widetilde{\Gamma}\prec 0\),
\[ \begin{equation} E(t) \le \kappa(K)\,E(0)\,e^{2\lambda_{\max}(H)t} \le \kappa(K)\,E(0)\,e^{-2\lambda_{\min}(K)|\lambda_{\max}(\widetilde{\Gamma})|t}, \end{equation} \](225)
where \(\lambda_{\max}(H)<0\) and
\[ \begin{equation} \kappa(K):=\frac{\lambda_{\max}(K)}{\lambda_{\min}(K)}. \end{equation} \](226)
-
If \(\widetilde{\Gamma}\succeq 0\), then the transformed energy \(\frac12\|\mathbf z(t)\|^2\) is nondecreasing. In the strictly positive definite case \(\widetilde{\Gamma}\succ 0\), the transformed dynamics are expansive, and the original residual energy admits the lower bound
\[ \begin{equation} E(t) \ge \kappa(K)^{-1}E(0)\,e^{2\lambda_{\min}(H)t} \ge \kappa(K)^{-1}E(0)\,e^{2\lambda_{\min}(K)\lambda_{\min}(\widetilde{\Gamma})t}. \end{equation} \](227)
- If \(\widetilde{\Gamma}\) is indefinite, then some modes decay while others grow, and the residual energy is not guaranteed to be monotone.
E.2.5 Proof.
For the extreme eigenvalue bounds, note that
is symmetric. If \(\widetilde{\Gamma}\succeq 0\), then for any \(x\neq 0\),
(228)
Taking the minimum over unit vectors \(x\) yields
Similarly,
(229)
and taking the maximum over unit vectors gives
Since \(M\) and \(H\) are similar by Proposition 10, they have the same eigenvalues, which proves (221).
Next, setting \(\mathbf z=K^{-1/2}\mathbf r\), we obtain
which proves (223). Since \(\mathbf r=K^{1/2}\mathbf z\), the norm equivalence (224) follows immediately.
If \(\widetilde{\Gamma}\preceq 0\), then \(H\preceq 0\). In the strictly negative definite case, \(\lambda_{\max}(H)<0\), and
(230)
Gr
specialChar{34}onwall’s inequality yields
Combining this with (224) gives
Moreover, since \(\widetilde{\Gamma}\prec 0\), we have
which proves (225).
If \(\widetilde{\Gamma}\succeq 0\), then \(H\succeq 0\), and therefore
(231)
so the energy is nondecreasing. In the strictly positive definite case \(\widetilde{\Gamma}\succ 0\), one has \(\lambda_{\min}(H)>0\), and
(232)
Applying Gr
specialChar{34}onwall’s inequality gives
which, together with (224), yields
Using
we obtain (227).
Finally, if \(\widetilde{\Gamma}\) is indefinite, then by Proposition 10, \(H\) has both positive and negative eigenvalues. Hence the linear system
contains both contractive and expansive modes, so some components decay while others grow, and monotonicity of the residual energy is no longer guaranteed. \( \square\)
E.2.6 Interpretation.
The three propositions above make the spectral distinction between the two input choices precise. With residual input, the discriminator acts as an external forcing projected onto each kernel mode. With squared-residual input, the residual state is fed back into the dynamics through the coupled operator \(K\widetilde{\Gamma}\). As a result, the squared-residual-input case forms a genuine multiplicative feedback system.
The energy behavior is therefore controlled by the sign structure of the coupled spectrum. If \(\widetilde{\Gamma}\preceq 0\), the active modes are non-expansive; if \(\widetilde{\Gamma}\prec 0\), the residual energy decays exponentially; if \(\widetilde{\Gamma}\succeq 0\), the active modes become expansive and the energy grows; and if \(\widetilde{\Gamma}\) is indefinite, the system contains both damping and amplification, which naturally leads to oscillatory or unstable behavior. This is why replacing the discriminator input \(r\) by \(r^2\) should be viewed not as a cosmetic modification, but as a structural change in the adversarial PINNs dynamics.
E.3 Interpretation and implications
The analysis above shows that replacing the discriminator input \(r\) by \(r^2\) changes the adversarial PINNs dynamics in a structural rather than merely parametric manner. In the residual-input case,
so the discriminator acts as an external sample-wise forcing term. In the squared-residual-input case,
so the discriminator enters multiplicatively through the current residual state. At the same time, the map \(r\mapsto r^2\) removes sign information, making the discriminator sensitive only to residual magnitude rather than to its sign.
This modification has two opposite consequences. On the positive side, the extra factor \(r_i\) naturally amplifies the contribution of large-residual samples, which makes the adversarial signal more consistent with the squared-residual structure of standard PINNs training and may help concentrate optimization on poorly fitted regions. On the negative side, the same mechanism automatically weakens the update as \(r_i\to 0\), which can slow late-stage refinement, and it may over-emphasize a small subset of large-residual modes, thereby worsening spectral imbalance.
Correspondingly, the energy law changes from the linear form
to the quadratic form
Thus, changing the discriminator input from \(r\) to \(r^2\) should not be viewed as a minor implementation choice: it changes the continuous-time training law, the NTK-level spectral coupling, and the mechanism by which the discriminator influences convergence.
F Soft-Constrained PINNs under Adversarial Training
In the hard-constrained setting considered in the main text, the solution ansatz is designed so that boundary and initial conditions are satisfied identically, and adversarial training is applied only to the interior PDE residual. In the soft-constrained setting, by contrast, the solution network does not enforce these constraints exactly. As a result, the PDE residual, the boundary-condition mismatch, and the initial-condition mismatch must all be controlled during training. This naturally leads to a multi-discriminator adversarial formulation, in which a single generator \(u_\theta\) is coupled to three discriminators \(f_1,f_2,f_3\), each acting on a different violation channel.
In this appendix, we derive the resulting coupled dynamics in a unified way and show that, unlike the hard-constrained case, soft-constrained adversarial PINNs training is governed by a block-structured kernel dynamics with cross-coupling among the interior, boundary, and initial-condition channels.
F.1 Problem setup and violation channels
Consider a PDE of the form
(233)
with boundary condition
(234)
and initial condition
(235)
where \(\Omega_0\) denotes the initial manifold (for example, \(t=0\) in time-dependent problems).
Let \(u_\theta\) be the generator (solution network). In the soft-constrained setting, we define three types of violations:
F.1.1 Interior residual.
Given interior collocation points \(\{x_i^r\}_{i=1}^{N_r}\subset\Omega\), define
(236)
Collect them into
(237)
F.1.2 Boundary-condition mismatch.
Given boundary collocation points \(\{x_j^b\}_{j=1}^{N_b}\subset\partial\Omega\), define
(238)
Collect them into
(239)
F.1.3 Initial-condition mismatch.
Given initial-condition points \(\{x_k^0\}_{k=1}^{N_0}\subset\Omega_0\), define
(240)
Collect them into
(241)
Thus, instead of a single residual vector, soft-constrained PINNs involve three generator-dependent violation vectors:
F.2 Adversarial formulation with three discriminators
As in the main text, the target for each violation channel is the zero state. Therefore, each channel induces a generated empirical distribution and a real distribution \(\delta_0\).
For the interior residual channel,
(242)
For the boundary channel,
(243)
For the initial channel,
(244)
We introduce three discriminators with separate parameters:
where
- \(f_1\) acts on interior residual samples \(r_i\),
- \(f_2\) acts on boundary mismatch samples \(b_j\),
- \(f_3\) acts on initial mismatch samples \(c_k\).
Using the unified GAN notation \(P,Q,R\) from the main text, the discriminator objectives are
(245)
Correspondingly, the generator minimizes the sum of three adversarial objectives:
(246)
where
(247)
Here \(\lambda_r,\lambda_b,\lambda_0>0\) are channel weights balancing the three soft constraints.
F.3 Generator gradient flow and channel-wise adversarial weights
We now derive the continuous-time generator dynamics with the discriminator parameters \(\phi_1,\phi_2,\phi_3\) frozen.
For notational convenience, define
(248)
Differentiating (246) with respect to \(\theta\) yields
(249)
Under generator gradient flow,
(250)
we define the three discriminator-induced sample-weight vectors
(251)
Let
Then (249) can be rewritten compactly as
(252)
where the three Jacobians are
(253)
and \(p=\dim(\theta)\).
Equation (252) already reveals the key difference from the hard-constrained case: the generator update is now the sum of three adversarially weighted Jacobian-transpose terms, one for each constraint channel.
F.4 Coupled violation dynamics and block kernel structure
We now derive the generator-induced dynamics for the three violation channels in the soft-constrained setting. Unlike the hard-constrained case, where the state is a single residual vector, the soft-constrained formulation involves three coupled channels: the interior residual, the boundary violation, and the initial-condition violation. Since all three channels depend on the same generator parameters, their dynamics are not independent. Instead, the resulting evolution is governed by a block kernel operator with both self-kernel and cross-kernel interactions.
Proposition 12 (Coupled soft-constrained violation dynamics)
Let \(\mathbf r\), \(\mathbf b\), and \(\mathbf c\) denote the interior, boundary, and initial-condition violation vectors, respectively, and let the generator flow be given by (252). Then the induced channel-wise dynamics satisfy
(254)
(255)
(256)
Define the self-kernel blocks
(257)
and the cross-kernel blocks
(258)
with
Then the channel-wise dynamics become
(259)
(260)
(261)
Further define the stacked violation vector
(262)
the stacked adversarial-weight vector
(263)
and the block kernel matrix
(264)
Then the coupled soft-constrained dynamics admit the compact form
(265)
F.4.1 Proof.
By differentiating the three violation vectors \(\mathbf r\), \(\mathbf b\), and \(\mathbf c\) along the generator flow (252), we obtain
(266)
This proves (254)–(256).
Introducing the self-kernel and cross-kernel blocks by
and
with the transpose relations
we immediately obtain (259)–(261).
Finally, stacking the three channels into the single vector
and defining the block kernel matrix \(K_{\mathrm{soft}}^G\) as in (264), the three coupled equations are exactly equivalent to the compact block form
which proves (265). \( \square\)
F.4.2 Interpretation.
Proposition 12 is the soft-constrained analogue of the hard-constrained adversarial PINNs dynamics in the main text. The key difference is that the state is no longer a single residual vector, but a stacked multi-channel violation vector. Accordingly, the generator-side kernel is no longer a single NTK matrix, but a block operator containing both self-channel and cross-channel interactions.
The cross-kernel blocks \(K_{rb}^G\), \(K_{r0}^G\), and \(K_{b0}^G\) are especially important. They show that even if a discriminator acts only on one channel, the resulting generator update will generally propagate to the other channels through the shared generator parameters. Thus, soft-constrained adversarial PINNs training is intrinsically coupled: interior, boundary, and initial-condition corrections cannot be analyzed independently, but must be treated as a single block dynamical system.
F.5 Diagonal-weight representation
To make the connection with the main text more explicit, define diagonal matrices
(267)
Let \(\mathbf 1_r,\mathbf 1_b,\mathbf 1_0\) denote the all-one vectors of dimensions \(N_r,N_b,N_0\), respectively. Then
Define
(268)
Then (265) can be rewritten as
(269)
This form makes clear that soft-constrained adversarial training is still a discriminator-induced sample-dependent preconditioned dynamics, but now at the level of a coupled three-channel system rather than a single residual channel.
F.6 Energy dynamics
A natural total violation energy is
(270)
Differentiating (270) and using (265) gives
(271)
Equivalently, in expanded form,
(272)
Therefore, unlike in the hard-constrained case, the decrease of total violation energy is jointly determined by
- the self-kernel blocks \(K_{rr}^G,K_{bb}^G,K_{00}^G\),
- the cross-kernel blocks \(K_{rb}^G,K_{r0}^G,K_{b0}^G\),
- the three discriminator-induced sample-weight vectors \(\boldsymbol{\gamma}^{(r)},\boldsymbol{\gamma}^{(b)},\boldsymbol{\gamma}^{(0)}\).
This means that improving one channel adversarially may either help or hurt the others, depending on the sign and structure of the corresponding cross-kernel couplings.
F.7 Constant-kernel regime and NTK interpretation
In the infinite-width or lazy-training regime, it is natural to approximate the block kernel matrix by a constant operator,
(273)
The soft-constrained adversarial dynamics then reduce to
(274)
where
Equation (274) shows that, unlike the hard-constrained case, the generator is no longer governed by a single residual-regression dynamics, but by a coupled block-NTK system in which the three discriminators jointly modulate the evolution of the interior residual, boundary mismatch, and initial-condition mismatch.
The essential new feature is the presence of cross-channel coupling. Although \(f_1,f_2,f_3\) act on the three channels separately and induce channel-wise diagonal weightings \(\Gamma_r,\Gamma_b,\Gamma_0\), these weightings are all propagated through the same shared generator kernel \(K_{\mathrm{soft}}^G\). As a consequence, an update encouraged by one discriminator generally affects not only its own channel, but also the other two through the off-diagonal kernel blocks. Therefore, the success of soft-constrained adversarial training depends not only on whether each discriminator provides a useful weighting within its own channel, but also on whether the three channels remain balanced under the shared generator dynamics.
A convenient coarse diagnostic for this balance can be obtained by examining the diagonal block interactions between the generator kernel and the discriminator-induced weighting matrices. More specifically, define
(275)
Each of these quantities summarizes, at an aggregated level, how strongly the generator-side sensitivity and the discriminator-induced sample weighting interact within the corresponding channel. In this sense, \(T_r,T_b,T_0\) can be viewed as coarse channel-wise driving strengths for the residual, boundary, and initial-condition modules, respectively.
To compare the three channels on a common scale, we further introduce the normalized balance indicators
(276)
whenever the denominator is nonzero. By construction,
(277)
These ratios provide a simple first-order measure of how the total effective adversarial drive is distributed across the three channels. If
then the residual, boundary, and initial-condition modules are being driven at roughly comparable levels. If one ratio is significantly larger than the others, then the corresponding channel is dominating the generator update; if one ratio is much smaller, then that channel is comparatively under-driven.
To quantify this deviation from balanced evolution, one may further define a scalar imbalance score
(278)
A smaller value of \(\mathcal B_{\mathrm{imb}}\) indicates that the three channels are being updated in a more balanced manner, whereas a larger value indicates that the training dynamics are biased toward only one or two components. In particular, a persistently large \(\mathcal B_{\mathrm{imb}}\) suggests that the soft-constrained adversarial training may be over-emphasizing one class of violations while neglecting the others.
Of course, these trace-based quantities do not capture the full off-diagonal coupling structure of \(K_{\mathrm{soft}}^G\). Nevertheless, they provide a useful and easily interpretable first-order diagnostic: they isolate the interaction between the generator-side diagonal kernel blocks and the discriminator-side weighting matrices, and thereby allow one to infer whether the residual, boundary, and initial-condition modules are evolving in a balanced way before analyzing the more detailed cross-channel effects.
F.8 Discriminator-side functional dynamics
For completeness, we also write the discriminator-side gradient-flow dynamics. Since the three discriminators are independent given the current generator state, each evolves on its own violation channel.
For \(\ell\in\{1,2,3\}\), let \(f_\ell(\cdot;\phi_\ell)\) denote the corresponding discriminator and let \(k_\ell^D\) be its NTK kernel. Let the fake samples be
Then, under discriminator gradient ascent,
(279)
the induced functional dynamics satisfy, for any input \(z\),
(280)
where \(N_1=N_r\), \(N_2=N_b\), and \(N_3=N_0\). Thus each discriminator learns a channel-specific weighting function shaped by its own NTK and its own empirical violation distribution.
G Existing Adaptive and Weak-Adversarial PINNs Methods as Constrained-Discriminator Special Cases
In this appendix, we show that several existing PINNs variants, including SA-PINN, LA-PINN, and weak-adversarial methods, can be embedded into a unified adversarial PINNs framework by restricting the discriminator to a prescribed functional subspace. Under this viewpoint, these methods do not require a fundamentally different generator-side training law; rather, they differ in the admissible class of discriminator functions and therefore in the induced sample-weighting or mode-weighting structure acting on the generator dynamics. However, in contrast to the label convention adopted in the main text—where the real target \( 0\) is assigned label \( 1\) and the generated data is assigned label \( 0\) —the following methods use a different normalization. Specifically, the discriminator is constrained to satisfy \( f(0)=y_{\mathrm{real}}:=0\) rather than 1.
G.1 General constrained-discriminator formulation
Let \(x\in\mathcal X\) denote the generated discrepancy variable. Depending on the method, \(x\) may represent a pointwise residual, a boundary or initial mismatch, or a weak-form discrepancy. The real target is always the zero-violation state, so the real distribution is concentrated at \(x=0\).
Consider the generator-side adversarial objective
(281)
where \(f(\cdot;\phi)\) is the discriminator (or critic), \(R\) is the generator-side adversarial potential, and \(\{x_i(\theta)\}_{i=1}^N\) are the generated discrepancies induced by the solution network \(u_\theta\).
We now restrict \(f\) to a structured family
and assume that every admissible discriminator satisfies the anchoring condition
(282)
where \(y_{\mathrm{real}}\) is the prescribed real label or real-score baseline. In other words, the discriminator is forced to match the real target exactly at the zero-violation state, and only its response on generated discrepancies \(x\neq 0\) is trainable.
Let
and define
If the generator is trained by gradient flow,
then the induced discrepancy dynamics are given by
(283)
Therefore, once the discriminator class is restricted, the generator-side dynamics are completely determined by the admissible form of \(f(x;\phi)\), or equivalently by the induced weighting law \(\gamma_i\).
G.2 SA-PINN as a monotone masked discriminator
In SA-PINN, the discriminator acts on pointwise violation quantities. More precisely, the input variable \(x\) denotes a generic pointwise discrepancy, which may represent any one of the three soft-constrained PINNs channels (236), (238) and (240):
where \(r_i\) is the PDE residual at an interior collocation point, \(b_j\) is the boundary-condition mismatch at a boundary point, and \(c_k\) is the initial-condition mismatch at an initial point.
From the viewpoint of the present paper, SA-PINN can be interpreted within the IPM-type (36) adversarial framework, namely the same general class as WGAN-type objectives, but with the discriminator restricted to a highly structured function family. Instead of optimizing over an unconstrained critic class, SA-PINN introduces trainable nonnegative pointwise weights \(m(\lambda_i)\), where the mask \(m\) is nonnegative, differentiable, and strictly increasing, and the mask parameters are updated by ascent while the solution network is updated by descent. This is equivalent to constraining the discriminator to the family
(284)
Thus the discriminator is restricted to a one-dimensional monotone family acting on squared discrepancies, rather than being free over an arbitrary nonlinear hypothesis class.
Moreover,
and, for the IPM/WGAN-type generator choice \(Q(t)=R(t)=-t\), the induced generator-side weighting reduces to
(285)
Hence SA-PINN can be interpreted as a constrained-discriminator adversarial PINNs under an IPM-type objective, whose critic is limited to a monotone mask family on \(x^2\), consistent with its original trainable-mask construction [19].
G.3 LA-PINN as a linear attentional discriminator
In LA-PINN, the discriminator also acts on pointwise violation quantities. As in the SA-PINN case, the input variable \(x\) denotes a generic pointwise discrepancy, which may represent any one of the three soft-constrained PINNs channels:
From the viewpoint of the present paper, LA-PINN can also be interpreted within an IPM-type adversarial framework, namely the same general class as WGAN-type objectives, but with the discriminator restricted to a structured linear attentional family. LA-PINN feeds pointwise squared errors into independent loss-attentional networks (LANs) and explicitly emphasizes that the attentional transformation is linear, without nonlinear activation in the LAN weighting stage. This leads naturally to the constrained discriminator family
(286)
where \(\mathrm{LAN}_{\xi}\) is a linear map parameterized by \(\xi\). The condition \(\mathrm{LAN}_{\xi}(0)=0\) ensures that
which is consistent with the zero-discrepancy target in the IPM setting.
If \(\mathrm{LAN}_{\xi}\) acts pointwise, for instance,
then the constraint \(\mathrm{LAN}_{\xi}(0)=0\) implies \(b_i(\xi)=0\), and hence
In our notation, this form is exactly equivalent to Eq. (17) in [9]. Therefore,
and, for the IPM/WGAN-type choice \(Q(t)=R(t)=-t\), the induced weighting becomes
(287)
Thus LA-PINN may be viewed as a constrained-discriminator adversarial PINNs under an IPM-type objective, with a more flexible linear attentional family than SA-PINN, but still far more structured than a generic MLP critic. This interpretation matches the original LAN design, where the pointwise squared errors are the inputs and the weighting transformation is purely linear [9].
G.4 Weak-adversarial methods as projection-type discriminators
Weak-adversarial methods such as WAN [20] do not act on pointwise strong-form residuals directly. Instead, they optimize a saddle-point problem induced by the weak formulation, where the adversarial network parameterizes a test function and seeks the most challenging weak direction for the current solution network. From the viewpoint of the present paper, this can be interpreted within the same IPM-type adversarial framework, namely the same general class as WGAN-type objectives with
but with the discriminator restricted to a normalized projection-type family in weak space:
(288)
where \(\varphi_\eta\) is the adversarial test function. Here \(x\) denotes the discrepancy object in weak space, such as the weak residual functional induced by the current generator. By contrast, the boundary and initial conditions are not modified through the discriminator, but are still enforced through standard \(L^2\)-type errors.
Unlike SA-PINN or LA-PINN, the discriminator is now restricted not to pointwise masks on \(x^2\), but to normalized rank-one projection-type forms in function space. The adversarial variable \(\eta\) searches for the most challenging direction \(\varphi_\eta\), while the generator attempts to suppress that weak projection.
Formally, writing
we have
(289)
Since
it follows that
(290)
Thus, under the IPM/WGAN-type choice \(Q(t)=R(t)=-t\), the induced generator update is still directed by the most adversarial weak test direction, but now with an additional inverse-projection normalization coming from the logarithmic form. The induced weighting becomes
(291)
Therefore, weak-adversarial PINNs can be interpreted as constrained-discriminator adversarial PINNs under an IPM-type objective, whose critic is limited to normalized weak-space projection forms, consistent with the original WAN saddle-point formulation based on induced operator norms [20].
G.5 Unified special-case theorem
G.5.1 Theorem 2 (SA-PINN, LA-PINN, and weak-adversarial methods as constrained-discriminator special cases).
Each of the following methods can be embedded into the general generator-side dynamics
by restricting the discriminator to a prescribed structured family:
(292)
In all three cases, the discriminator is anchored at the zero-violation state,
and is only allowed to vary on generated violations inside a prescribed structured subspace. Consequently, these methods differ primarily in the admissible structure of the discriminator-induced weighting \(\boldsymbol{\gamma}\), rather than in the generator-side kernel form itself.
G.5.2 Proof.
For each of the three constructions above, the discriminator family is explicitly restricted to a structured class and satisfies the zero-state anchoring condition \(f(0)=0\). Moreover, in all cases the generator-side weighting can be written in the unified form
or its weak-space analogue in the weak-adversarial case. Therefore, by Proposition 1, each method admits the same generator-side representation
Hence they are all realizations of a broader adversarial PINNs framework with constrained discriminator classes. \(\square\)
Theorem 2 has an immediate consequence for the NTK-based analysis developed in the main text.
G.5.3 Corollary 3.
The unified discriminator-induced weighting framework is not limited to standard GAN or IPM critics, but also applies to SA-PINN, LA-PINN, and weak-adversarial PINNs once their discriminators are interpreted as belonging to constrained subspaces.
G.5.4 Interpretation.
Under this viewpoint,
- SA-PINN corresponds to a monotone mask-induced weighting on pointwise squared violations;
- LA-PINN corresponds to a linear attentional weighting on pointwise squared violations;
- weak-adversarial methods correspond to a projection-induced weighting in weak space.
Therefore, these methods can be analyzed within the same generator-side NTK framework, even though their discriminator classes are much more restricted than those of generic adversarial neural networks.
More importantly, this viewpoint clarifies that the essential distinction among these methods is not whether they are formally written as GANs, but rather the functional subspace in which the discriminator is allowed to live. Once this subspace is fixed, the induced weighting law \(\boldsymbol{\gamma}\) is fixed accordingly, and the generator dynamics follow the same kernel-driven mechanism as in the general adversarial PINNs analysis.
H A broader design space for adversarial PINNs training / Future work
The analysis developed in this paper suggests that adversarial PINNs training should be viewed as a broad design space rather than as a small collection of isolated named algorithms. Two largely orthogonal design dimensions emerge naturally: the discriminator input and the discriminator objective or functional class. Existing methods only occupy a limited subset of this space.
An additional important point is that several methods that appear distinct from standard adversarial PINNs can in fact be understood within the same general framework. More precisely, many such methods do not leave the GAN/IPM paradigm altogether; instead, they can be interpreted as standard GAN-type or IPM-type adversarial schemes in which the discriminator is restricted to a structured function space. From this viewpoint, the essential distinction among methods is often not whether they are “adversarial” in a completely different sense, but rather what discriminator input is used and in what functional class the discriminator is allowed to live.
H.1 Dimension I: residual input versus squared-residual input
The first design choice concerns the quantity fed into the discriminator. Two natural options are the residual itself,
and the squared residual,
When the discriminator takes \(r\) as input, the sign information of the violation is preserved. The discriminator can therefore distinguish positive and negative residuals and induce a direction-sensitive correction signal. In the NTK residual-dynamics formulation of the main text, this leads to an additive forcing law
When the discriminator instead takes \(r^2\) as input, the sign information is removed and only the magnitude of the violation is retained. In this case, the discriminator naturally emphasizes large residuals, and the generator dynamics become multiplicatively coupled:
Thus the distinction between \(r\) and \(r^2\) is structural: the former yields a sign-sensitive additive mechanism, while the latter yields a sign-free magnitude-gated multiplicative mechanism.
H.2 Dimension II: discriminator objective and functional class
The second design choice concerns the discriminator itself. At a broad level, the methods considered in this work may be grouped into the following categories.
H.2.1 Classification-type discriminators (GAN).
These discriminators are trained through a classification-style adversarial objective and produce a signal tied primarily to separation of the zero target from the generated residual samples.
H.2.2 Regression-type discriminators (LSGAN).
These discriminators learn a smooth regression from residual samples to target labels. Compared with classification-type objectives, they generate a smoother and more local correction signal, which can be particularly suitable when the training goal is to drive every residual point toward zero.
H.2.3 Witness-function discriminators (IPM/WGAN).
These discriminators learn a witness function measuring the discrepancy between the empirical generated distribution and the real target distribution. Under the NTK view, the generator is driven by the input derivative of this witness field, so these methods emphasize global distributional discrepancy rather than direct pointwise fitting.
H.2.4 Regularized witness discriminators (WGAN-GP).
These methods retain the witness-function interpretation of IPM/WGAN while additionally regularizing the discriminator, for instance through gradient penalties, in order to improve smoothness and stability.
H.2.5 Monotone constrained discriminators (SA-PINN-type).
Here the discriminator is restricted to a trainable monotone mask acting on pointwise discrepancy magnitudes. From the present viewpoint, SA-PINN is naturally interpreted as a standard adversarial framework with the discriminator restricted to a monotone mask family.
H.2.6 Linear constrained discriminators (LA-PINN-type).
In this class, the discriminator is restricted to a linear attentional transform of pointwise squared errors. Thus LA-PINN may be understood as an adversarial scheme with a more expressive but still highly structured discriminator class.
H.2.7 Weak-space discriminators (WAN-type).
Weak adversarial methods do not act directly on strong-form pointwise residuals. Instead, they parameterize an adversarial test function in weak space and maximize the normalized weak-form discrepancy. From the present viewpoint, this again fits naturally into the same broader adversarial framework, but with the discriminator restricted to a weak-space projection class.
H.3 Why SA-PINN, LA-PINN, and WAN each have distinctive advantages
This taxonomy helps explain why SA-PINN, LA-PINN, and WAN can all be effective, but for different reasons.
H.3.1 SA-PINN.
The strength of SA-PINN lies in its simplicity and robustness. By assigning a trainable nonnegative weight to each point and updating these weights by ascent, it automatically increases the emphasis on points whose losses remain large. This makes it particularly effective at highlighting stiff or hard-to-fit regions.
H.3.2 LA-PINN.
LA-PINN inherits the pointwise weighting idea but replaces the simple monotone mask with a more expressive linear attentional map. This gives the discriminator more flexibility in redistributing emphasis across points while preserving the stability and interpretability of a structured weighting rule.
H.3.3 WAN.
The advantage of WAN is of a different nature. Rather than refining pointwise weighting, WAN changes the space in which adversarial training is performed. The discriminator becomes a weak test function, and the adversarial interaction takes place over the weak formulation of the PDE. This is particularly attractive when the weak form is more natural or more stable than the strong form.
H.4 A two-dimensional taxonomy of adversarial PINNs training strategies
The discussion in Appendix H.1–H.3 is summarized here by the two-dimensional taxonomy in Tables 3 and 4. The first table emphasizes the role of the discriminator input, while the second highlights the role of the discriminator objective or structured function class. Here, a check mark indicates that a representative method already exists in the literature or has a clear correspondence, while \( \triangle\) highlights combinations that we believe are particularly promising.
| Discriminator input | GAN | LSGAN | IPM-GAN |
| Residual \(r\) | ✓ [6] | ✓ [This paper] | ✓ [20] |
| Squared residual \(r^2\) | \( \times\) | \( \triangle\) | ✓ [7, 9, 19] |
| Adversarial framework | Unconstrained | Spectra Normalization | Gradient-Penalty(GP) | Monotone-Mask | Linear-Transform | Weak-Space |
| GAN | \( \times\) | ✓ [6] | \( \times\) | \( \times\) | \( \times\) | \( \times\) |
| LSGAN | ✓[This paper] | \( \times\) | \( \times\) | \( \times\) | \( \times\) | \( \triangle\) |
| IPM-GAN | \( \times\) | \( \times\) | ✓ [7] | ✓ [19] | ✓ [9] | ✓ [20] |
Several observations follow immediately. First, the current literature only explores a limited subset of the full design space. Second, many methods that appear quite different can still be interpreted within a unified adversarial framework once the discriminator is viewed as living in a structured function space. Third, the missing entries are themselves informative: they indicate combinations that are meaningful from the present theory but have not yet been systematically explored.
In particular, we believe that the two highlighted cases, namely \(r^2\)+LSGAN and LSGAN with weak-space constraints, are especially promising and may potentially yield stronger performance.
H.5 Summary
In summary, adversarial PINNs training is best understood as a broad design space. The choice between \(r\) and \(r^2\) determines whether the discriminator preserves sign information or focuses on magnitude, and whether the induced generator dynamics are additive or multiplicative. The choice of discriminator objective or function class determines whether the discriminator behaves like a classifier, a regression map, a witness function, or a constrained weighting operator. Existing methods such as SA-PINN, LA-PINN, and WAN should therefore be viewed not as isolated constructions, but as representative points in a much larger structured family of adversarial PINNs training strategies.
I Experiments
I.1 Experimental setup details
This appendix provides the detailed experimental setup corresponding to the results reported in Section 5.
I.1.1 Compared methods
We compare the following adversarial PINNs methods:
- DEQGAN,
- GAN,
- LSGAN,
- WGAN-GP,
- GAN-RB,
- LSGAN-RB,
- WGAN-GP-RB.
Here the suffix “RB” denotes the proposed rollback strategy. For the rollback-enhanced variants, rollback selection is performed within at most 20 generator updates and 20 discriminator updates in each outer iteration.
Although the preceding theoretical discussion shows that a broader family of existing methods can also be interpreted as special cases of adversarial training under constrained discriminator classes, in the experiments we deliberately focus on the most basic adversarial objectives, namely GAN, LSGAN, and WGAN-GP. The reason is that these methods represent the canonical adversarial design choices and therefore provide the cleanest setting for evaluating the proposed framework. By starting from these standard baselines, we can more clearly separate the effect of the adversarial objective itself from the effect of the proposed update-control mechanism, without introducing additional architectural or formulation-specific factors.
DEQGAN is included for a different reason. In fact, our study was initially motivated by the distinctive phenomena observed in DEQGAN under different generator–discriminator update ratios. These observations led us to identify a gap between the empirical behavior of adversarial PINNs training and the classical divergence-based explanation, and further motivated the unified residual-dynamics framework developed in this paper. In this sense, DEQGAN serves not only as a strong existing baseline, but also as the starting point from which we first discovered the phenomenon, constructed the explanatory framework, and then generalized the resulting analysis to a broader class of adversarial PINNs methods.
The purpose of including both the original and rollback-enhanced variants is therefore twofold: first, to evaluate the adversarial objectives themselves under a unified setting; and second, to isolate the contribution of the proposed rollback mechanism from the choice of adversarial objective.
I.1.2 Benchmark equations
To evaluate the robustness of the proposed framework across different PDE types, we consider the following benchmark problems:
- Poisson equation,
- Laplace equation,
- Viscous Burgers equation,
- Reaction–Diffusion equation,
- Klein–Gordon equation.
These problems cover a diverse range of settings, including linear and nonlinear equations, stationary and time-dependent systems, as well as equations with different propagation, diffusion, and stiffness characteristics. This diversity is intended to test whether the conclusions drawn from the proposed residual-dynamics analysis generalize beyond a single PDE family.
I.1.3 Network architectures
For fairness, the generator architecture is kept fixed across all methods and uses residual links. The discriminator architecture depends on the adversarial objective.
For DEQGAN, GAN, and GAN-RB, we use the standard discriminator configuration of GAN-style adversarial training:
- residual links,
- spectral normalization,
- a non-regressive output with a final sigmoid activation.
For LSGAN, LSGAN-RB, WGAN-GP, and WGAN-GP-RB, the discriminator is treated as a critic:
- no residual links,
- no spectral normalization,
- no final sigmoid activation.
All hidden layers use \(\tanh\) activation.
I.1.4 Optimization settings
A central practical distinction in our study is that, except for DEQGAN, all methods are trained under the same simple default optimization setting:
(293)
with Adam optimizer and default hyperparameters
(294)
By contrast, DEQGAN follows its original tuning-dependent protocol (Table 5), which relies on:
- two-timescale generator/discriminator learning rates,
- tuned Adam hyperparameters,
- Ray Tune-based hyperparameter search.
This distinction is particularly important because one of the central goals of our work is to demonstrate that favorable alternating training dynamics can be effectively recovered through the proposed rollback principle, while avoiding the substantial hyperparameter tuning burden and implementation cost typically required by DEQGAN.
| Hyperparameter | Poisson | Laplace | Viscous–Burgers | Reaction–Diffusion | Klein–Gordon |
| Num. iterations | 3000 | 3000 | 3400 | 4500 | 4000 |
| Num. grid points | \( 32\times32\) | \( 32\times32\) | \( 32\times32\) | \( 32\times32\) | \( 32\times32\) |
| G units/layer | 50 | 50 | 50 | 50 | 50 |
| G num. layers | 4 | 4 | 3 | 3 | 3 |
| D units/layer | 30 | 50 | 20 | 20 | 20 |
| D num. layers | 2 | 2 | 5 | 5 | 5 |
| Activations | tanh | tanh | tanh | tanh | tanh |
| G learning rate | 0.019 | 0.012 | 0.012 | 0.007 | 0.012 |
| D learning rate | 0.021 | 0.088 | 0.005 | 0.009 | 0.005 |
| G \( \beta_1\) (Adam) | 0.139 | 0.295 | 0.185 | 0.185 | 0.185 |
| G \( \beta_2\) (Adam) | 0.369 | 0.358 | 0.594 | 0.594 | 0.594 |
| D \( \beta_1\) (Adam) | 0.745 | 0.575 | 0.093 | 0.093 | 0.093 |
| D \( \beta_2\) (Adam) | 0.759 | 0.133 | 0.184 | 0.184 | 0.184 |
| Exponential LR decay \( (\gamma)\) | 0.957 | 0.953 | 0.954 | 0.954 | 0.954 |
| Decay step size | 9 | 10 | 25 | 10 | 10 |
| PDE | DEQGAN | GAN | LSGAN | WGAN-GP | GAN-RB | LSGAN-RB | WGAN-GP-RB |
| Poisson | 0.24 | 0.33 | 0.28 | 0.29 | 3.17 | 3.02 | 2.98 |
| Laplace | 0.28 | 0.27 | 0.27 | 0.29 | 3.33 | 3.27 | 3.24 |
| Burgers | 0.46 | 0.41 | 0.42 | 0.37 | 4.62 | 4.48 | 4.47 |
| Reaction | 0.33 | 0.32 | 0.29 | 3.86 | 4.11 | 4.09 | 4.07 |
| Klein | 0.45 | 0.43 | 0.44 | 0.36 | 4.98 | 4.82 | 4.69 |
I.1.5 Training protocol
For each outer training iteration, the generator and discriminator are updated alternately. The rollback-enhanced variants additionally evaluate candidate intermediate states and retain only those updates that improve the corresponding first-order criterion or residual-energy criterion, as described in the main text.
Unless otherwise specified, all methods are trained under the same sampling and optimization pipeline on each benchmark problem. This design ensures that performance differences are primarily attributable to the adversarial objective and the rollback mechanism, rather than to unrelated implementation choices. All experiments are implemented in PyTorch and conducted on a laptop workstation equipped with an NVIDIA GeForce RTX 4070 Laptop GPU and an Intel Core i7-14700HX CPU. The detailed hardware configuration and running-time (hours) statistics are summarized in Table 6.
I.1.6 Evaluation metrics
We report both training error (L2 error) and validation error (L2 error) as the primary accuracy metrics. To connect the experiments with the theory developed in the main text, we additionally track:
-
the residual energy
\[ \begin{equation} E=\frac12\|\mathbf r\|^2, \end{equation} \](295)
-
the first-order indicator
\[ \begin{equation} S=\mathbf r^\top K_{rr}^{G}\boldsymbol{\gamma}, \end{equation} \](296)
- and, for rollback-based methods, the selected discriminator and generator rollback depths.
These quantities are not only useful diagnostics but also provide a direct empirical bridge to the NTK-based residual-dynamics analysis developed in the paper.
I.1.7 Relation to the main text
The main text reports the core quantitative comparisons and the principal mechanistic observations. Detailed training curves, per-equation dynamics, rollback-step statistics, and additional ablation results are deferred to the appendix in order to keep the main paper focused on the central theoretical and algorithmic message.
We note that the mechanistic analysis of DEQGAN under different \(G:D\) ratios has already been presented systematically in Section 5.2 of the main text. There, based on the first-order indicator and the NTK-based failure-mode analysis, we explained the phenomena identified in Section 3, clarified the apparent mismatch between empirical observations and the classical theoretical picture, reconstructed the training-dynamics viewpoint, and supported the resulting interpretation with numerical evidence. Since these components already form a coherent narrative in the main paper, we do not repeat them again in the appendix.
I.2 Benchmark equations Comparison
I.2.1 Laplace
I.2.1.1 Laplace equation:
We first consider a two-dimensional Laplace equation posed on the unit square domain
with Dirichlet boundary conditions. The governing equation is
The Laplace equation is one of the most fundamental elliptic partial differential equations and commonly arises in steady-state heat conduction, electrostatic potential analysis, incompressible fluid flow, and other equilibrium problems.
In the present example, the problem is defined on a bounded rectangular domain with mixed homogeneous and non-homogeneous Dirichlet boundary conditions, making it a standard benchmark for evaluating the approximation quality of physics-informed methods.
For this problem, an analytical solution is available:
Hence, the numerical prediction can be directly compared with the exact solution in both solution error and PDE residual.
I.2.1.2 Comparison among different methods:
Fig. 5(a)–(c) reports the training error, validation error, and residual energy dynamics of different adversarial PINNs variants on the Laplace equation. Overall, the rollback-based methods significantly improve the corresponding original adversarial methods. In particular, LSGAN-RB and GAN-RB achieve the best performance, with both the residual energy and prediction errors reduced by several orders of magnitude compared with their non-rollback counterparts. This demonstrates that the proposed rollback strategy can effectively stabilize and accelerate adversarial PINNs training by selecting more favorable generator–discriminator update states.
An interesting observation is that WGAN-GP does not perform well in this experiment, although it is commonly believed to alleviate the gradient-vanishing issue in standard GAN training. Even after applying the rollback strategy, WGAN-GP-RB still fails to achieve comparable accuracy. This suggests that avoiding discriminator saturation alone is not sufficient for adversarial PINNs training. From the perspective of our framework, the key factor is whether the discriminator-induced weighting \( \gamma\) provides an informative and well-aligned residual descent direction under the generator NTK. In this case, the WGAN-type gradient field appears to be less effective for the residual dynamics of the Laplace equation.
We further analyze the three rollback variants in Fig. 5(d)–(f), which show the evolution of the first-order indicator, the selected generator update steps, and the selected discriminator update steps during training. For better visualization, all three quantities are smoothed by a moving average over 300 iterations. In the following experiments, we adopt the same analysis strategy. The main quantity of interest is the number of selected generator update steps. According to the rollback criterion, the generator is allowed to continue updating only when the current discriminator-induced direction leads to a decrease in the residual energy. Therefore, a larger selected generator update ratio indicates that the discriminator feedback remains useful over a longer generator trajectory and can continuously drive the residual energy downward.
Fig. 5(d) and (e) correspond to GAN-RB and LSGAN-RB, respectively. We observe that both methods maintain a high generator update ratio for most of the training process, indicating that their discriminator-induced gradient fields provide effective descent directions for the generator. This is consistent with their strong performance in Fig. 5(a)–(c). In contrast, Fig. 5(f) shows that WGAN-GP-RB keeps the generator update steps at a very low level throughout training. This means that the rollback criterion frequently rejects generator updates, implying that the WGAN-GP discriminator fails to provide a sufficiently useful residual descent direction in this problem.
The behavior of the first-order indicator further supports this interpretation. Under our plotting convention, a negative value corresponds to a favorable energy-decreasing direction. For GAN-RB and LSGAN-RB, the first-order indicator is largely suppressed in the negative region during training, which explains why their generator updates can be repeatedly accepted. By contrast, for WGAN-GP-RB, the indicator rapidly moves toward the positive region at the early stage and then becomes ineffective, indicating poor alignment between the discriminator-induced weighting and the generator NTK descent direction. This provides direct empirical evidence for our theoretical claim that the success of adversarial PINNs training depends not only on the adversarial loss itself, but more fundamentally on the quality of the induced residual-space weighting and its alignment with the generator residual dynamics.
I.2.2 Poisson
I.2.2.1 Poisson equation
We next consider a two-dimensional Poisson equation posed on the unit square domain
with homogeneous Dirichlet boundary conditions. The governing equation is given by
The Poisson equation is one of the most classical elliptic partial differential equations and arises widely in electrostatics, steady-state heat conduction, gravitational potential modeling, and incompressible flow analysis. In the present example, the problem is defined on a bounded rectangular domain with a prescribed source term and zero boundary values, which makes it a standard benchmark for testing the approximation accuracy and training stability of PINNs-type methods.
For this problem, an analytical solution is available:
Therefore, both the solution error and the PDE residual can be evaluated directly, making this example particularly suitable for quantitative Comparison among different methods: .
I.2.2.2 Comparison among different methods:
For the Poisson equation, Fig. 12(a)–(c) shows that LSGAN-RB achieves the best convergence behavior among all compared methods. It attains lower training error, validation error, and residual energy than the original DEQGAN baseline, indicating that the proposed rollback strategy can further improve adversarial PINNs training beyond the existing GAN-based formulation. Here, WGAN-GP still performs poorly, and the application of rollback does not lead to a clear improvement. This again suggests that the non-saturating property of WGAN-type objectives alone is insufficient to guarantee effective residual descent.
Fig. 12(d)–(f) further explain this behavior from the perspective of the rollback dynamics. GAN-RB and LSGAN-RB maintain relatively high selected generator update steps during most of the training process, which means that the discriminator-induced directions can support a longer sequence of energy-decreasing generator updates. In contrast, WGAN-GP-RB keeps the generator update steps at a much lower level, implying that many candidate generator updates are rejected by the rollback criterion.
The first-order indicator shows a consistent trend. For GAN-RB and LSGAN-RB, the indicator remains at a lower level during training, suggesting better alignment between the discriminator-induced residual weighting and the generator NTK dynamics. By contrast, the WGAN-GP-RB indicator becomes large already in the early stage, indicating an unfavorable or poorly aligned descent direction. Since the qualitative interpretation is consistent with the Laplace experiment, we omit repeated details here. Overall, the Poisson results further confirm that the effectiveness of adversarial PINNs training depends on the quality of the induced residual-space weighting rather than merely on the nominal GAN objective.
I.2.3 Reaction-Difussion
I.2.3.1 Reaction-Difussion equation
We consider a one-dimensional time-dependent parabolic equation on the space–time domain
subject to homogeneous Dirichlet boundary conditions and a zero initial condition. The governing equation is
This type of parabolic equation is commonly used to describe diffusion and heat-transfer processes, and more generally appears in time-dependent transport-reaction problems. For the present example, an analytical solution is available:
Hence, the predicted solution can be directly compared with the exact solution, which makes this problem convenient for evaluating both approximation accuracy and training behavior.
I.2.3.2 Comparison among different methods:
For the reaction–diffusion equation, Fig. 19(a)–(c) shows that GAN-RB achieves the best accuracy among all compared methods. Compared with the other baselines, GAN-RB improves the training accuracy by approximately four orders of magnitude, demonstrating that the rollback strategy can be particularly effective for nonlinear or reaction-dominated residual dynamics.
Fig. 19(d)–(f) further illustrate the underlying update behavior. Both GAN-RB and LSGAN-RB maintain relatively high selected generator update steps during training, indicating that their discriminator-induced directions can still support repeated energy-decreasing generator updates. However, compared with GAN-RB, the first-order indicator of LSGAN-RB remains relatively large at the early stage. Since this indicator is an aggregated quantity, this suggests that the discriminator feedback of LSGAN-RB is less well aligned with the global residual-energy descent direction, although it can still produce decreasing residual energy. This explains why LSGAN-RB converges, but not as rapidly or accurately as GAN-RB.
For WGAN-GP-RB, the rollback strategy brings a slight improvement over the original WGAN-GP. As shown in Fig. 19(f), the WGAN-GP discriminator provides a useful descent direction at the very beginning of training. Nevertheless, this favorable behavior quickly disappears, and the selected generator updates become less effective afterward. As a result, the overall training performance remains unsatisfactory. These results again indicate that the success of the rollback mechanism depends on whether the discriminator-induced weighting can provide a persistently informative residual descent direction.
I.2.4 Viscous-Burgers
I.2.4.1 Viscous Burgers equation
We consider a one-dimensional viscous Burgers equation on the space–time domain
with viscosity coefficient \(\nu=0.001\). The governing equation is
In the implementation, the boundary values at \(x=\pm 5\) are fixed to \(1/\cosh(5)\), which is numerically very close to zero.
The viscous Burgers equation is a classical nonlinear convection–diffusion model and is widely used as a benchmark for studying nonlinear transport, diffusion, and sharp-gradient dynamics. Owing to the competition between nonlinear advection and viscosity, it provides a representative test case for assessing the stability and accuracy of physics-informed methods.
For this problem, the reference solution is not available in closed form. Instead, a high-accuracy numerical solution computed by an FFT-based solver is used for comparison.
I.2.4.2 Comparison among different methods:
For the Burgers equation, Fig. 26(a)–(c) shows that all methods exhibit strongly oscillatory training dynamics. This behavior is reasonable because the Burgers equation contains nonlinear convection, and its residual landscape is more sensitive to local sharp structures and dynamically evolving error modes. As a result, the adversarial feedback may change rapidly during training, leading to more pronounced fluctuations than in the linear elliptic problems.
Despite these oscillations, GAN-RB achieves the best overall performance. This indicates that, for nonlinear convection-dominated residual dynamics, the sharper discriminator-induced weighting of GAN, together with the rollback selection mechanism, can provide more effective residual descent directions than the smoother LSGAN-type weighting.
Fig. 26(d)–(f) further support this interpretation. Among the three rollback variants, GAN-RB maintains the largest selected generator update steps, followed by LSGAN-RB, while WGAN-GP-RB keeps the lowest generator update level. This ordering is consistent with their final accuracy. The first-order indicator of GAN-RB is also the most favorable and remains better suppressed during training, suggesting that its discriminator-induced weighting is better aligned with the generator residual dynamics.
For LSGAN-RB, the first-order indicator is negative at the beginning, indicating an initially useful descent direction. However, it gradually moves toward the positive region and remains there, which weakens the effectiveness of subsequent generator updates and leads to an inferior final result compared with GAN-RB. WGAN-GP-RB performs the worst: its first-order indicator is poorly suppressed and the selected generator update steps remain low, implying that the WGAN-GP-RB discriminator fails to provide a persistently useful residual descent direction for this nonlinear problem.
I.2.5 Klein-Gordon
I.2.5.1 Klein-Gordon equation
We further consider a one-dimensional Klein–Gordon-type equation on the space–time domain
with homogeneous Dirichlet boundary conditions. The governing equation is
The Klein–Gordon equation is a classical hyperbolic partial differential equation and is widely used to describe wave propagation with restoring or reaction effects. It also serves as a representative benchmark for testing whether learning-based solvers can accurately capture second-order temporal dynamics together with spatial oscillations.
For the default choice \(c=1\), the problem admits the analytical solution
Hence, the predicted solution can be directly compared with the exact solution in both solution error and PDE residual.
I.2.5.2 Comparison among different methods:
For the Klein–Gordon equation, Fig. 33(a)–(c) shows that GAN-RB achieves the best accuracy among all methods. Its final error is approximately four orders of magnitude lower than the other baselines, demonstrating that GAN-RB is particularly effective for this oscillatory and wave-like problem.
From Fig. 33(d)–(f), we observe that both GAN-RB and LSGAN-RB maintain relatively high selected generator update steps during training. However, GAN-RB obtains a much larger improvement after applying the rollback strategy. This indicates that, in the Klein–Gordon problem, the GAN-induced residual weighting is better aligned with the generator residual dynamics than the smoother LSGAN-type weighting.
The first-order indicator provides further evidence. For GAN-RB, the indicator is globally well suppressed in the favorable region throughout training, which is a strong indication that the discriminator feedback persistently provides useful descent directions. In contrast, LSGAN-RB also produces a favorable indicator at the beginning, but it gradually increases and eventually stays near a small positive value around zero. This means that the discriminator-induced direction becomes less effective in later training stages, explaining why LSGAN-RB is inferior to GAN-RB.
For WGAN-GP-RB, the rollback strategy also brings a clear improvement, reducing the error by about two orders of magnitude compared with the original WGAN-GP. Different from the previous examples, Fig. 33(f) shows that the selected generator update steps of WGAN-GP-RB are significantly higher in this experiment. This suggests that the WGAN-GP discriminator can provide more useful descent directions for the Klein–Gordon residual dynamics than in the Laplace, Poisson, or reaction–diffusion cases. Nevertheless, its first-order indicator is still less stable and less favorable than that of GAN-RB, so its final performance remains worse.
I.3 Ablation experiment
I.3.1 Laplace equation (Under different boundary conditions)
Here, we conduct the first ablation experiment to examine the robustness of the proposed method under changes in boundary conditions. The motivation for this experiment is that, even when the governing PDE remains unchanged, the solution can be highly sensitive to the prescribed boundary conditions. Therefore, it is important to verify whether the proposed algorithm can still maintain stable and accurate performance when the boundary conditions are modified.
Specifically, we keep the differential operator and the computational domain fixed, and only change the boundary condition setting. This allows us to isolate the effect of boundary-induced changes in the solution structure, while avoiding interference from changes in the PDE type itself. If the proposed method still achieves strong performance under this setting, it provides evidence that the improvement is not limited to one specific boundary configuration, but reflects a more robust residual optimization mechanism.
I.3.1.1 Laplace with different boundary condition:
We also consider a two-dimensional Laplace equation on the unit square domain
subject to Dirichlet boundary conditions:
Compared with simpler homogeneous-boundary examples, this test case involves nontrivial non-homogeneous boundary data and is therefore useful for evaluating the capability of a method to handle more complex boundary structures.
For this problem, an analytical solution is available:
Hence, the numerical prediction can be directly compared with the exact solution.
I.3.1.2 Comparison among different methods:
For the second Laplace setting, Fig. 40 evaluates the same Laplace equation under different boundary conditions. As shown in Fig. 40(a)–(c), both GAN-RB and LSGAN-RB achieve strong convergence performance. Compared with the other methods, they improve the final accuracy by about two orders of magnitude. This result indicates that the proposed rollback strategy remains effective even when the boundary conditions are changed, suggesting good robustness with respect to boundary-induced variations in the solution structure.
Fig. 40(d) and (e) further show that GAN-RB and LSGAN-RB maintain a large number of selected generator update steps during training. At the same time, the discriminator updates help keep the first-order indicator mostly below zero, meaning that the discriminator-induced directions remain favorable for residual-energy reduction. This explains why both methods can achieve stable and accurate convergence in this modified boundary setting.
In contrast, WGAN-GP-RB still performs poorly. As shown in Fig. 40(f), its first-order indicator quickly moves to a large positive value at the early training stage, indicating poor alignment between the WGAN-GP-induced weighting and the generator residual dynamics. Consequently, even with rollback, WGAN-GP-RB cannot provide sufficiently effective descent directions for this problem.
I.3.2 Compatibility with DEQGAN
To further evaluate the compatibility of the proposed rollback strategy, we combine it with DEQGAN and test the resulting method on the Laplace equation introduced in Section . The purpose of this experiment is not merely to seek another performance gain, but to clarify an important conceptual point: rollback is not a standalone algorithm disconnected from existing adversarial PINNs methods. Rather, it should be understood as a dynamical modification of the alternating optimization process motivated by the NTK-based residual analysis developed in this paper.
From this viewpoint, rollback does not replace the underlying adversarial objective or discriminator design. Instead, it operates at the level of update selection, using the first-order residual-dynamics criterion to filter candidate alternating steps and retain those more consistent with descent-oriented residual evolution. Therefore, if our interpretation is correct, the rollback principle should be compatible not only with GAN and LSGAN, but also with existing adversarial PINNs frameworks such as DEQGAN.
To verify this, we apply rollback on top of DEQGAN under the Laplace benchmark. This setting is particularly informative because DEQGAN already relies on carefully tuned hyperparameters and a specific two-timescale training protocol. If rollback can still improve performance in this case, then its benefit cannot be attributed merely to replacing weak baselines or to using simpler optimization settings. Instead, it would support the stronger claim that rollback captures a more general principle of alternating adversarial PINNs training.
The results confirm this expectation: even when combined with DEQGAN, the rollback-enhanced variant achieves improved convergence behavior and better final accuracy on the Laplace problem. This indicates that the proposed rollback mechanism is compatible with existing adversarial PINNs methods and can serve as an additional optimization layer on top of them. In this sense, the contribution of rollback is not to define a completely separate training paradigm, but to provide a principled update-control mechanism that can be integrated into broader adversarial PINNs frameworks.
I.3.2.1 Comparison among different methods:
In Fig. 47, we further combine the rollback strategy with the existing DEQGAN method to demonstrate the flexibility and broad applicability of the proposed strategy. As shown in Fig. 47(a)–(c), the resulting DEQGAN-RB method achieves the best accuracy among all compared methods. This indicates that rollback is not restricted to the standard GAN, LSGAN, or WGAN-GP formulations, but can also be incorporated into existing adversarial PINNs frameworks to further improve their stability and accuracy.
An interesting observation is that the original DEQGAN performs poorly when the generator–discriminator update ratio is fixed to \(20:20\). This is reasonable because the original DEQGAN hyperparameters were tuned for a different update setting, such as a \(1:1\) update ratio. In contrast, after incorporating rollback, DEQGAN-RB can adaptively select more favorable update states and therefore achieves much better performance under the same update budget. We also include GAN-RB as a comparison. The results show that, when rollback is applied on top of a properly tuned adversarial training method, it can produce more stable training dynamics and higher final accuracy.
This observation suggests that rollback can serve as a general stabilization module for adversarial PINNs training. Across the previous five numerical experiments, adversarial training methods often exhibit strong fluctuations, especially on nonlinear or oscillatory PDEs. Therefore, combining rollback with existing adversarial PINNs methods is a meaningful direction for improving both robustness and training stability.
Fig. 47(d) and (e) compare the rollback dynamics of DEQGAN-RB and GAN-RB. We observe that DEQGAN-RB selects fewer generator update steps than GAN-RB. This does not contradict the previous analysis, because the two methods use different learning rates. DEQGAN-RB adopts a larger learning rate, so each accepted generator update can produce a larger decrease in the residual energy. As a result, fewer accepted generator steps are sufficient to achieve strong convergence. In contrast, GAN-RB uses more generator updates, but its first-order indicator shows much stronger fluctuations. After hyperparameter tuning and integration with the existing DEQGAN framework, the rollback indicator becomes more stable, which explains the superior performance of DEQGAN-RB in this experiment.
In Fig. 47(f), we compare the first-order indicator across four methods. It can be observed that the DEQGAN method achieves its best performance under a \( 1{:}1\) generator–discriminator update ratio. This is because, when the number of alternating updates per outer iteration is small, the generator and discriminator—after careful hyperparameter tuning—are less likely to become excessively strong or weak relative to each other. However, since the total number of updates is strictly limited under the \( 1{:}1\) setting, the overall optimization process remains insufficiently explored, and the final performance is therefore suboptimal. When the update budget is increased, e.g., to a \( 20{:}20\) ratio, a different phenomenon emerges. We observe that the first-order indicator rapidly approaches zero within a very small number of iterations. This suggests that the discriminator becomes effectively over-trained relative to the generator: the discriminator gradients quickly vanish, leading to \( \boldsymbol{\gamma} \approx \mathbf{0}\) and thus a near-degenerate training signal for the generator. In other words, simply increasing the update budget of DEQGAN is not sufficient to guarantee improved optimization performance. For the GAN-RB method, although no extensive hyperparameter tuning is performed, it achieves better results than the \( 1{:}1\) baseline. However, its first-order indicator is noticeably unstable. As shown in Fig. F, there exist several abrupt positive spikes (i.e., indicator values exceeding zero), which correspond to regions where the loss exhibits significant oscillations. This instability reflects inconsistent descent behavior in the residual energy. Finally, we emphasize that the proposed DEQGAN-RB method achieves the most stable first-order indicator under the \( 20{:}20\) update ratio. This stability directly translates into more consistent energy decay and smoother training dynamics, ultimately leading to higher solution accuracy. These observations suggest that combining existing adaptive alternating training strategies with the proposed first-order rollback mechanism could yield new methods with both improved stability and accuracy, which provides an important direction for future research.
I.3.3 Does rollback help beyond increasing the update budget?
A natural concern about the proposed rollback strategy is that it evaluates multiple candidate updates within each outer iteration, and therefore may appear to benefit simply from using more update steps. To rule out this possibility, we perform a controlled ablation and examine whether the observed performance gain truly comes from rollback itself.
We focus on the two rollback-enhanced methods that perform best overall in our experiments, namely GAN-RB and LSGAN-RB. For GAN-RB, we use the Klein–Gordon equation benchmark in Section , where GAN-RB performs best. For LSGAN-RB, we use the Laplace equation in Section , where this method achieves its strongest performance.
For each method, we compare three training settings: (i) no rollback with \(G:D=1:\!1\), (ii) no rollback with \(G:D=20:\!20\), and (iii) rollback with \(G:D=20:\!20\). This design is chosen to answer two separate questions. The comparison between settings (i) and (ii) tests whether simply increasing the number of generator and discriminator updates can improve the solution accuracy. The comparison between settings (ii) and (iii) then isolates the effect of rollback under the same \(20:\!20\) update budget, and asks whether adding the proposed optimization strategy on top of that fixed budget leads to further improvement. Therefore, if the third setting consistently outperforms the second, the gain cannot be attributed merely to more candidate updates, but must come from the rollback mechanism itself.
I.3.3.1 Laplace.
From Fig. 54, several clear observations can be made. First, Fig. 54 (a)–(c) show that among the three compared settings, the fixed \(G:D=20{:}20\) scheme without rollback performs the worst, while the rollback-enhanced \(20{:}20\) scheme achieves the best overall convergence. A particularly interesting phenomenon is that the non-rollback \(20{:}20\) method actually exhibits very good optimization behavior before roughly 1700 iterations, but then suddenly collapses, with both the training and validation errors increasing sharply.
This abrupt failure can be better understood from Fig. 54 (d), where we plot, for the non-rollback \(20{:}20\) method, the best discriminator/generator inner-step statistics together with the first-order indicator \(r^\top K_{rr}\gamma\). At the beginning of training, this indicator remains stably negative, suggesting that the discriminator-induced signal is well aligned with residual descent. However, around iteration 1500, the indicator undergoes a large jump, and at almost the same time the best generator update depth drops sharply to around 6 steps. Since the generator update budget is fixed to 20 steps, this means that after the useful descent-oriented regime has already been exhausted, the generator continues to update for many additional steps under a deteriorated discriminator signal. As a result, the accumulated error is rapidly amplified, causing the sudden jump in loss and the eventual training failure.
By contrast, Fig. 54 (e) shows that after introducing rollback, the first-order indicator is maintained within a slightly negative but much more stable range, and the effective generator update depth remains consistently high and stable throughout training. At the same time, the discriminator also keeps a relatively high effective update ratio during the early stage, indicating that it is still able to push the first-order indicator toward a favorable regime. In this sense, rollback acts on both sides of the alternating process: it prevents the generator from taking too many harmful steps once the discriminator signal becomes unreliable, and it also prevents the discriminator from continuing unnecessary updates after a sufficiently good state has already been reached.
This observation also helps explain why the non-rollback \(20{:}20\) scheme can perform well at the beginning of training. In the early stage, allowing the discriminator to take many updates is indeed beneficial, because it helps drive the first-order indicator into a good negative region and thus yields rapid error reduction. However, as training proceeds, keeping the discriminator fixed at 20 updates becomes excessive. At that point, the discriminator no longer needs such a strong update budget, and continuing to optimize it too aggressively starts to destroy the previously favorable dynamical balance. The rollback mechanism effectively corrects this issue by adapting the usable update depth to the current training state, thereby maintaining stability while preserving accuracy.
Finally, Fig. 54 (f) compares the evolution of \(r^\top K_{rr}\gamma\) across the three settings. One can see that the overall indicator of the \(1{:}1\) baseline is comparable to that of \(20{:}20\) with rollback. However, the important point is that once the generator and discriminator update budgets are both increased, maintaining a favorable first-order indicator becomes much more difficult. This is precisely why the plain \(20{:}20\) scheme fails: without rollback, the enlarged update budget makes the dynamics far more vulnerable to abrupt misalignment, which then causes the catastrophic jump observed in training. In this sense, the comparison shows that the benefit of rollback is not merely to improve performance, but to make larger alternating-update budgets practically usable.
I.3.3.2 Klein–Gordon.
From Fig. 61, a different but equally informative pattern can be observed on the Klein–Gordon benchmark. Fig. 61 (a)–(c) show that the \(G:D=1{:}1\) baseline gives the worst final accuracy, the fixed \(20{:}20\) scheme without rollback performs better, and the rollback-enhanced \(20{:}20\) scheme achieves the best overall result. In this example, increasing the generator and discriminator update budgets does indeed improve the solution accuracy, which is different from the Laplace case above.
At the same time, Fig. 61 (d) shows that for the non-rollback \(20{:}20\) scheme, the first-order indicator \(r^\top K_{rr}\gamma\) is kept in a relatively stable negative region throughout most of training. This indicates that the alternating dynamics are more stable here than in the previous example. However, the same Fig. 61 also shows that the best generator update depth is only around 10 steps in the early stage. Therefore, when the generator budget is fixed to 20 updates, a substantial fraction of these updates are already beyond the most effective descent-oriented range. In other words, even though the overall indicator remains negative, the generator is still being updated more than necessary.
Fig. 61 (e) shows the effect of adding rollback under the same \(20{:}20\) budget. In this case, the generator consistently operates within a higher and more stable effective update range, while the first-order indicator is further improved. This suggests that rollback does not merely preserve stability, but also makes the enlarged update budget more efficient by filtering out less useful updates and retaining those that are better aligned with the current descent direction.
Finally, Fig. 61 (f) compares the evolution of \(r^\top K_{rr}\gamma\) across the three settings. One can see that although the \(1{:}1\) baseline has the most favorable first-order indicator overall, its small update budget leads to inferior final accuracy. Once the generator and discriminator update numbers are increased, the first-order indicator deteriorates immediately in the plain \(20{:}20\) setting. After rollback is introduced, however, the indicator is clearly improved again. This explains why the rollback-enhanced method achieves the best performance: it not only allows the larger alternating-update budget to be used, but also restores a better first-order descent quality under that larger budget.
I.3.3.3 Conclusion:
This ablation supports the interpretation that rollback improves adversarial PINNs training by selecting more descent-consistent alternating updates, rather than by merely enlarging the optimization budget. In other words, the performance improvement is attributable to better update control, not to more iterations.
J Contributions, Limitations, and Future Work
J.1 Contributions:
Although the present work still has several limitations, we believe that its main contributions are already clear. First, from the perspective of residual dynamics and NTK analysis, we develop a unified framework for adversarially trained PINNs. In contrast to the classical static picture based on the optimal discriminator and divergence minimization, we emphasize that practical training is governed by finite-step alternating dynamics, and we show how different adversarial objectives induce different residual-weighting mechanisms through the discriminator input derivative. Second, based on this framework, we propose a first-order rollback strategy and demonstrate across multiple PDE benchmarks that it can substantially improve GAN-based and LSGAN-based adversarial PINNs without requiring expensive hyperparameter search. Third, we show that this viewpoint is not limited to standard GAN variants: it also provides a unified interpretation of squared-residual input, soft-constrained settings, and existing methods such as SA-PINN, LA-PINN, and WAN, thereby suggesting a broader design space for adversarial PINNs training.
J.2 Limitations:
Against this background, several limitations remain.
First, the rollback strategy, while effective, can still introduce noticeable oscillations during training. In some sense, this is natural, since rollback explicitly compares and reverts among candidate alternating updates, so the resulting trajectory is not expected to be completely smooth. Nevertheless, such oscillations indicate that the current rollback mechanism still has room for improvement in terms of stability. At the same time, our ablation results (I.3.2) show that when rollback is combined with an already well-tuned model, it can achieve both higher accuracy and more stable dynamics. This suggests that rollback should not be viewed only as a substitute for hyperparameter tuning; it may also serve as an additional control layer on top of a good optimization configuration, further improving training quality.
Second, our interpretation of discriminator behavior across different GAN variants is still built on the NTK regime. In particular, the present theory describes how, in the infinite-width and linearized-training limit, the discriminator influences generator residual descent through its input derivative. Compared with the finite-width and often shallow networks used in practice, this analysis remains an approximation. However, the value of the framework is precisely that it does not aim to reproduce every finite-network phenomenon exactly; rather, it provides a mechanistic picture that explains success and failure modes and, more importantly, guides algorithm design. The numerical results suggest that this infinite-width viewpoint remains informative for finite-width training.
Third, the current theory does not explicitly cover common training techniques such as spectral normalization or gradient penalties. This limitation is not specific to our analysis alone, but reflects a broader boundary of the standard NTK framework itself. In particular, classical NTK theory is typically developed under assumptions such as infinite network width, smooth activations, and a near-linearized training regime. By contrast, mechanisms such as spectral normalization actively modify the network parameterization and training geometry, and therefore are not naturally captured by the usual fixed-kernel NTK approximation. For this reason, our present theory does not yet characterize how such techniques alter the discriminator NTK, the induced gradient field, or the resulting residual dynamics in a strict sense. At the same time, this does not prevent us from analyzing the resulting training behavior empirically through the network output and residual-based indicators. One practical strength of the present framework is that these indicators remain informative even when such regularization mechanisms are used in training. A fuller theoretical integration of spectral normalization, gradient penalties, and related techniques would make the framework both more complete and more realistic.
Fourth, the generator-side analysis relies on the assumption that the generator NTK remains approximately constant. For relatively regular problems such as Poisson and Laplace equations, this approximation appears reasonable and is consistent with both existing studies and our numerical observations [5]. However, for more complex strongly nonlinear, multiscale, stiff, or higher-order PDEs, the generator NTK may evolve more substantially during training, and the constant-kernel approximation may become less accurate [21, 22]. In this sense, the present theory is better suited to describing early-stage or weakly nonlinear training dynamics than fully global dynamics in strongly nonlinear regimes. Even so, one of the main messages of this paper is that a linearized infinite-width analysis can still provide practically useful guidance for finite-network training control.
Fifth, under finite-width neural network training, the first-order quantity \(r^\top K_{rr}^{G}\gamma\) should be understood as an aggregated and low-cost first-order indicator rather than as a strict per-step descent certificate. It effectively separates successful regimes from representative failure modes and provides a practical basis for rollback, but it remains a low-dimensional global scalar and cannot fully resolve finer interactions among spectral modes, spatial regions, or multiple loss channels. In difficult PDEs, the dominant error may come from a small subset of slow modes, boundary channels, or localized regions, while a single aggregated score may not fully reflect such structural imbalance. At the same time, this indicator is not spectrally blind. Through the decomposition of \(r^\top K_{rr}^{G}\gamma\), it implicitly constrains how discriminator-induced gradients are distributed across modes. In particular, fast modes associated with larger kernel eigenvalues typically decay first, so their residual amplitudes become small early in training. If one still wishes to maintain \(r^\top K_{rr}^{G}\gamma<0\) afterwards, then the discriminator-induced weighting must become more consistent with the slower unresolved modes that continue to dominate the residual energy. In this sense, the first-order indicator does not explicitly optimize spectral modes one by one, but it implicitly favors a training process that first reduces the fast modes and then increasingly emphasizes the slow ones. It is precisely because this quantity is global, low-cost, and available online that it provides a workable bridge between theory and algorithm.
Sixth, while our experiments already cover a diverse set of PDEs, including Poisson, Laplace, viscous Burgers, Reaction–Diffusion, and Klein–Gordon equations, the overall benchmark range is still limited. We have not yet systematically studied higher-dimensional PDEs, complex geometries, noisy inverse problems, parameter identification tasks, or more extreme stiffness and boundary-condition regimes. Therefore, although the current conclusions already show a certain degree of generality, their broader scope still requires further validation. On the other hand, the existing experiments are sufficient to show that the proposed framework is not tied to a single equation or isolated phenomenon, but has explanatory and algorithmic value across multiple PDE types.
J.3 Future Work:
These limitations also point naturally to several directions for future work.
First, the rollback strategy itself can be made more stable. For example, one may incorporate rollback as an additional control layer on top of an already well-tuned optimization setting, or design smoother rollback criteria with momentum, confidence thresholds, or multi-step prediction mechanisms to reduce oscillations.
Second, it would be valuable to move beyond the constant-NTK or weakly varying NTK approximation and study more general time-varying kernel dynamics, especially for strongly nonlinear, multiscale, and higher-order PDEs. Such an analysis could lead to a deeper understanding of how generator kernels and discriminator feedback co-evolve across residual modes.
Third, common practical techniques such as spectral normalization, gradient penalties, and residual links should be incorporated into a unified theoretical framework. Understanding how these mechanisms reshape the discriminator NTK, its input-gradient field, and the induced preconditioned residual descent may lead not only to a more faithful theory, but also to better training strategies.
Fourth, the broader design space suggested by this work deserves more systematic exploration. In particular, many combinations among residual input versus squared-residual input, GAN/LSGAN/IPM-type objectives, and constrained discriminator function classes remain unexplored. Such combinations may exhibit better stability, spectral balance, or stiffness-handling properties.
Fifth, future work may develop more refined control criteria beyond the scalar quantity \(r^\top K_{rr}^{G}\gamma\), for instance by using spectral information, channel-wise loss decomposition, or block-structured residual diagnostics for interior, boundary, and initial-condition terms. This could yield a more fine-grained and PDE-aware update-control mechanism. Balancing the computational cost of solving the spectrum and the resulting accuracy during training remains an open problem.
Sixth, the present framework could be extended to broader scientific machine learning settings, including soft-constrained PINNs, inverse problems, optimal control, weak-form methods, and function-space generative models. Since the key idea of this paper is that the discriminator modulates generator dynamics by inducing a gradient field on residual space, this mechanism is not restricted to standard PINNs and may be relevant in a wider class of problems.
References
[1] Limitations of physics informed machine learning for nonlinear two-phase transport in porous media Journal of Machine Learning for Modeling and Computing 2020 1 1
[2] Deep hidden physics models: Deep learning of nonlinear partial differential equations Journal of Machine Learning Research 2018 19 25 1–24
[3] Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data Journal of computational physics 2019 394 56–81
[4] Understanding and mitigating gradient flow pathologies in physics-informed neural networks SIAM Journal on Scientific Computing 2021 43 5 A3055–A3081
[5] When and why PINNs fail to train: A neural tangent kernel perspective Journal of Computational Physics 2022 449 110768
[6] Deqgan: Learning the loss function for pinns with generative adversarial networks arXiv preprint arXiv:2209.07081 2022
[7] A physics-informed GAN framework based on model-free data-driven computational mechanics Computer Methods in Applied Mechanics and Engineering 2024 424 116907
[8] Physics-informed generative adversarial networks for stochastic differential equations SIAM Journal on Scientific Computing 2020 42 1 A292–A317
[9] Loss-attentional physics-informed neural networks Journal of Computational Physics 2024 501 112781
[10] Adversarial physics-informed neural networks with hard constraints for optimal control of PDEs Journal of Computational Physics 2025 114307
[11] Generative adversarial networks Communications of the ACM 2020 63 11 139–144
[12] f-gan: Training generative neural samplers using variational divergence minimization Advances in neural information processing systems 2016 29
[13] Least squares generative adversarial networks Proceedings of the IEEE international conference on computer vision 2017 2794–2802
[14] Neural tangent kernel: Convergence and generalization in neural networks Advances in neural information processing systems 2018 31
[15] A neural tangent kernel perspective of GANs International Conference on Machine Learning 2022 6660–6704 PMLR
[16] Wasserstein generative adversarial networks International conference on machine learning 2017 214–223 Pmlr
[17] Artificial neural networks for solving ordinary and partial differential equations IEEE transactions on neural networks 1998 9 5 987–1000
[18] Solving differential equations using neural network solution bundles arXiv preprint arXiv:2006.14372 2020
[19] Self-adaptive physics-informed neural networks Journal of Computational Physics 2023 474 111722
[20] Weak adversarial networks for high-dimensional partial differential equations Journal of Computational Physics 2020 411 109409
[21] Is the neural tangent kernel of PINNs deep learning general partial differential equations always convergent? Physica D: Nonlinear Phenomena 2024 457 133987
[22] The positivity of the neural tangent kernel SIAM Journal on Mathematics of Data Science 2025 7 2 495–515