CIG-RL: Curiosity-Driven Information-Guided Reinforcement Learning for Source Term Estimation in Uncertain Environments

Junhee Lee Affiliation: J. Lee, H. Jang, H. Kim and H. Oh are with the Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea. (Corresponding author: Hyondong Oh) {ljh0124, hlomk, gudwls124z, h.oh}@kaist.ac.kr Seunghwan Kim Affiliation: S. Kim, H. Park and C. Kim are with the Department of Mechanical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan 44919, Republic of Korea. {kevin6960, gudgh1630, pon02124}unist.ac.kr Hongro Jang Affiliation: J. Lee, H. Jang, H. Kim and H. Oh are with the Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea. (Corresponding author: Hyondong Oh) {ljh0124, hlomk, gudwls124z, h.oh}@kaist.ac.kr Hyungjin Kim Affiliation: J. Lee, H. Jang, H. Kim and H. Oh are with the Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea. (Corresponding author: Hyondong Oh) {ljh0124, hlomk, gudwls124z, h.oh}@kaist.ac.kr Hyoungho Park Affiliation: S. Kim, H. Park and C. Kim are with the Department of Mechanical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan 44919, Republic of Korea. {kevin6960, gudgh1630, pon02124}unist.ac.kr Changseung Kim Affiliation: S. Kim, H. Park and C. Kim are with the Department of Mechanical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan 44919, Republic of Korea. {kevin6960, gudgh1630, pon02124}unist.ac.kr Hyondong Oh ^†^†thanks: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Affiliation: J. Lee, H. Jang, H. Kim and H. Oh are with the Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea. (Corresponding author: Hyondong Oh) {ljh0124, hlomk, gudwls124z, h.oh}@kaist.ac.kr

Abstract

Source term estimation (STE), which aims to estimate key properties of the gas source, is essential for identifying hazardous gas releases. Information-theoretic approaches have been adopted for autonomous STE using mobile sensors due to robustness in noisy environments, yet their online action selection incurs substantial computational cost. Deep reinforcement learning (DRL) provides a promising alternative with its fast decision-making capability. In DRL-based STE, the agent selects actions based on belief states of the source term updated from noisy measurement sequences. However, existing methods rely on random exploration or solely on belief uncertainty reduction without an effective exploration strategy in DRL, which can limit policy robustness in noisy environments. To address this, we propose a curiosity-driven information-guided reinforcement learning for robust and efficient STE. The proposed method promotes active exploration of novel belief state transitions that have not been sufficiently explored during training. We further introduce an uncertainty-adaptive active perception reward to guide efficient source search under uncertainty. Simulations under high-noise conditions and real-world experiments demonstrate the robustness and feasibility of the proposed framework, highlighting its potential for practical STE problems.

I INTRODUCTION

Hazardous gas leakage poses serious risks to human health, requiring rapid identification of the gas source properties. Estimating key properties (e.g., source location and release strength) is referred to as a source term estimation (STE) problem. STE problems are challenging because gas leakage is often invisible and strongly affected by turbulent atmospheric conditions. Since it is dangerous for humans to directly search for the gas source, autonomous STE strategies using mobile sensors have attracted considerable attention [1].

One of the representative strategies for STE is the information-theoretic approach, which employs Bayesian inference to estimate the source term and utilizes information theory to select actions that maximize the information gain [2, 3, 4, 5]. Although this approach is effective in addressing noisy measurements, it requires computing the expected entropy reduction for all action candidates, resulting in high computational cost. Consequently, this computational cost limits planning over long horizons.

Deep reinforcement learning (DRL) offers a promising solution for addressing these issues in STE. DRL can learn search policies offline and execute them efficiently during deployment, enabling fast decision-making and non-myopic search behavior [6, 7]. Consequently, many studies have adopted end-to-end DRL approaches that focus on rapid source search [8, 9, 10, 11]. However, these methods do not use explicit source estimators and thus cannot easily determine when the source search is complete or reliably estimate the source location.

Refer to caption — Fig. 1: Overview of the real-world validation: (a) experimental setup with the mobile sensor and gas source; and (b) comparison of real-world search trajectories of Infotaxis, MVG-RDDPG, and CIG-RL.

To address this limitation, recent studies have integrated source estimators into DRL-based STE frameworks [12, 13, 14, 15, 16]. In these frameworks, the source estimator probabilistically estimates the source location, typically using particle filters. Nevertheless, developing a robust search policy under highly noisy conditions remains a significant challenge. In DRL-based STE, the agent makes decisions based on particle filter-based belief states updated from noisy measurement sequences. Under severe noise, these belief states can become biased toward incorrect source hypotheses. Therefore, effective exploration of how the belief state changes with respect to time (i.e., belief state transitions) is crucial during training. However, existing methods rely on random exploration [12, 13, 14], which lacks a mechanism for effectively acquiring informative observations in noisy environments. To mitigate this, recent studies have introduced information gain into policy learning [15, 16]. These methods primarily guide the policy by reducing uncertainty in the current belief. However, because the belief can be distorted by noisy measurements, uncertainty reduction alone without an effective exploration may be insufficient for robust policy learning. Beyond these limitations, real-world validation of DRL-based STE methods with explicit source estimation has rarely been reported, leaving their practical robustness insufficiently verified.

In this paper, we propose a DRL-based framework that enables robust source search in highly noisy environments. The key idea is to improve policy robustness by encouraging exploration based on the novelty of belief transitions, rather than relying on random exploration [12, 13, 14] or solely reducing current belief uncertainty [15, 16]. To this end, we develop an exploration module by leveraging a curiosity network [17]. Our exploration module predicts the next latent belief from the current latent belief, agent position, and action, and uses the discrepancy between the predicted and actual next latent beliefs as an exploration signal (i.e., auxiliary reward). This encourages the agent to explore belief transitions that remain difficult to predict from prior training experience, promoting more robust policy learning. To prevent excessive exploration while reliably guiding the agent toward the source, we further propose an uncertainty-adaptive active perception reward. This reward adaptively balances information gathering (i.e., exploration) and source-directed approach (i.e., exploitation) according to the level of the particle filter variance, enabling efficient search under noisy conditions. To validate the effectiveness and feasibility of the proposed framework, we conduct simulations in highly noisy environments and real-world experiments with $CO_{2}$ leakages. The main contributions are:

1.

A curiosity-driven belief space exploration framework for DRL-based STE, which encourages the agent to actively explore novel belief state transitions for robust source search;
2.

An uncertainty-adaptive active perception reward that integrates mutual information with particle filter variance, enhancing robustness to noisy conditions while ensuring efficient source search; and
3.

Validation in challenging high-noise simulations and real-world experiments with $CO_{2}$ leakage scenarios, demonstrating feasibility and practical applicability.

II Related Work

II-A Information-Theoretic Approaches

Information-theoretic approaches have gained attention for their effectiveness in handling noisy gas measurements. Infotaxis [2], cognitive search [3], and dual-mode planner [5] determine actions that maximize the reduction in the entropy of the source term, whereas Entrotaxis [4] selects actions that maximize the entropy of the future measurement distribution over all action candidates at each step. However, the computational cost of uncertainty calculation for action candidates can limit real-time implementation.

II-B End-to-End DRL

Recent studies have adopted DRL due to its real-time applicability and ability to learn efficient source search policies. In end-to-end DRL approaches, most studies employ memory networks to improve source search performance. Hu et al. [8] propose a long short-term memory (LSTM)-based plume-tracing framework, while Li et al. [9] introduce proximal policy optimization based on a gated recurrent unit (GRU-PPO) approach to effectively leverage historical measurements. Singh et al. [10] further develop a recurrent neural network agent that robustly tracks turbulent gas plumes under varying wind conditions. Beyond memory-based approaches, He et al. [11] introduce a dueling deep Q-network (Dueling DQN) method using computational fluid dynamics (CFD) simulations to reflect more realistic environments. However, because these approaches do not include an explicit source estimator, they cannot reliably determine when the search is complete or accurately identify the source location.

II-C DRL with Explicit Source Estimators

Unlike end-to-end DRL approaches, several studies incorporate explicit source estimators into DRL frameworks to enable more reliable source search. PC-DQN [12] employs density-based spatial clustering of applications with noise (DBSCAN) to extract belief features from the particle filter, while MVG-RDDPG [13] improves belief representation using Gaussian mixture model (GMM)-based feature extraction and leverages historical measurements through a GRU. AGDC [14] further introduces an autonomous goal detection mechanism that adopts the convergence of the particle filter to effectively determine search termination. Although these methods improve belief representation or search termination, their random exploration is sample-inefficient and lacks guidance on how to explore informative states in the belief space under uncertainty. Furthermore, their methods are trained under low environmental noise conditions, which can limit robustness in real-world scenarios.

AID-RL [15] and GMM IG-RL [16] attempt to handle this issue by utilizing information-theoretic principles in DRL. AID-RL [15] combines greedy exploitation toward high-concentration regions with information-directed exploration. However, its concentration-based exploitation can make the policy vulnerable to spurious high-concentration measurements in highly noisy environments. More recently, GMM IG-RL [16] improves search robustness by designing GMM information gain rewards that reduce current belief uncertainty. Nevertheless, since belief can be distorted by noisy measurements, relying primarily on uncertainty reduction without an effective exploration may limit policy robustness.

III Problem Statement

It is assumed that a hazardous gas source located at $\mathbf{p}_{s}=\left[x_{s},y_{s}\right]^{T}$ emits gas particles with a release strength $q_{s}$ . The source term vector is parameterized as $\theta_{s}=\left[\mathbf{p}_{s}^{T},{q_{s}}\right]^{T}$ , and this study focuses on localizing $\mathbf{p}_{s}$ . At each time step, the agent equipped with a gas sensor acquires gas measurements at its position $\mathbf{p}_{t}=[x_{t},y_{t}]^{T}$ and updates its estimate of the source term distribution using a particle filter, based on the predefined gas dispersion and sensor models. Leveraging the estimated source term, CIG-RL is applied to find the best source search policy under uncertainty.

III-A Gas Dispersion Model

For the gas dispersion model, we use the isotropic plume model [2]. In this model, gas particles diffuse with a particle life time $\tau$ and diffusivity $D$ , and are advected by an average wind speed $V$ with direction $\chi$ . The mean gas concentration $C(\mathbf{p}_{t}|\theta_{s})$ acquired at the sensing position $\mathbf{p}_{t}$ is defined as:

	$\displaystyle C(\mathbf{p}_{t}\|\theta_{s})$	$\displaystyle=\frac{q_{s}}{4\pi D\,\|\mathbf{p}_{t}-\mathbf{p}_{s}\|}\exp\frac{{V}(x_{t}-x_{s})\sin\chi}{2D}$
		$\displaystyle\quad\cdot\exp\frac{-{V}(y_{t}-y_{s})\cos\chi}{2D}\cdot\exp\frac{-\|\mathbf{p}_{t}-\mathbf{p}_{s}\|}{\lambda},$		(1)

where $\lambda=\sqrt{D\tau/\left(1+\frac{V^{2}\tau}{4D}\right)}$ .

III-B Sensor Model

In real-world scenarios, sensor measurements are affected by wind turbulence and sensor noise. To capture these effects, we adopt a Gaussian noise model as the sensor model [5]. The sensor measurement $z_{t}$ obtained by the agent at the sensing position $\mathbf{p}_{t}$ is expressed as:

z_{t}=C(\mathbf{p}_{t}|\theta_{s})+\nu_{{env}}+\nu_{{sensor}},

(2)

where $\nu_{{env}}$ and $\nu_{{sensor}}$ denote the noise arising from the wind turbulence and the sensor measuring process, respectively. Both terms are assumed to follow Gaussian distributions, i.e., $\nu_{env}\sim\mathcal{N}(0,\sigma_{env}^{2})$ and $\nu_{sensor}\sim\mathcal{N}(0,\sigma_{sensor}^{2})$ , where $\sigma_{env}$ represents the instability of the wind conditions. Here, $\sigma_{sensor}=\beta\cdot C(\mathbf{p}_{t}|\theta_{s})$ , where $\beta$ denotes the sensor noise level. The likelihood of the sensor measurement $z_{t}$ for a given source term $\theta_{s}$ is defined as:

p(z_{t}|\theta_{s})=\frac{1}{\sigma_{T}\sqrt{2\pi}}\exp-\frac{(z_{t}-C(\mathbf{p}_{t}|\theta_{s}))}{2\sigma_{T}^{2}},

(3)

where $\sigma_{T}=\sqrt{\sigma_{\textit{}{env}}^{2}+\sigma_{\textit{}{sensor}}^{2}}$ denotes the overall standard deviation of the total noise.

III-C Particle Filter

We utilize a particle filter to estimate the source term, as it is well-suited for handling the nonlinearity of the source term and remains robust under high levels of sensor measurement noise [18]. The source term probability distribution can be expressed by $N_{p}$ particles as:

p(\theta_{t}|z_{1:t})=\sum_{i=1}^{N_{p}}w_{t}^{i}\delta(\theta_{t}-\theta_{t}^{i}),

(4)

where $w_{t}^{i}$ indicates the normalized weight of each particle, $\delta(\cdot)$ is the Dirac delta function, and $\theta_{t}^{i}$ denotes each sampled particle. Upon receiving a new sensor measurement, the unnormalized particle weight can be sequentially updated as:

\widetilde{w}_{t+1}^{i}=p(z_{t+1}|\theta_{t}^{i})\cdot w_{t}^{i}.

(5)

The measurement likelihood $p(z_{t+1}|\theta_{t}^{i})$ is computed using the defined gas dispersion and sensor models given in (1) and (3), respectively. Then, the normalized particle weight $w_{t+1}^{i}$ is calculated as:

w_{t+1}^{i}=\frac{\widetilde{w}_{t+1}^{i}}{\sum_{i=1}^{N_{p}}\widetilde{w}_{t+1}^{i}}.

(6)

Since the particle filter can suffer from the degeneracy, in which most particle weights collapse toward zero, resampling is applied. After resampling, the Markov chain Monte Carlo (MCMC) method [19] is employed to improve particle impoverishment.

IV Curiosity-Driven Information-Guided Reinforcement Learning

In this section, we introduce a curiosity-driven information-guided reinforcement learning framework to achieve robust source search in noisy environments. State and action are first introduced, and the curiosity-driven belief space exploration for STE is explained. Then, the uncertainty-adaptive active perception reward is presented, and the overall learning framework is outlined.

IV-A State and Action

IV-A1 State

To enhance the efficiency of source estimation, Gaussian mixture model (GMM) clustering [20] is applied to extract features from the particle filter. The belief state is approximated using a GMM, characterized by the mean of Gaussian distribution $\bm{M}_{t}=[\mu_{t}^{1},\mu_{t}^{2},\ldots,\mu_{t}^{L}]$ , its covariance $\bm{\Sigma}_{t}=[\Sigma_{t}^{1},\Sigma_{t}^{2},\ldots,\Sigma_{t}^{L}]$ , and the mixing weights $\bm{\Pi}_{t}=[\pi_{t}^{1},\pi_{t}^{2},\ldots,\pi_{t}^{L}]$ , where $L$ denotes the number of GMM clusters. Here, $L$ is set to 3. To avoid GMM label-switching ambiguity, the GMM components are consistently ordered. The GMM probability of each particle is calculated as:

p(\theta_{t}^{i}|\bm{M}_{t},\bm{\Sigma}_{t},\bm{\Pi}_{t})=\sum_{l=1}^{L}\pi_{t}^{l}\mathcal{N}(\theta_{t}^{i}|\mu_{t}^{l},\Sigma_{t}^{l}),

(7)

where $\mathcal{N}$ denotes the probability density function of the $l$ -th GMM component. The state is then defined as:

{s}_{t}=\left[b_{t},\mathbf{p}_{t}\right],

(8)

where $b_{t}=[\bm{M}_{t},\bm{\Sigma}_{t,\mathit{diag}},\bm{\Pi}_{t},m_{t}]$ denotes the belief state, $\bm{\Sigma}_{t,\mathit{diag}}=[\mathit{diag}(\Sigma_{t}^{1}),\mathit{diag}(\Sigma_{t}^{2}),\ldots,\mathit{diag}(\Sigma_{t}^{L})]$ contains the diagonal terms of the GMM component covariance matrices, and $m_{t}$ represents the mean of all particles.

IV-A2 Action

The action space is parameterized by the heading direction $a_{t}$ , while the movement distance is fixed at $\Delta d$ for each step. The next sensing position is given by:

\mathbf{p}_{t+1}=\mathbf{p}_{t}+\begin{bmatrix}\cos(a_{t})\\ \sin(a_{t})\end{bmatrix}\cdot\Delta d,

(9)

IV-B Curiosity-Driven Belief Space Exploration for STE

As discussed earlier, policy learning in STE depends on observing informative measurement sequences under uncertainty, making an effective exploration strategy essential. Particularly, since noisy measurements can mislead the belief, the agent must sufficiently and efficiently experience belief state transitions during training. To this end, we enhance exploration by introducing a curiosity-based learning signal, inspired by [17]. Our approach formulates curiosity in the belief space using particle filter-based belief features, where a high prediction error serves as an indicator of potentially novel or insufficiently learned belief transitions.

As shown in Fig. 2, we design a curiosity network as an exploration module composed of a feature extractor, a forward network, and an inverse network, parameterized by $\xi_{\mathit{feat}},\xi_{\mathit{fwd}},$ and $\xi_{\mathit{inv}}$ , respectively. The feature extractor encodes the particle filter-based belief state into a latent belief representation. Given the consecutive belief states $b_{t}$ and $b_{t+1}$ , the latent belief representations are obtained as $\varphi(b_{t})$ and $\varphi(b_{t+1})$ , respectively. The forward network predicts the next latent belief representation $\hat{\varphi}(b_{t+1})$ from $\varphi(b_{t})$ , $\mathbf{p}_{t}$ , and $a_{t}$ . The prediction error between $\varphi(b_{t+1})$ and $\hat{\varphi}(b_{t+1})$ is used as the forward network loss:

\displaystyle\mathcal{L}_{\mathit{fwd}}

\displaystyle=\left\lVert\varphi(b_{t+1})-\hat{\varphi}(b_{t+1})\right\rVert_{2}^{2}.

(10)

To ensure that the latent representation retains meaningful action-relevant belief in the forward network prediction, we also employ an inverse network. The inverse network predicts the action $\hat{a}_{t}$ that caused the transition between consecutive belief states given $\varphi(b_{t})$ and $\varphi(b_{t+1})$ , and is trained with the following loss function:

\displaystyle\mathcal{L}_{\mathit{inv}}

\displaystyle=\mathrm{MSE}(a_{t},\hat{a}_{t}).

(11)

The curiosity network is trained by minimizing a weighted sum of the forward and inverse network losses, given by:

\displaystyle\min_{\xi_{\mathit{feat}},\xi_{\mathit{fwd}},\xi_{\mathit{inv}}}\mathcal{L}_{\mathit{c}}=(1-\varepsilon)\,\mathcal{L}_{\mathit{fwd}}+\varepsilon\,\mathcal{L}_{\mathit{inv}},

(12)

where $\varepsilon$ balances the forward and inverse loss terms.

For STE problems, note that the belief state is extracted by GMM clustering, as in (8). As illustrated in Fig. 3, point $A$ represents the latent belief representation of the next belief state, while point $B$ denotes the latent belief representation predicted by the forward network. In the early stage of training, a noticeable gap exists between point $A$ and $B$ , representing the prediction error measured as $|\overline{AB}|=\left\|\varphi(b_{t+1})-\hat{\varphi}(b_{t+1})\right\|_{2}^{2}$ . A larger $\left|\overline{AB}\right|$ indicates an insufficiently learned belief transition and can be used as an exploration signal to encourage exploration of such transitions. Thus, a large prediction error can encourage the policy to explore belief transitions that remain difficult for the forward network to predict. Importantly, since $\left|\overline{AB}\right|$ corresponds to the loss of the forward network, it tends to decrease as the forward network is trained. Consequently, the agent can be guided to learn in a way that gradually reduces this prediction error. Based on this insight, we adopt the forward loss, representing the prediction error, as the auxiliary reward for DRL to encourage the agent to efficiently explore novel belief transitions, as:

\displaystyle r_{t}^{a}=\mathcal{L}_{\mathit{fwd}}.

(13)

IV-C Uncertainty-Adaptive Active Perception Reward

Although encouraging exploration in DRL to seek novel belief transitions is critical, relying heavily on exploration is insufficient to achieve an optimal policy and may delay the search process. Therefore, an additional mechanism is required to reliably and efficiently guide the agent to the source. To this end, we propose an uncertainty-adaptive active perception reward. This reward augments mutual information with a distance-based source approach term, and adaptively balances the two through the particle filter variance to enable robust search under uncertainty while facilitating rapid approach to the source.

Utilizing mutual information based on Shannon’s entropy reduction can provide an effective way to enhance robustness under noisy conditions [16]. The mutual information is employed as a reward to drive the policy toward reducing the uncertainty of the estimated source term, defined as:

	$\displaystyle I\!\left(\mathbf{p}_{t+1}\right)$	$\displaystyle=-\sum_{i=1}^{N_{p}}w_{t}^{i}\log w_{t}^{i}$
		$\displaystyle\quad+\sum_{\hat{z}_{t+1}=0}^{z_{\mathit{max}}}p(\hat{z}_{t+1}\|\theta_{t})\left(\sum_{i=1}^{N_{p}}\hat{w}_{t+1}^{i}\log\hat{w}_{t+1}^{i}\right),$		(14)

where $\hat{z}_{t+1}$ denotes all possible future measurements, $z_{\mathit{max}}$ is the maximum value of the discretized future measurements, and $p(\hat{z}_{t+1}|\theta_{t})$ represents the likelihood of the future measurements, computed using (3). Here, the potential particle weight $\hat{w}_{t+1}^{i}$ is updated based on (5) and (6). Furthermore, to accelerate the source search, we introduce a distance term between the agent’s position and the estimated source location, defined as:

d_{t}=\left|\mathbf{p}_{t}-\hat{\mathbf{p}}_{t,s}\right|,

(15)

where $\hat{\mathbf{p}}_{t,s}$ denotes the current estimated source location.

Unlike [16], to enable an efficient balance between the mutual information term and the distance term, an automatic adjustment method based on the particle filter variance is proposed. The particle filter variance can serve as the indicator of the reliability in the source estimation. The variance of all particles at time step $t$ is calculated as:

\text{Cov}(\theta_{t})=\sum_{i=1}^{N_{p}}w_{t}^{i}\left(\theta_{t}^{i}-m_{t}\right)\left(\theta_{t}^{i}-m_{t}\right)^{T}.

(16)

Then, the active perception reward function is defined as:

r_{t,\mathit{step}}=I(\mathbf{p}_{t+1})-\left(\sqrt{\mathit{tr}\!\left(\operatorname{Cov}(\theta_{t})\right)}\right)^{-1}d_{t}.

(17)

Here, to address the unit inconsistency between the source location and release strength in $\theta_{t}$ , each dimension is normalized prior to variance computation. For numerical stability, $\mathit{tr}\!\left(\mathrm{Cov}({\theta}_{t})\right)$ is clipped with a lower bound of $(0.1)^{2}$ .

In this method, as a high particle filter variance indicates that the belief over the estimated source term is uncertain, the mutual information term has a greater relative influence on the reward to promote uncertainty reduction in such cases. As more measurements are collected and the variance decreases, the reliability in source estimation increases. Consequently, the distance term can contribute more strongly to the reward, guiding the agent toward the estimated source location.

Finally, the total extrinsic reward function is defined as:

r_{t}^{e}=\left\{\begin{array}[]{ll}+10,&\text{successfully find the source,}\\[6.0pt] r_{t,\mathit{step}},&\text{for each step,}\\[6.0pt] -5,&\text{outside search boundary.}\end{array}\right.

(18)

IV-D Learning Framework for CIG-RL

To train a robust source search policy under uncertainty, we propose curiosity-driven information-guided reinforcement learning (CIG-RL), a learning framework using the soft actor-critic (SAC) [21] while formulating auxiliary learning to enhance exploration in STE.

We design the replay buffer $\mathit{D}$ that stores tuples of agent-environment interaction in the form $(s_{t},a_{t},r_{t}^{e},s_{t+1})$ . This design is to facilitate diverse experiences and efficient sample reuse by randomly sampling mini-batches. As depicted in Fig. 2, the extrinsic active perception reward $r_{t}^{e}$ , which reflects the uncertainty reduction of the estimated source term, is stored and sampled from the replay buffer. In contrast, the auxiliary reward $r_{t}^{a}$ , capturing novelty in the belief state transition is updated by the curiosity network rather than replayed from past transitions. This decoupled sampling strategy prevents auxiliary reward values that were high in previously explored uncertain regions from being repeatedly reused, thereby mitigating bias toward excessive exploration during later stages of training. The extrinsic active perception and auxiliary rewards are then combined to form the total reward, which is used to update the critic network consisting of the Q-network and the target Q-network, as:

\displaystyle r_{t}^{\textit{total}}=r_{t}^{e}+\eta r_{t}^{a},

(19)

where $\eta$ is the weight parameter between the extrinsic and auxiliary rewards. Based on the combined extrinsic and auxiliary rewards, the target Q-value is computed as:

	$\displaystyle y_{t}$	$\displaystyle=r_{t}^{\textit{total}}$		(20)
		$\displaystyle+\gamma\,\mathbb{E}_{a_{t+1}\sim\pi_{\psi}}\left[Q_{\bar{\phi}}(s_{t+1},a_{t+1})-\alpha\log\pi_{\psi}(a_{t+1}\|s_{t+1})\right],$		(20)

where $\bar{\phi}$ denotes the parameter of the target Q-network, $\psi$ represents the parameter of the actor network, and $\alpha$ is the temperature parameter which balances the entropy term against returns. The critic network is then trained by minimizing the following loss function:

\displaystyle\mathcal{L}_{\textit{critic}}(\phi)=\mathbb{E}_{(s_{t},a_{t})\sim\mathit{D}}\left[\left(Q_{\phi}(s_{t},a_{t})-y_{t}\right)^{2}\right].

(21)

To optimize the policy, the actor network is trained by minimizing the following:

\displaystyle\mathcal{L}_{\textit{actor}}(\psi)=\mathbb{E}_{s_{t}\sim\mathit{D},\,a_{t}\sim\pi_{\psi}}\left[\alpha\log\pi_{\psi}(a_{t}|s_{t})-Q_{\phi}(s_{t},a_{t})\right].

(22)

Finally, we adopt automatic entropy tuning to effectively adjust the temperature parameter $\alpha$ . The objective function for tuning $\alpha$ can be written as:

\displaystyle\mathcal{L}_{\alpha}=\mathbb{E}_{a_{t}\sim\pi_{\psi}}\left[-\alpha\log\pi_{\psi}(a_{t}|s_{t})+\widetilde{\mathcal{H}}\right],

(23)

where $\widetilde{\mathcal{H}}$ is the target entropy which determines the desired level of policy stochasticity.

TABLE I: Parameter values of environments 1 and 2

Symbol	Env 1	Env 2	Unit
$q_{s}$	$U(500,3000)$	$U(500,3000)$	mg/s
$\tau$	$U(100,1500)$	$U(100,1500)$	s
$D$	$U(1,15)$	$U(1,15)$	m²/s
$V$	$U(0,5)$	$U(0,5)$	m/s
$\chi$	$U(0,360)$	$U(0,360)$	deg
$\sigma_{env}$	0.4	0.5	mg/s
$\beta$	0.25	0.4	-

TABLE II: Parameter lists and corresponding values

Parameter	Value
First fully connected layer size	256
Second fully connected layer size	64
Learning rate (actor, critic)	0.0003
Learning rate (curiosity network)	0.00005
Replay buffer size	1,000,000
Mini-batch size	256
Auxiliary reward weight ( $\eta$ )	5
Curiosity loss weighting factor $(\varepsilon)$	0.8
Discount factor $(\gamma)$	0.99

V Numerical Simulation

V-A Simulation Setup

The numerical simulations are conducted in two distinct environments, where gas properties and wind conditions are systematically varied, as summarized in Table I. Both environments 1 and 2 are configured under high-noise conditions, while environment 2 incorporates more challenging levels of environmental and sensor noise. Particularly, the environmental noise levels are set to 0.4 and 0.5, which are substantially higher than those used in prior studies (e.g., 0.2 in MVG-RDDPG [13] and AGDC [14]), resulting in more severe fluctuations in gas measurements. The effect of environmental noise is illustrated in Fig. 4, which shows gas measurement maps under different noise levels. During training, the agent moves to the next sensing position with a fixed step size of $2\,\text{m}$ within a $60\,\text{m}\times 60\,\text{m}$ search area. A simulation terminates if the agent exceeds 300 steps or if the standard deviation of the particle filter with 2,000 particles falls below 0.1. At the end of each simulation, the source search is considered successful if the distance between the estimated and the true source is within $3\,\text{m}$ . All algorithms are trained for 80,000 episodes, and the training hyperparameters of the proposed method are listed in Table II. At the beginning of each episode, both the source and agent locations are randomly initialized, and the environmental parameters listed in Table I are randomly sampled from corresponding uniform distributions.

V-B Evaluation Metrics

The trained policies are evaluated using two key metrics: success rate (SR) and mean travel distance (MTD). SR denotes the percentage of episodes in which the agent successfully estimates the source location. MTD is defined as the average travel distance computed over all episodes with successful source estimation. For each evaluation, 1,000 random scenarios that are not seen during training are executed. In addition, a stricter success criterion than in training is applied by requiring the distance between the true and estimated source locations to be within $1\,\mathrm{m}$ .

TABLE III: Performance comparison with existing methods

Method	Env 1		Env 2
Method	SR (%)	MTD (m)	SR (%)	MTD (m)
Infotaxis [2]	87.6	155.6	83.0	169.0
Entrotaxis [4]	85.4	142.3	82.7	149.2
PC-DQN [12]	82.3	100.5	78.5	112.4
MVG-RDDPG [13]	90.6	91.8	84.0	98.4
AGDC [14]	86.4	98.4	82.8	104.3
AID-RL [15]	73.9	120.4	63.4	138.2
GMM IG-RL [16]	91.6	95.8	86.0	100.4
CIG-RL (Ours)	98.3	80.2	95.0	86.2

V-C Performance Comparison with Existing Methods

We compare our method with both information-theoretic and DRL-based approaches. The information-theoretic baselines include Infotaxis [2] and Entrotaxis [4]. The DRL-based baselines consist of PC-DQN [12], MVG-RDDPG [13], AGDC [14], AID-RL [15], and GMM IG-RL [16]. To ensure a fair comparison, all methods use the same gas dispersion and sensor models with 2,000 particles in the particle filter. In addition, all DRL-based methods are trained under identical noise settings listed in Table I.

As shown in Table III, CIG-RL demonstrates the most robust performance in both environments, achieving SR of 98.3% and 95.0%, respectively. The training curves in Fig. 5 show that CIG-RL achieves the lowest SR variability and the highest SR among DRL baselines, indicating that CIG-RL learns a robust search policy under noisy conditions. Sample trajectories of CIG-RL in environment 2 are presented in Fig. 6. Larger red circles denote higher gas concentrations.

V-D Ablation Study

V-D1 Ablation Analysis of CIG-RL

An ablation analysis is conducted in both environments to evaluate the contributions of the proposed components in CIG-RL. As shown in Table IV, removing both the proposed exploration strategy and the active perception reward significantly degrades performance. This result suggests that insufficient belief space exploration and less informative reward guidance can hinder robust policy learning under highly noisy measurements. When curiosity and the active perception reward are removed individually, the performance improves compared with removing both components, but SR remains lower than that of CIG-RL. These results demonstrate that coupling belief space exploration with active perception reward guidance is beneficial for robust STE in highly noisy environments.

In addition, the results in Table IV show that MTD increases when the particle filter variance is not incorporated in the active perception reward. This highlights that the uncertainty-adaptive adjustment between the mutual information term and the distance term plays a crucial role in improving search efficiency. Furthermore, CIG-RL, which uses the decoupled sampling strategy, outperforms the variant that samples the auxiliary reward from the replay buffer. Since reusing samples with previously high curiosity rewards in later training stages can encourage excessive exploration, this can lead to inefficient source search behavior.

V-D2 Sensitivity Analysis of CIG-RL

Next, we analyze the sensitivity to the weighting factor $\eta$ between the extrinsic and auxiliary rewards. As outlined in Table V, when $\eta$ is too small, SR degrades since the curiosity becomes insufficient to promote exploration of novel belief transitions. In contrast, when $\eta$ becomes larger than 5, the auxiliary reward leads to excessive exploration in the early stage of training, consequently increasing the MTD.

V-D3 Analysis of Curiosity in CIG-RL

Fig. 7 shows the curiosity auxiliary reward, defined as the forward network loss, during training under two high-noise settings. The auxiliary reward is initially high but consistently decreases as the agent explores belief transitions and the forward network is trained, converging to a low value in both environments. This demonstrates stable convergence of the proposed belief-based curiosity despite noisy measurements.

TABLE IV: Ablation study of CIG-RL in two environments

Method	Env 1		Env 2
Method	SR (%)	MTD (m)	SR (%)	MTD (m)
w/o curiosity & active reward	86.0	96.1	82.5	103.2
w/o curiosity	91.8	88.5	88.3	97.5
w/o active reward	93.0	84.3	89.7	92.6
w/o particle filter variance	96.1	88.8	94.3	93.4
w/o decoupled sampling	95.4	90.4	92.8	96.2
CIG-RL (Ours)	98.3	80.2	95.0	86.2

TABLE V: Performance evaluation of auxiliary reward sensitivity

Method	Env 1		Env 2
Method	SR (%)	MTD (m)	SR (%)	MTD (m)
$\eta=0.5$	95.0	83.0	92.1	88.8
$\eta=1$	95.6	81.6	93.0	86.4
$\eta=2.5$	97.1	78.4	93.3	85.6
$\eta=5$	98.3	80.2	95.0	86.2
$\eta=10$	97.6	85.8	94.6	91.7

VI Real-World Experiments

As illustrated in Fig. 1, a $CO_{2}$ tank is used to generate gas dispersion for the real-world experiments. An AgileX Limo equipped with a SEN15112 gas sensor serves as the mobile sensing platform, with onboard computation performed by an NVIDIA Jetson Xavier NX. A Livox Mid-360 LiDAR provides environmental perception, while FAST-LIO2 [22] is used for mobile sensor localization. Each algorithm is tested 10 times in an $8\,\text{m}\times 8\,\text{m}$ gym, where the mobile sensor starts at $x=0\,\text{m}$ and $y=0\,\text{m}$ and moves $0.5\,\text{m}$ per step. Each trial terminates when the particle filter variance falls below 0.1, and success of source search is declared if the distance between the estimated and true source is within $0.5\,\text{m}$ . In environment A, shown in Fig. 8(a), the source is located at $x=4\,\text{m}$ and $y=8\,\text{m}$ and released toward the negative $y$ -direction. In environment B, shown in Fig. 8(b), it is at $x=8\,\text{m}$ and $y=4\,\text{m}$ and released along a $45^{\circ}$ upward-left direction, resulting in a more challenging scenario where the plume is advected far from the mobile sensor, leading to sparse measurements. To ensure steady gas dispersion, $CO_{2}$ is released for 4 minutes before each trial. Measurements are collected 5 seconds after the mobile sensor reaches each sensing position considering the sensor response time, and the gym windows remain open to introduce sufficient external airflow.

TABLE VI: Results of real-world experiments

Method	Env A			Env B
Method	SR (%)	SN	CT (ms)	SR (%)	SN	CT (ms)
Infotaxis	70	49.3	420.3	60	61.3	425.4
MVG-RDDPG	80	34.8	80.4	60	48.6	79.8
CIG-RL (Ours)	100	26.4	82.6	90	30.8	83.8

In the real-world experiments, we compare Infotaxis, MVG-RDDPG, and CIG-RL, while additionally evaluating the average computation time (CT) for one-step decision making. As shown in Table VI, Infotaxis exhibits the highest CT in both environments because it calculates the expected entropy reduction for four action candidates by considering possible future measurements. In contrast, MVG-RDDPG and CIG-RL show substantially lower CT by learning policies through DRL. Although CIG-RL shows slightly higher CT than MVG-RDDPG, it shows more robust performance in terms of SR and step number (SN). Fig. 1 shows that CIG-RL achieves a more efficient search path toward the source than Infotaxis and MVG-RDDPG. Particularly, CIG-RL approaches the source without the oscillatory movements observed in Infotaxis and MVG-RDDPG, resulting in fewer steps to particle filter convergence. A video of the real-world experiments is available at [URL will be added upon acceptance].

VII Conclusions and future work

In this study, we propose a curiosity-driven information-guided reinforcement learning framework for robust source search in noisy environments. The proposed framework couples curiosity-driven belief space exploration with an uncertainty-adaptive active perception mechanism, enabling the agent to efficiently obtain informative measurements while maintaining robust policy updates under noisy conditions. Simulations under highly noisy environmental conditions and real-world experiments show that CIG-RL outperforms existing approaches in terms of success rate and search efficiency, while retaining practical computational cost for real-time deployment. For future work, we plan to extend the framework to multi-agent systems to further improve search efficiency and scalability in more complex environments.

References

[1] M. Hutchinson, H. Oh, and W. Chen (2017) A review of source term estimation methods for atmospheric dispersion events using static or mobile sensors. Inf. Fusion 91, pp. 83–100. Cited by: §I.
[2] M. Vergassola, E. Villermaux, and B. I. Shraiman (2007) ‘Infotaxis’ as a strategy for searching without gradients. Nature 445 (7126), pp. 406–409. Cited by: §I, §II-A, §III-A, §V-C.
[3] B. Ristic, A. Skvortsov, and A. Gunatilaka (2016) A study of cognitive strategies for an autonomous search. Inf. Fusion 28, pp. 1–9. Cited by: §I, §II-A.
[4] M. Hutchinson, H. Oh, and W. Chen (2018) Entrotaxis as a strategy for autonomous search and source reconstruction in turbulent conditions. Inf. Fusion 42, pp. 179–189. Cited by: §I, §II-A, §V-C.
[5] S. Kim, J. Seo, H. Jang, C. Kim, M. Kim, J. Pyo, and H. Oh (2024) Gas source localization in unknown indoor environments using dual-mode information-theoretic search. IEEE Robot. Autom. Lett. 10 (1), pp. 588–595. Cited by: §I, §II-A, §III-B.
[6] J. Kober, J. A. Bagnell, and J. Peters (2013) Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32 (11), pp. 1238–1274. Cited by: §I.
[7] Q. Yang, D. Li, J. Huang, P. Zhou, and S. Zhang (2025) Close-range air combat autonomous maneuver decision-making of uav based on td3-transformer. Int. J. Aeronaut. Space Sci. 27 (3), pp. 2378–2394. Cited by: §I.
[8] H. Hu, S. Song, and C. L. P. Chen (2019) Plume tracing via model-free reinforcement learning method. IEEE Trans. Neural Netw. Learn. Syst. 30 (8), pp. 2515–2527. Cited by: §I, §II-B.
[9] H. Li, J. Yuan, and H. Yuan (2024) An active olfaction approach using deep reinforcement learning for indoor attenuation odor source localization. IEEE Sens. J. 24 (9), pp. 14561–14572. Cited by: §I, §II-B.
[10] S. H. Singh, F. van Breugel, R. P. Rao, and B. W. Brunton (2023) Emergent behaviour and neural dynamics in artificial agents tracking odour plumes. Nat. Mach. Intell. 5 (1), pp. 58–70. Cited by: §I, §II-B.
[11] Y. He, L. Cheng, Y. Pan, D. Wang, Y. Li, and H. Zheng (2024) Gas source localization using dueling deep Q-network with an olfactory quadruped robot. Int. J. Adv. Robot. Syst. 21 (3), pp. 17298806241255797. Cited by: §I, §II-B.
[12] Y. Zhao, B. Chen, X. Wang, Z. Zhu, Y. Wang, G. Cheng, R. Wang, R. Wang, M. He, and Y. Liu (2022) A deep reinforcement learning based searching method for source localization. Inf. Sci. 588, pp. 67–81. Cited by: §I, §I, §II-C, §V-C.
[13] M. Park, P. Ladosz, and H. Oh (2022) Source term estimation using deep reinforcement learning with Gaussian mixture model feature extraction for mobile sensors. IEEE Robot. Autom. Lett. 7 (3), pp. 8323–8330. Cited by: §I, §I, §II-C, §V-A, §V-C.
[14] Y. Shi, M. Wen, Q. Zhang, W. Zhang, C. Liu, and W. Liu (2025) Autonomous goal detection and cessation in reinforcement learning: a case study on source term estimation. In Proc. AAAI Conf. Artif. Intell., Vol. 39, pp. 738–745. Cited by: §I, §I, §II-C, §V-A, §V-C.
[15] Z. Li, W. Chen, J. Yang, and Y. Yan (2023) AID-RL: Active information-directed reinforcement learning for autonomous source seeking and estimation. Neurocomputing 544, pp. 126281. Cited by: §I, §I, §II-C, §V-C.
[16] J. Lee, H. Jang, M. Park, and H. Oh (2025) Enhanced reward function design for source term estimation based on deep reinforcement learning. IEEE Access 13, pp. 87777–87792. Cited by: §I, §I, §II-C, §IV-C, §IV-C, §V-C.
[17] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell (2017) Curiosity-driven exploration by self-supervised prediction. In Proc. Int. Conf. Mach. Learn. (ICML), pp. 2778–2787. Cited by: §I, §IV-B.
[18] M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp (2002) A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Trans. Signal Process. 50 (2), pp. 174–188. Cited by: §III-C.
[19] B. Ristic, S. Arulampalam, and N. Gordon (2003) Beyond the Kalman filter: particle filters for tracking applications. Artech House. Cited by: §III-C.
[20] C. A. Bouman, M. Shapiro, G. Cook, C. B. Atkins, and H. Cheng (1997) Cluster: an unsupervised algorithm for modeling gaussian mixtures. Cited by: §IV-A1.
[21] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. Int. Conf. Mach. Learn. (ICML), pp. 1861–1870. Cited by: §IV-D.
[22] W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang (2022) FAST-LIO2: Fast direct lidar-inertial odometry. IEEE Trans. Robot. 38 (4), pp. 2053–2073. Cited by: §VI.

	$\displaystyle C(\mathbf{p}_{t}\|\theta_{s})$	$\displaystyle=\frac{q_{s}}{4\pi D\,\|\mathbf{p}_{t}-\mathbf{p}_{s}\|}\exp\frac{{V}(x_{t}-x_{s})\sin\chi}{2D}$
		$\displaystyle\quad\cdot\exp\frac{-{V}(y_{t}-y_{s})\cos\chi}{2D}\cdot\exp\frac{-\|\mathbf{p}_{t}-\mathbf{p}_{s}\|}{\lambda},$		(1)