Scene2Sound: Auditory-Grounded Soundscape Generation
for 3D Gaussian Worlds

Our task requires generating the audio content that composes a soundscape. Recent generative audio models synthesize plausible waveforms from text or visual inputs at open-domain scale [20, 19]. Efficiency-oriented text-to-audio models such as AudioLCM [21], FlashAudio [22], and TangoFlux [23] reduce sampling cost, while Lumina-Next [24] is a general flow-based multimodal generation framework with an audio variant; video-to-audio models [25, 26, 27] such as MMAudio [28] strengthen temporal and semantic alignment with visual content. These approaches generate audio tied to the conditioning observation (a prompt, image, or video clip) rather than a reusable scene-level representation that supports re-rendering under viewpoint changes.

Two lines of work are closer to our setting. Spatial audio generation methods directly produce immersive formats such as first-order ambisonics (FOA) [29, 11] or 5.1-channel surround [10], but their outputs are anchored to the conditioning camera pose or equirectangular frame, giving viewpoint-locked waveforms. SonoWorld [14] reconstructs a 3D audio-visual scene from a single image and synthesizes ambisonic audio for it, sharing our goal of spatially grounded sound; it regenerates the scene from one panoramic view, whereas we ground sound in a given 3DGS world and associate detections across views into object-based, re-renderable 3D instances, and we compare against a re-implementation of its pipeline in Sec. IV. Soundscape-oriented generation instead produces coherent environmental audio from object- or layer-level components, including object-aware generation conditioned on visual cues [9].

While these directions provide strong generators or composition paradigms, to our knowledge none associates multiple sound-emitting instances with persistent 3D anchors for consistent rendering under arbitrary listener motion; our pipeline supplies this authoring step.

Audio rendering in 3D scenes involves two questions: what sound is produced where, and how it propagates to the listener. Our work addresses the former, identifying and anchoring sound-emitting instances, and adopts object-based rendering for real-time playback.

Propagation- and field-based approaches model how sound travels through a scene, via impulse responses [30, 31] or acoustic fields [32, 33]. These methods capture room acoustics accurately but assume known sources or require multiple reference acoustic measurements of the target scene, a fundamentally different problem setting from ours.

Novel-view acoustic synthesis instead synthesizes audio from new listener viewpoints given recorded reference observations [34, 35], including a material- and geometry-aware Gaussian representation [36] and a joint radiance–acoustic field informed by 3D scene structure [37]. These methods reproduce view-dependent audio well but rely on captured audio or per-scene acoustic supervision (e.g., room impulse responses) and neural waveform synthesis, which is inapplicable when audio must be generated from scratch and played back efficiently under interactive viewpoint changes, as with generated or freshly reconstructed 3DGS worlds.

Object-based audio rendering instead represents a scene as a set of audio objects with associated metadata (source signals, 3D positions, rendering parameters), enabling real-time playback in audio engines [38, 39]. Our approach follows this paradigm, generating audio assets and estimating instance-level 3D anchors for a reusable scene-level representation. Interactive authoring systems such as Sonora [13] share this paradigm but rely on user-in-the-loop specification rather than fully automatic, scene-grounded anchoring.

Our framework requires separating sound-emitting object instances and estimating their 3D positions from a pre-trained 3DGS scene. Most scene-segmentation methods [40, 41, 15] obtain instance-level structure by augmenting Gaussians with learnable parameters optimized from multi-view 2D supervision, using per-Gaussian embeddings, identity encodings, or object-centric formulations, and thus require additional training beyond the original 3DGS reconstruction; object-aware Gaussian representations have also been explored for robotic manipulation [42], under a different supervision setting.

Since our scenario assumes a reconstructed 3DGS and favors lightweight deployment, we avoid further training. Training-free directions on 3DGS exist [43, 44] but are not tailored to audio-conditioned instance anchoring or to separating multiple instances under the same sound query. Lifting by Gaussians (LBG) [45] instead lifts 2D foundation-model signals into 3D, fusing masks and learned features to obtain instance-level predictions. We avoid feature learning or fusion and instead perform training-free instance association by measuring Gaussian set overlap across views with the Jaccard index, directly grouping observations into sound-source instances atop the original 3DGS representation. DCSEG [46] uses Jaccard-based matching to assign 2D open-vocabulary semantic labels to class-agnostic 3D masks; we instead use Gaussian-set overlap as the primary cue for cross-view instance association under a fixed sound query.

Given a 3DGS scene $\mathcal{G}=\{g_{1},\ldots,g_{|\mathcal{G}|}\}$ of $|\mathcal{G}|$ Gaussian primitives, where each $g_{i}$ is parameterized by its center $\boldsymbol{\mu}_{i}\in\mathbb{R}^{3}$, opacity, and appearance attributes, we generate a soundscape

where $\mathcal{E}=\{e_{j}\}$ is a set of positioned audio events, each $e_{j}=(a_{j},\mathbf{p}_{j})$ pairing an audio signal $a_{j}$ with a 3D position $\mathbf{p}_{j}\in\mathbb{R}^{3}$, and $a_{\mathrm{amb}}$ is a non-directional ambient sound. Unlike channel- or ambisonics-based outputs, which are tied to the viewpoint at which they were produced, this object-based representation can be re-rendered for any listener pose via standard audio engines, making it suitable for free navigation.

For generated 3DGS scenes, unlike reconstructed scenes, no reference camera positions exist. We automatically select $K$ camera positions $\{\mathbf{v}_{1},\ldots,\mathbf{v}_{K}\}$ that jointly maximize visual quality (for robust VLM predictions) and scene coverage (for comprehensive audio event detection), using only static Gaussian parameters before rendering.

In typical navigable 3DGS scenes, Gaussians form surfaces surrounding an open interior space through which users move; we assume this hollow structure and design viewpoint selection accordingly. We compute a robust, outlier-filtered scene center $\mathbf{c}$, cast $N$ uniformly distributed rays from $\mathbf{c}$, and place one candidate camera per ray toward the scene shell. For each candidate $\mathbf{v}_{i}$, we identify contributing Gaussians $\mathcal{G}^{\mathrm{vis}}_{i}\subseteq\mathcal{G}$ via differentiable-rendering visibility and evaluate visual quality $q_{i}$ with CLIP-IQA [47] (implementation details in the supplementary material). Let $\mathcal{K}\subseteq\{1,\ldots,N\}$ index the selected viewpoints; we greedily select $K$ viewpoints maximizing quality-weighted Gaussian coverage:

where $d_{ij}=\|\mathbf{v}_{i}-\boldsymbol{\mu}_{j}\|$ is the distance from camera $i$ to Gaussian $j$, and $w_{d}(\cdot)$ is a distance weight decaying with distance (inverse-distance weighting; supplementary material). We use $N{=}20$ and $K{=}5$. At each selected viewpoint, we render equirectangular panoramic images for downstream scene understanding.

Once viewpoints are selected, we render panoramic images from the 3DGS scene and analyze them with a Vision-Language Model (VLM), Qwen2.5-VL-32B [17], to infer sound events, identifying both what sounds should be present and where they originate for subsequent audio generation and spatial grounding.

We design a structured prompt instructing the VLM to distinguish two event types. Point sources are spatially localized sounds with identifiable visual sources (e.g., fountain, car engine) that require precise 3D placement. Ambient beds are global soundscape layers without specific spatial locations (e.g., wind, distant traffic) that contribute to overall atmosphere but do not require spatial grounding.

For each point source, the VLM assigns a source_id shared by observations of the same semantic sound event across viewpoints; distinct physical instances within an event are resolved later by Gaussian set matching (Sec. III-E). It also generates three fields: (1) a grounding query as a simple noun phrase for the segmentation model, (2) an audio prompt providing a detailed description for text-to-audio synthesis, and (3) an estimated_loudness (quiet, moderate, or loud) for audio rendering. For ambient beds, we specify only the audio prompt. The VLM jointly analyzes all selected viewpoints and outputs a unified scene-level event collection; the prompt schema and design rules are in the supplementary material.

Given the VLM’s audio prompts, we synthesize waveforms using Stable Audio Open [19], a latent diffusion-based text-to-audio model trained on large-scale audio data, generating 30-second, $44.1$ kHz waveforms for all sources; point-source audio is converted to mono for 3D spatialization, while ambient-bed audio remains stereo for an immersive atmosphere.

We next lift 2D visual observations into consistent 3D sound source positions that remain stable across viewpoints.

Instances sharing the same source_id reuse the audio signal $a_{j}$ generated in Section III-D, completing the positioned audio events $\mathcal{E}=\{e_{j}=(a_{j},\mathbf{p}_{j})\}$ defined in the problem formulation.

Generated audio signals and their 3D positions pass to an audio engine for real-time playback; as users navigate, the engine adjusts volume and spatialization from listener position and orientation. Base volume follows the VLM’s estimated_loudness; instances sharing a source label are energy-normalized by $1/\sqrt{n}$, with $n$ the number of instances sharing the label, so that repeated objects contribute the label’s energy once rather than $n$ times, and the ambient bed plays at a fixed gain below the loudest sources. We implement a lightweight object-based renderer following the standard web-audio spatialization node (PannerNode). Quantitative evaluations use power-law distance attenuation with azimuth-based stereo panning; the interactive demo and subjective-study stimuli play positioned sources through the browser-native PannerNode with head-related transfer function (HRTF) filtering for headphone playback. The representation is compatible with any audio engine supporting 3D spatialization.

Table I presents quantitative comparisons on SoundscapePLY. Bold marks the best among spatial audio methods, our primary comparison; shading ranks all methods, including the non-spatial quality references. Non-spatial methods are included as quality references, since they generate scene-level audio without object-based representation; the primary comparison is among spatial audio methods, and spatial consistency is evaluated in Sec. V-B.

Figure 3 illustrates instance association for a fan across four viewpoints: per-view masks are lifted to Gaussian sets and merged into a single instance whose re-projected 3D position falls within each original mask.

Figure 4 shows instance association across four diverse scenes spanning the types SoundscapePLY covers. The bottom-right scene illustrates a failure case, where the segmentation model fails to ground any sound-emitting object and produces no point-source instances; the VLM still identifies ambient sound events, and Scene2Sound thus falls back to an ambient bed providing a plausible background atmosphere. Video results of generated soundscapes, with navigable binaural audio, are available on the project page.

We ablate key components of Scene2Sound: Scene-level replaces per-source generation with one scene-wide clip from a VLM-produced prompt (cf. the SAO baseline in Table I); Flat mix mixes all source audio into flat stereo without spatial rendering; Single viewpoint uses one viewpoint ($K{=}1$) instead of multiple; Random cameras samples $K{=}5$ cameras uniformly within the scene bounding box instead of automatic viewpoint selection; w/o instance association removes cross-view Gaussian set matching, treating each observation as a separate instance; and w/o ambient fold disables ambient bed separation, treating all sounds as point sources. We report both spatial axes (LMC and CGC with its precision; Sec. V-B) for each variant, along with the mean number of detected source labels ($N_{\mathrm{src}}$) and spatial instances ($N_{\mathrm{inst}}$) per scene; neither is a ground-truth quantity, since $N_{\mathrm{src}}$ counts VLM-identified sources and $N_{\mathrm{inst}}$ counts instances remaining after Gaussian set matching.

Table V shows the results: Scene-level and Flat mix both reduce to fixed-waveform playback ($\mathrm{LMC}{=}0$, $R{=}0$), confirming that object-level decomposition and 3D spatial rendering are both essential for navigation-consistent soundscapes. A single viewpoint detects fewer source labels ($N_{\mathrm{src}}{=}3.0$ vs. $3.4$) and fewer instances ($N_{\mathrm{inst}}{=}13.3$ vs. $21.0$), since sound events visible only from certain perspectives are missed; without informed camera placement (Random cameras), held-out grounding collapses to CGC $0.060$ on its 5-scene subset, though this ablates the entire camera-generation strategy; a milder raycast-random contrast is in the supplementary material. Removing instance association leaves CGC F1 essentially unchanged ($0.2593$ vs. $0.2589$ before rounding) but triples the instances ($66.9$ vs. $21.0$) and shifts the operating point from precision toward recall ($0.302$/$0.316$ vs. our $0.441$/$0.229$); the precision difference is significant (paired 95% CI $[0.05,0.22]$), isolating the contribution of Gaussian set matching at a fixed inventory. Descriptively, placed sources rest on $2.72$ supporting views on average, and $65.6\%$ are observed from multiple views; most placed sources are thus corroborated by multiple views. Without ambient fold, $N_{\mathrm{src}}$ rises to $5.2$ and the listener axis weakens ($+0.253\to+0.217$ at the matched pre-normalization rendering; grounding nearly unchanged); the ambient/local distinction therefore matters for rendering rather than placement.

Quantitative evaluations use fixed-viewpoint settings for fair comparison with baselines lacking viewpoint-dependent rendering; to assess perceptual benefits under viewpoint changes, we ran a subjective evaluation with moving viewpoints (protocol and configuration note in the supplementary material): 19 self-reported normal-hearing participants rated the proposed method against three spatial audio baselines (See2Sound, OmniAudio, ViSAGe) on pre-rendered video clips with identical visual content, each evaluating four of 24 scenes over headphones on a 1–5 MOS scale for Preference (overall quality), Spatial Congruency (sound location vs. visual cues), and Semantic Congruency (sound content vs. visual events). All four conditions produce spatialized binaural audio (cf. the binaural statistics in Table I); the method is therefore not trivially identifiable by spatial cues alone. A residual confound remains, since the conditions differ in how they track the moving viewpoint (ours adapts in full 6DoF via persistent 3D sources, OmniAudio follows head yaw only, ViSAGe re-infers audio from the recorded video), and a fully matched-cue study equalizing this is left as future work.

Figure 5 summarizes the results: Scene2Sound consistently outperforms all baselines on all three criteria (Preference $3.77$, Spatial Congruency $3.57$, Semantic Congruency $3.75$ vs. best-baseline MOS of $2.68$, $2.48$, $2.75$; means over all ratings, per-scene statistics in the supplementary material). All nine pairwise comparisons (three baselines $\times$ three criteria) are significant under paired Wilcoxon signed-rank tests on participant–scene pairs with Bonferroni correction ($p<0.001$; effect sizes $r{=}0.55$–$0.76$). The largest margins appear in Spatial Congruency, consistent with Scene2Sound’s object-based binaural audio, which adapts dynamically as the listener moves, whereas the baselines lack explicit listener-position-dependent cues.