Obshazard-bench: Benchmarking Multimodal Foundation Models for Real-Time Disaster Intelligence from Raw Earth Observation Streams

Remote sensing (RS) datasets have evolved from early archival time-series analysis [31, 32] to large-scale multimodal benchmarks. Recent works such as XLRS-Bench [11], VRSBench [13], and RSRSD-5M [14] have advanced ultra-high-resolution (UHR) remote sensing perception. However, most of these datasets are still built around static optical imagery. This limits their use in disaster scenarios, where cloud interference is common and where atmospheric structure cannot be captured from surface appearance alone [33, 34].

Several benchmarks have started to include temporal or multi-modal observations. DynamicEarthNet [35] provides daily multi-spectral sequences, and FoMo-Bench [36] introduces multi-modal forest monitoring. Other specialized datasets, such as TreeFinder [37] and SegMunich in SpectralGPT [4], further enrich remote sensing data coverage. Yet these datasets mainly describe surface land-cover or vegetation changes. They do not focus on fast-changing physical processes in the atmosphere, which are critical for many extreme events.

Recent Earth foundation models and benchmarks also point to a broader trend toward cross-sphere and heterogeneous Earth-system reasoning. Examples include AnySat [38], TerraMind [39], OmniEarth-Bench [12], OmniGAIA [40], Earth-Agent [41], and TerraBench [42]. These works expand Earth AI evaluation beyond isolated visual perception, covering broader data sources, cross-domain reasoning, and in some cases tool-augmented agent workflows. However, high-frequency physical observations, such as vertical temperature and humidity sounding signals, are still underrepresented in benchmark design. Existing benchmarks also rarely organize such observations around disaster response stages. Obshazard-bench complements these efforts by focusing on raw sounding streams and lifecycle-oriented disaster reasoning.

Obshazard-bench addresses this gap by using high-frequency sounding streams from AMSU-A, HIRS, and MHS. These instruments provide multi-channel physical observations of the atmosphere, including AMSU-A with 15 channels and HIRS with 20 channels. Compared with optical-centric benchmarks, Obshazard-bench offers a complementary physical view of disasters. It allows models to reason over atmospheric signals that are not visible in traditional surface imagery.

Remote sensing benchmarks for disaster response have moved from basic perception toward more complex reasoning. Early datasets such as xBD [19], RescueNet [43], and CRASAR-U-DRoIDs [23] mainly support building damage assessment from bi-temporal satellite images or unmanned aerial vehicle (UAV) imagery. These datasets are important for post-disaster mapping, but they are less suited to early warning or event evolution analysis.

Later benchmarks extend the task scope and hazard coverage. DisasterM3 [24] and RSCC [44] introduce multi-modal reasoning and change captioning. ZeShot-VQA [25] and DisasterVQA [26] explore disaster-related visual question answering. Other works, including Anomaly-CD [45], Shield [46], BRIGHT [28], EBD [29], and MONITRS [30], further improve multi-hazard coverage and dataset scale.

Despite these advances, most disaster benchmarks still rely on static or bi-temporal imagery. Their tasks are often centered on damage classification, change detection, or basic VQA. They rarely use raw high-frequency physical streams, and they do not fully reflect the operational needs of disaster response. In practice, disaster management requires more than post-event perception. Models must support early warning, track event evolution, and estimate humanitarian or socio-economic impacts.

Meteorological foundation models provide useful evidence that high-frequency physical data can support such reasoning. Models such as Aurora [47], Aardvark Weather [48], and Prithvi WxC [49] show strong potential for weather forecasting and early warning. However, these works mainly focus on model development and prediction tasks. They are not designed as benchmarks for evaluating multimodal disaster intelligence across multiple hazards and lifecycle stages.

This leaves an operational gap between experimental Earth AI systems and professional disaster management [17, 5]. Recent reviews also note that current systems still struggle to assimilate high-frequency physical data for response-oriented reasoning [50, 51, 16]. Obshazard-bench is designed to address this gap. It covers 8 major disaster categories and 28 sub-categories across 62 countries. It aligns daily physical observation sequences with 4,202 lifecycle-oriented VQA samples, providing a realistic testbed for evaluating disaster-oriented MLLMs.

Obshazard-bench is built upon a multi-source architecture that connects verified disaster records with high-frequency geophysical observations. We use the EM-DAT international disaster database ¹¹ 1 https://www.emdat.be/ as the event metadata source, from which historical disaster cases are selected and organized. To focus on rapidly evolving extreme events, we retain disasters with an active duration no longer than 9 days. The resulting benchmark contains 127 verified historical extreme-event cases across 62 countries, covering 8 major disaster categories and 28 disaster sub-categories.

Departing from conventional optical-centric or post-event disaster benchmarks, Obshazard-bench uses raw satellite sounding streams as the core observational input. Specifically, we incorporate a complementary tri-sensor constellation consisting of AMSU-A with 15 channels, HIRS with 20 channels, and MHS with 5 channels. These instruments provide multi-channel atmospheric sounding observations related to temperature, humidity, and other physical conditions, enabling models to reason from raw geophysical signals rather than relying only on surface-level visual appearance. In addition to satellite streams, the benchmark integrates disaster metadata, ground-station observations, historical disaster records, and socio-economic indicators when constructing lifecycle-oriented disaster reasoning tasks.

The processing procedure is designed to preserve the raw-stream nature of the observations while making heterogeneous sources usable for multimodal evaluation. For each selected disaster event, satellite observations are temporally aligned with the event window and spatially associated with the affected region. The benchmark does not replace raw observations with delayed reanalysis products, expert-interpreted maps, or physically inverted variables. Instead, it organizes raw multi-channel sensor observations into event-centric samples, allowing Obshazard-bench to evaluate whether MLLMs can transform raw physical observations into temporally grounded and decision-relevant disaster reasoning.

Obshazard-bench is constructed through a multi-stage pipeline that transforms heterogeneous disaster records and raw satellite observations into reasoning-oriented benchmark samples. The pipeline begins with disaster event metadata. For each event, we extract the disaster identifier, start date, end date, longitude, latitude, and disaster type. Events with durations longer than 9 days are filtered out, ensuring that the benchmark focuses on rapidly evolving disaster processes. To support pre-disaster reasoning, the observation window is extended by shifting the start date 14 days earlier than the recorded event onset.

For each retained event, raw satellite observations are collected within the corresponding temporal window. The current benchmark uses AMSU-A, HIRS, and MHS observations. For each day in the event-centered time span, the pipeline reads observations within a 24-hour window, typically from 00:00 to 23:00 at hourly intervals. These observations are retrieved from the storage system according to the corresponding sensor type, time stamp, and event location.

After temporal retrieval, the observations are spatially cropped around the event center. Given the event longitude and latitude, a local region is extracted using a padding size of 72 pixels, resulting in a $144\times 144$ spatial crop for each channel and each time step. This produces a raw multi-channel observation tensor over the event region.

Because satellite observations within a 24-hour window may contain multiple valid measurements for the same spatial location, we apply a temporal compression step to organize the raw stream into a daily observation layer. For each channel and each pixel, the pipeline scans the hourly observations and retains the latest valid observation within the 24-hour window. This step removes the hourly dimension while preserving the raw sensor values, producing a tensor of shape $(C,H,W)$ for each sensor and each day. Importantly, this operation does not convert the observations into reanalysis variables or expert-derived physical products; it only organizes the raw sensor stream into a consistent event-level format.

Finally, each processed observation is saved together with its event metadata, including the disaster identifier, start date, end date, longitude, and latitude. The resulting event-centric observations are then used to construct lifecycle-oriented VQA samples covering pre-disaster anticipation, active event evolution, and post-disaster impact quantification. In total, Obshazard-bench contains 4,202 benchmark samples derived from 127 historical disaster events.

After building event-centric observation samples, we convert them into visual question answering (VQA) instances. Each instance is defined by three elements: a disaster event, an observation timestep, and a target variable. This makes each sample traceable to its source event, raw observations, and ground-truth record.

For each event, we first build a temporal sequence around the event date. We use $t-14$, $t-7$, and $t-3$ for pre-event observations. These correspond to 14, 7, and 3 days before the recorded event onset. We use $t_{15}$ for event-stage observations. When available, we also use $t_{\max-1}$ and $t_{\max}$ for later event-related observations. Each timestep is paired with raw multi-channel observations from AMSU-A, HIRS, and MHS. Event metadata, such as location, event date, and disaster category, is also attached.

We then generate VQA samples by matching each timestep with eligible target variables. Pre-event timesteps are used for early-warning targets, including disaster occurrence, disaster type, arrival time, and initial duration. Event-stage and later timesteps are used for evolution and impact-related targets, including recovery time, magnitude, total deaths, total affected, number affected, number injured, number homeless, and Consumer Price Index (CPI)-related economic disruption. In this way, one disaster event can produce multiple VQA samples across different timesteps and target variables.

Each VQA instance contains four parts: visual input, question, answer space, and ground-truth answer. The visual input comes from the raw multi-channel observation tensor. The question is generated from task-specific templates and filled with event-related information. The answer space depends on the target type. Categorical targets use discrete labels. Temporal, numerical, and impact-related targets are converted into standardized answer intervals. The ground-truth answer is taken from verified disaster metadata and outcome records.

We also record metadata for each sample, including disaster category, disaster sub-category, lifecycle stage, subtask name, timestep, event identifier, and answer type. These fields are used for evaluation and analysis, not as shortcut inputs to the model. With this event–timestep–target construction, Obshazard-bench contains 4,202 VQA samples from 127 historical disaster events.

Obshazard-bench defines its evaluation space through the Cartesian combination of two dimensions: the disaster taxonomy dimension and the lifecycle reasoning dimension. The disaster taxonomy dimension specifies the physical hazard type, while the lifecycle reasoning dimension specifies the operational stage and reasoning target. This design enables evaluation not only across different disaster categories, but also across different phases of disaster response.

Disaster Taxonomy Dimension. The disaster taxonomy dimension follows a two-level hazard hierarchy. At the first level, Obshazard-bench covers 8 major disaster categories: Flood, Mass movement (wet), Wildfire, Earthquake, Storm, Volcanic activity, Heat-wave, and Cold-wave. At the second level, these categories are divided into 28 disaster sub-categories. Flood includes flash flood, coastal flood, general flood, and riverine flood. Mass movement (wet) includes landslide and mudslide. Wildfire includes land fire, forest fire, and general wildfire. Earthquake includes tsunami and ground movement. Storm includes hail, tornado, blizzard or winter storm, derecho, extra-tropical storm, general storm, lightning or thunderstorm, sand or dust storm, severe weather, storm surge, and tropical cyclone. Volcanic activity includes ash fall, lava flow, general volcanic activity, and pyroclastic flow. Heat-wave and Cold-wave are retained as individual sub-categories.

Lifecycle Reasoning Dimension. The lifecycle reasoning dimension is designed according to the operational workflow of disaster management and is divided into three stages.

Following the evaluation protocol of Earth AI [5], Obshazard-bench adopts a task-aware scoring scheme for heterogeneous disaster reasoning tasks. Each model response is normalized into a comparable answer format before scoring. For categorical tasks, such as risk detection and disaster type classification, predictions are evaluated according to their semantic or lexical consistency with the ground-truth answer. For numerical and interval-based tasks, including arrival time prediction, duration prediction, recovery time prediction, magnitude deduction, humanitarian burden estimation, and socio-economic intensity estimation, the score is computed according to the distance between the predicted value or interval and the ground truth. All scores are normalized to the range of $[0,1]$.

To avoid bias toward sample-rich disaster categories or frequently occurring subtasks, we aggregate performance using an equal-weight protocol across timesteps, subtasks, lifecycle stages, and disaster categories. This metric design enables fair comparison across the three lifecycle stages of Obshazard-bench: Predictive Crisis Anticipation, Active Evolution Reasoning, and Multi-faceted Impact Quantification.

To ensure the reliability of Obshazard-bench, we conduct quality control from both automated data construction and expert manual review.

Automated Data Quality Control: During data generation, we automatically check the consistency between disaster metadata, satellite observations, and constructed benchmark instances. Specifically, the pipeline verifies event time windows, location coordinates, satellite data availability, channel completeness, missing-value patterns, and the correspondence between generated samples and source records. Cases that fail these checks are removed or regenerated to ensure that each sample is grounded in valid event metadata and aligned multi-source observations.

Expert Manual Quality Control: In addition to automated checks, domain experts manually inspect the constructed cases and task annotations. They review whether the selected observations are relevant to the target disaster, whether the generated questions are scientifically meaningful, and whether the answers are consistent with disaster records and available evidence. Ambiguous, weakly supported, or scientifically invalid samples are revised or excluded from the final benchmark.

We evaluate four representative foundation models on Obshazard-bench, including three closed-source frontier models and one open-source large language model. The closed-source models are Claude Opus 4.8 [52], GPT-5.5 [53], and Kimi-k2.6-1T [54]; the open-source model is Qwen3.5-397B-A17B [55]. All models are evaluated under a zero-shot setting with the same input representation and uniform prompts for fair comparison. As described in Section III-E, we report normalized task-aware scores following the Earth AI evaluation protocol [5]. Additional implementation details, prompt templates, and model configurations are provided in the Appendix.

We evaluate representative general-purpose and domain-adapted models on Obshazard-bench, including frontier MLLMs, an open-source large language model, and a fine-tuned disaster-oriented model. All tasks are evaluated under the three lifecycle stages defined in Section III-D: Predictive Crisis Anticipation (PCA), Active Evolution Reasoning (AER), and Multi-faceted Impact Quantification (MIQ). Scores are normalized to $[0,1]$ and aggregated with an equal-weight protocol across timesteps, subtasks, lifecycle stages, and disaster categories, preventing the final score from being dominated by sample-rich tasks or high-frequency disaster types. Following the evaluation practice in Earth AI [5], we use task-aware soft scoring rather than strict binary matching, where numeric answers are graded according to their distance from the ground truth and categorical or textual answers receive partial credit when semantically or lexically close.

Table IV reports the overall performance across the eight disaster categories. Obshazard-bench remains challenging for all evaluated models: even the best-performing model achieves an average score below 0.30. Among the evaluated general-purpose models, GPT-5.5 achieves the highest average score, followed by Kimi-k2.6-1T and Claude Opus 4.8. While GPT-5.5 provides the strongest overall performance, the remaining models exhibit complementary strengths across different disaster categories and lifecycle stages. These results show that current models still struggle to transform raw multi-channel Earth observations into lifecycle-aware disaster intelligence.

To better understand model behavior, we analyze model performance along two benchmark dimensions: lifecycle stage and disaster type. Table V shows that different models exhibit distinct lifecycle-stage capability profiles. Claude Opus 4.8 achieves the highest score on Predictive Crisis Anticipation, indicating strong capability in pre-disaster risk detection and early forecasting. However, its performance drops sharply on Active Evolution Reasoning, suggesting that strong pre-event reasoning does not necessarily translate into reliable reasoning about event evolution or recovery time.

GPT-5.5 shows the strongest performance on Multi-faceted Impact Quantification and achieves the highest overall score among the evaluated models, indicating better capability in connecting event-related evidence with consequence-level estimates. Kimi-k2.6-1T exhibits a capability profile between GPT-5.5 and Claude Opus 4.8: it performs strongly on Predictive Crisis Anticipation and moderately on Multi-faceted Impact Quantification, but remains limited on Active Evolution Reasoning. Claude Opus 4.8 performs competitively on Predictive Crisis Anticipation, while Qwen3.5-397B-A17B remains consistently behind the stronger closed-source models. These results indicate that model capacity and general reasoning capability remain important for raw-observation-based disaster intelligence.

These lifecycle-stage results suggest that disaster intelligence is not a single homogeneous capability. PCA emphasizes early physical signal interpretation, AER requires temporal evolution modeling, and MIQ requires connecting physical observations with humanitarian and socio-economic consequences. A model that performs well in one stage may still fail in another, demonstrating the necessity of evaluating the full disaster lifecycle rather than reporting only a single aggregated score.

The disaster-level results in Table IV further reveal strong model–hazard interactions. GPT-5.5 performs best on earthquake, wildfire, and mass movement events, which may benefit from broader world knowledge and semantic reasoning about disaster mechanisms. Claude Opus 4.8 achieves the best score on storm, while Qwen3.5-397B-A17B shows weaker performance across most disaster categories. These results demonstrate that model capability varies substantially across hazard families, even under the same benchmark format.

These results demonstrate that model capability varies substantially across hazard families. A model that performs well on one disaster type may not generalize to another, even under the same benchmark format. This supports the need for a multi-hazard benchmark such as Obshazard-bench, where evaluation spans hydrological, meteorological, geophysical, and environmental disasters rather than focusing on a single hazard family. Overall, the lifecycle-stage and disaster-type analyses jointly show that current MLLMs exhibit structured capability gaps across both operational stages and physical hazard mechanisms.

We further conduct a temporal-window ablation study to examine how model performance changes when observations are taken at different timesteps. Unlike the previous analysis, which focuses on model capability differences across stages and hazard types, this study investigates whether different lifecycle tasks benefit from observations closer to the disaster event. This is important for raw-stream disaster intelligence, where practical systems must decide how much historical observation to retain, how frequently to run inference, and which time windows are most informative for different operational tasks.

Table VI shows the ablation results for Predictive Crisis Anticipation using observations collected 14, 7, and 3 days before event onset. Claude Opus 4.8 increases from 0.3973 with a 14-day lead time to 0.5731 with a 3-day lead time, suggesting that near-onset observations can contain more informative physical signals for early-warning tasks. However, GPT-5.5 shows a counterintuitive decline from 0.3893 to 0.2908 as the lead time shortens. Kimi-k2.6-1T follows a non-monotonic pattern, dropping at the 7-day lead time and recovering at the 3-day lead time, suggesting that its temporal sensitivity is weaker and less stable than Claude Opus 4.8. This indicates that temporal proximity alone does not guarantee improved reasoning if a model cannot effectively integrate sequentially updated observations. The result highlights the value of evaluating multiple pre-event lead times rather than relying on a single static pre-disaster snapshot.

Table VII further evaluates Active Evolution Reasoning using event-stage observations. GPT-5.5 performs best at both checkpoints and remains stable at the late-stage checkpoint, while Claude Opus 4.8 obtains near-zero performance at the late-stage checkpoint. Kimi-k2.6-1T improves from the event-day checkpoint to the late-stage checkpoint, indicating a mild benefit from additional event-tail observations. This highlights the importance of temporal grounding for reasoning about disaster evolution and recovery.

Table VIII shows a different pattern for Multi-faceted Impact Quantification. Across all models, MIQ performance remains relatively stable across timesteps. Kimi-k2.6-1T follows the same broadly time-invariant pattern, with only small fluctuations across the five checkpoints. This suggests that consequence-level estimation is less sensitive to temporal proximity than early-warning tasks. One possible explanation is that humanitarian burden and socio-economic intensity cannot be inferred from physical observations alone; they also require robust grounding in event metadata, exposure, vulnerability, and historical context. Therefore, additional observations closer to the event do not automatically improve consequence-level estimation.

Overall, the temporal-window ablation reveals that different lifecycle tasks respond differently to observation timing. PCA is generally time-sensitive and often benefits from near-onset signals, MIQ is relatively time-invariant and likely requires additional contextual grounding beyond raw observations, while AER can benefit from late-stage event-tail evidence. These findings demonstrate that Obshazard-bench can evaluate not only whether a model performs well, but also when observations become useful for different types of disaster reasoning.

Beyond model ranking, the experimental results provide several implications for practical disaster-intelligence workflows. First, the temporal trend in PCA suggests the possibility of adaptive observation-window retention. If future models can reliably improve as event onset approaches, operational systems may preserve dense observations only within shorter high-risk rolling windows, reducing storage and inference costs while maintaining early-warning utility. However, this should be balanced against the value of longer lead time, since earlier but less accurate predictions may still be useful for evacuation planning and emergency resource pre-positioning.

Second, the weak temporal sensitivity of MIQ indicates that consequence-level estimation may not require continuous inference at every observation timestep. In practice, MIQ modules could be executed at selected checkpoints, while more system resources are allocated to collecting exposure, vulnerability, infrastructure, and socio-economic context. This suggests that improving impact assessment may depend less on simply increasing raw observation frequency and more on integrating decision-relevant contextual information.

Third, the results show that no single evaluated model performs consistently well across all lifecycle stages and disaster categories. General-purpose models, although strong in some settings, still exhibit clear stage-specific and hazard-specific capability gaps. This suggests that raw-observation-based disaster intelligence may involve multiple technical bottlenecks rather than a single unified reasoning ability. From an operational perspective, future disaster-response systems may benefit from combining complementary models or modules, where different models specialize in early warning, event evolution, or impact quantification.

Finally, given the low absolute scores, current MLLMs should be viewed as decision-support tools rather than autonomous disaster-response agents. For high-stakes tasks such as evacuation planning, humanitarian burden estimation, and recovery scheduling, model outputs should be coupled with uncertainty thresholds, expert review, and escalation mechanisms.

Overall, the experimental results show that current MLLMs still struggle to transform raw multi-channel Earth observations into lifecycle-aware disaster intelligence. The failures are not uniform: they depend on the lifecycle stage, temporal distance to the event, and disaster type. Obshazard-bench thus serves not only as a leaderboard, but also as a diagnostic benchmark for identifying stage-specific, time-dependent, and hazard-dependent limitations in current foundation models.