A large-scale ego-centric CS dataset with synchronized video, replay telemetry, action chains, segments, and captions.
Quick Navigation: Overview · Dataset Composition · Data Construction
EgoCS-400K aligns first-person gameplay video with demo-derived tick-level telemetry, keyboard/mouse inputs, atomic actions, protected action chains, DP-based temporal segments, and multi-grained video-language captions.
🎥 Player-view video + 🕹️ Per-tick game state + ⚔️ Action/segment annotations + 📝 Multi-grained captions
📢 Release status. EgoCS-400K is publicly released. The dataset, viewer, annotations, code, technical report, and project page will continue to be expanded and updated.
| ⏱️ 10,000+ hrs gameplay video |
🔁 40,000+ rounds round-level trajectories |
🗺️ 13 maps CSGO / CS2 scenes |
👥 10 players per-round perspectives |
🧩 6 annotations from ticks to captions |
- Release the EgoCS-400K dataset on Hugging Face.
- Release the interactive dataset viewer on Hugging Face Spaces.
- Release the technical report.
- Release the processing and annotation code.
Beyond raw gameplay video: EgoCS connects every visible moment to the player's controls, game state, actions, and temporally coherent behaviors.
|
First-person video aligned with per-tick camera, spatial, weapon, player, round, and event states. |
Fine-grained keyboard and mouse inputs, including movement, fire, reload, inspect, and scope. |
One hierarchy spanning per-tick states, atomic actions, protected chains, segments, and full sequences. |
|
Dynamic programming creates model-ready clips without breaking meaningful action chains. |
Visual, spatial, and action signals remove invalid renders and low-information segments. |
Structured captioning context plus a viewer for inspecting videos, timelines, actions, and segments. |
EgoCS is designed for models that need more than a caption over a clip. Each sample can be studied at multiple temporal and semantic levels:
| 🧩 Level | 🎮 What it represents | 🎯 Typical supervision |
|---|---|---|
| Player-view sequence | A full first-person round/player trajectory | Long video understanding, player behavior modeling |
| Coherent segment | A model-ready clip with stable temporal boundaries | Segment captioning, retrieval, video-language alignment |
| Protected action chain | A group of actions that should not be cut apart | Action-aware segmentation, long action modeling |
| Atomic replay action | Fire, inspect, reload, switch, jump, grenade, scope, etc. | Fine-grained action recognition/localization |
| Per-tick state trace | 32 Hz movement, camera, input, weapon, and player state | Grounded control/action analysis and evaluation |
The videos below present two EgoCS-400K samples with synchronized first-person video, keyboard/mouse states, replay-grounded actions, environment attributes, and VLM-generated prompts.
How EgoCS-400K is built from public replay files into synchronized videos, structured annotations, filtered segments, and multi-grained captions — the full step-by-step construction pipeline is documented in a dedicated guide.
Demo Collection · Video Rendering · Rendered Video Filter · Per-tick Annotation · DP Segmentation · Spatial / Action Filter · VLM Captioning
.
|-- README.md
|-- data/
| |-- derived/parsed_rounds/ # Lightweight CSV/JSON examples
| `-- viewer_assets/round_player_viewer/ # Viewer HTML and annotations
|-- docs/
| |-- data-construction.md # Data construction & pipeline guide
| `-- images/ # README figures
| |-- egocs-teaser.png
| `-- pipeline-overview.png
|-- pipeline/
| |-- parse_dem/ # Demo parsing
| |-- overlay/ # Video overlay rendering
| |-- action_timeline/ # Timeline and protected chains
| |-- viewer_build/ # Viewer asset generation
|-- requirements.txt
`-- .gitignore # Excludes demos, videos, caches
- Python 3.10 or newer
ffmpegandffprobeonPATHfor video processing- An API key only if VLM caption generation is used
python -m venv .venv
# Linux/macOS
source .venv/bin/activate
# Windows PowerShell
# .venv\Scripts\Activate.ps1
pip install -r requirements.txtInstall pyarrow as well when creating Parquet metadata:
pip install pyarrowpython pipeline/parse_dem/cs2_round_player_parser.py \
data/raw/demos/9ine-vs-sparta-m1-dust2.dem \
--out-dir data/derived/parsed_rounds/9ine-vs-sparta-m1-dust2 \
--write-eventsUse --steamid or --sequence to limit the export. Both options can be
repeated.
Build the two samples included in this release:
python pipeline/viewer_build/build_round_player_viewer.py \
--game-root data/derived/videos_overlay/9ine-vs-sparta-m1-dust2 \
--all-sequences-root data/derived/parsed_rounds/9ine-vs-sparta-m1-dust2 \
--original-video-root data/raw/videos_original \
--output-root data/viewer_assets/round_player_viewer \
--round 8 \
--round 10Overlay video is optional:
python pipeline/viewer_build/build_round_player_viewer.py \
--all-sequences-root data/derived/parsed_rounds/9ine-vs-sparta-m1-dust2 \
--original-video-root data/raw/videos_original \
--output-root data/viewer_assets/round_player_viewer \
--skip-overlay \
--round 8 \
--round 10Use --overlay-optional when only some samples have overlay videos.
If you use this dataset or pipeline, please cite:
@misc{guo2026egocs400k,
title={EgoCS-400K: An Egocentric Gameplay Dataset for World Models},
author={Guo, Rongjin and Liang, Dong and Liu, Yuhao and Liu, Fang and Huang, Tianyu and Hancke, Gerhard P. and Lau, Rynson W. H.},
year={2026},
note={Project page: https://EgoCS-400K.github.io}
}