An Informed RL Agent for Portfolio Optimization
Western AI โ 2025-2026 Project
Dataset: https://drive.google.com/drive/folders/1DzsK6fLDA-q-fbjGWCoMtdj4BDn_JrkO?usp=sharing
Western AI โ 2025-2026 Research Project
This project develops a state-of-the-art AI trading agent that makes portfolio allocation decisions using:
| Module | Technology | Purpose |
|---|---|---|
| ๐ฎ DeepAR | Probabilistic LSTM | Forecasts returns with uncertainty estimates |
| ๐ FRED API | Federal Reserve Data | Tracks macro-economic regimes (VIX, yield curve, Fed rate) |
| ๐ฐ FinBERT | Transformer NLP | Extracts sentiment from financial news (planned) |
| ๐ง PPO Agent | Reinforcement Learning | Makes portfolio allocation decisions |
The agent observes a 64-dimensional "Super-State" combining forecasts, macro data, and sentiment to make regime-aware investment decisions.
- Probabilistic Forecasting: Not just "price will be $150" but "90% chance between $145-$155"
- Regime Awareness: Agent adapts strategy based on economic conditions (bull/bear/crisis)
- Uncertainty-Aware: Takes smaller positions when forecasts are uncertain
- ReST Training: Novel "Grow/Improve" methodology adapted from language modeling
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ PPO AGENT (Brain) โ
โ Outputs: Portfolio Weights โ
โ [AAPL: 0.3, MSFT: 0.5, CASH: 0.2] โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โฒ
โโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโ
โ โ โ
โโโโโโโโโดโโโโโโโโ โโโโโโโดโโโโโโ โโโโโโโโโดโโโโโโโโ
โ DeepAR โ โ FinBERT โ โ FRED API โ
โ (Forecaster) โ โ (NLP) โ โ (Macro Data) โ
โโโโโโโโโฌโโโโโโโโ โโโโโโโฌโโโโโโ โโโโโโโโโฌโโโโโโโโ
โ โ โ
โโโโโโโโโดโโโโโโโโ โโโโโโโดโโโโโโ โโโโโโโโโดโโโโโโโโ
โ Price History โ โ News โ โ VIX, Yields, โ
โ OHLCV Data โ โ Articles โ โ Fed Rates โ
โโโโโโโโโโโโโโโ๏ฟฝ๏ฟฝ๏ฟฝโ โโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโ
| Component | Status | Description |
|---|---|---|
| DeepAR Model | โ Complete | Trained on 9 securities, 60-day context |
| FRED Data | โ Complete | VIX, Yield Curve, Fed Funds Rate |
| SuperStateBuilder | โ Complete | 64-dim observation vector |
| PortfolioEnv | โ Complete | Gymnasium-compliant trading environment |
| PPO Training | ๐ In Progress | ReST training methodology |
| FinBERT Sentiment | ๐ Planned | NLP module |
| Dashboard | ๐ Planned | React/Streamlit visualization |
- Python 3.10+
- UV (recommended) or pip
# Clone the repository
git clone https://github.com/Western-Artificial-Intelligence/rl-portfolio-optimization.git
cd rl-portfolio-optimization
# Create virtual environment with UV
uv venv
.venv\Scripts\activate # Windows
source .venv/bin/activate # macOS/Linux
# Install dependencies
uv sync# Train the forecasting model
uv run python deepAR/train_deepar.py --epochs 30# Test SuperStateBuilder
python -m ppo.super_state
# Test PortfolioEnv
python -c "
import pandas as pd
from backtesting.core.PortfolioEnv import PortfolioEnv
df = pd.read_csv('data/deepar_dataset.csv')
env = PortfolioEnv(df=df, use_super_state=True)
obs, info = env.reset()
print(f'Observation shape: {obs.shape}') # (64,)
print('โ Environment ready!')
"Portfolio-Optimizer/
โโโ ๐ data/ # Market data
โ โโโ FRED/ # Macro-economic data
โ โ โโโ VIXCLS.csv # VIX volatility index
โ โ โโโ T10Y2Y.csv # Yield curve spread
โ โ โโโ FEDFUNDS.csv # Federal funds rate
โ โโโ deepar_dataset.csv # Processed training data
โ โโโ *.csv # Price data files
โ
โโโ ๐ deepAR/ # Forecasting module
โ โโโ model.py # DeepARModel + DeepARForecaster
โ โโโ train_deepar.py # Training pipeline
โ โโโ preprocessing.py # Data utilities
โ
โโโ ๐ ppo/ # RL Agent module
โ โโโ __init__.py
โ โโโ super_state.py # SuperStateBuilder class
โ
โโโ ๐ backtesting/ # Trading environment
โ โโโ core/
โ โโโ PortfolioEnv.py # Gymnasium environment
โ
โโโ ๐ checkpoints/ # Saved models
โ โโโ deepar/
โ โโโ deepar_best.pt # Best validation model
โ โโโ training_summary.json
โ
โโโ ๐ tests/ # Unit tests
โ โโโ test_super_state.py
โ โโโ test_portfolio_env.py
โ
โโโ ๐ docs/ # Documentation
โโโ ARCHITECTURE.md
The agent observes a 64-dimensional vector at each step:
| Index | Features | Count | Source |
|---|---|---|---|
| 0-53 | Per-stock forecasts (mean, std, skew, confidence, q10, q90) | 54 | DeepAR |
| 54-59 | Macro indicators (VIX, yield curve, fed rate) | 6 | FRED |
| 60-63 | Sentiment placeholders | 4 | FinBERT (TBD) |
All values are normalized to [-1, 1] range for stable training.
The DeepAR model is trained on 9 securities:
| Symbol | Name | Type |
|---|---|---|
| AAPL | Apple Inc. | Stock |
| AMZN | Amazon.com Inc. | Stock |
| META | Meta Platforms Inc. | Stock |
| MSFT | Microsoft Corp. | Stock |
| NVDA | NVIDIA Corp. | Stock |
| TSLA | Tesla Inc. | Stock |
| NDX | NASDAQ-100 Index | Index |
| SPX | S&P 500 Index | Index |
| PSQ | ProShares Short QQQ | Inverse ETF |
| Category | Technologies |
|---|---|
| ML/RL | PyTorch, Stable-Baselines3, Gymnasium |
| Data | Pandas, NumPy, SciPy |
| Finance | Bloomberg API, FRED API |
| NLP | FinBERT, Transformers |
| DevOps | UV, pytest, Git |
- DeepAR: Salinas et al., "DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks"
- PPO: Schulman et al., "Proximal Policy Optimization Algorithms"
- ReST: Gulcehre et al., "Reinforced Self-Training (ReST) for Language Modeling"
- FinBERT: Araci, "FinBERT: Financial Sentiment Analysis with Pre-trained Language Models"
Western AI Research Group โ 2025-2026
This project is licensed under the MIT License - see the LICENSE file for details.