2025
Reinforcement Learning Market-Making Agent
RL-based market-making agent trained in a custom Level-2 order-book simulation, benchmarked against a classical quoting baseline.
Problem
Classical market-making strategies like Avellaneda-Stoikov rely on fixed stochastic assumptions that don't adapt to changing order-book conditions, leaving PnL and inventory risk on the table.
Architecture
Custom OpenAI Gym environment simulating Level-2 order-book dynamics, dynamic quoting, inventory controls, and transaction-cost modeling. PPO and DQN agents trained against the environment with reward shaping tied to inventory risk and realized PnL.
Technologies
- Python
- OpenAI Gym
- Stable-Baselines3
- PPO
- DQN
- NumPy
- Pandas
Engineering decisions
- Modeled transaction costs and inventory penalties directly in the reward function to keep the agent from overtrading.
- Compared PPO and DQN under identical environment seeds to isolate algorithm choice from environment variance.
- Benchmarked against the Avellaneda-Stoikov closed-form strategy as a classical baseline.
Outcomes
- Achieved a Sharpe ratio of 3.2 and ~35% higher PnL than the Avellaneda-Stoikov baseline in simulation.
- Held inventory within ±10% of target even under high-volatility scenarios, indicating strong generalization.