S&P Index Rebalancing Analysis Tool

Overview

This project enables research and analysis of S&P index rebalancing events (particularly index additions) to study price effects and develop trading strategies. The tool follows a structured approach to:

1. Clean and process S&P index change announcements data
2. Fetch and prepare historical market data
3. Conduct event studies around index addition events
4. Backtest trading strategies based on these events

Project Structure

Index_Rebalancing/
├── data/                      # Data storage
│   ├── processed/             # Processed data files
│   │   └── market_data_cache/ # Cache for market data
│   └── sp_global_detailed_index_changes_clean.csv  # Raw S&P index changes data
├── scripts/                   # Python scripts for main workflow
│   ├── 01_data_acquisition_and_preparation.py      # Data processing script
│   ├── 02_event_study_analysis.py                  # Event study script
│   ├── 03_trading_strategy_backtesting.py          # Backtesting script
│   └── prepare_data.py        # Data preparation utility script
├── src/                       # Source code modules
│   ├── backtesting/           # Backtesting engine and utilities
│   ├── data/                  # Data processing modules
│   ├── event_study/           # Event study implementation
│   └── visualization/         # Visualization utilities
├── tests/                     # Test modules
├── results/                   # Generated results and visualizations
├── requirements.txt           # Project dependencies
└── README.md                  # Project documentation

Development Workflow

This project follows a modular development approach, where core functionality is in Python modules (src/) and main workflows are implemented as executable scripts (scripts/).

Utility Modules Structure

The project uses utility modules for each major component:

• src/data/clean_sp_data.py - Cleans S&P Global index change announcement data
• src/data/index_changes.py - Processes cleaned data into a master event list
• src/data/market_data.py - Fetches and processes market data using yfinance
• src/event_study/event_study.py - Core event study functionality
• src/event_study/event_study_utils.py - Supporting utilities for event studies
• src/backtesting/backtester.py - Main backtesting engine implementation
• src/backtesting/backtest_utils.py - Supporting utilities for backtesting
• src/visualization/visualization_utils.py - Visualization functions

Each module's functionality is exposed through its package's __init__.py file, allowing easy importing.

Main Workflow Scripts

The main analysis workflow is implemented in three Python scripts:

1. scripts/01_data_acquisition_and_preparation.py - Processes raw data into usable format
2. scripts/02_event_study_analysis.py - Analyzes abnormal returns around index changes
3. scripts/03_trading_strategy_backtesting.py - Tests trading strategies based on the events

These scripts provide a step-by-step workflow for the complete analysis pipeline.

Installation and Setup

1. Clone the repository
2. Create a virtual environment: python -m venv venv
3. Activate the environment:
   • Windows: venv\Scripts\activate
   • macOS/Linux: source venv/bin/activate
4. Install dependencies: pip install -r requirements.txt

Usage

Running the Analysis Pipeline

The core analysis is performed by running the scripts in the scripts/ directory in numerical order. Each script depends on the output of the previous one.

1. python scripts/01_data_acquisition_and_preparation.py

   • Purpose: Cleans the raw S&P index change data (data/sp_global_detailed_index_changes_clean.csv), identifies relevant addition events, fetches historical price/volume data for the involved tickers and benchmark indices, and saves the processed data.
   • Key Outputs:
     • data/processed/master_event_list.csv
     • data/processed/market_data.pkl
   • Note: This script can take some time to run, especially on the first execution when it needs to download market data. Subsequent runs will be faster due to caching.

2. python scripts/02_event_study_analysis.py

   • Purpose: Performs an event study using the master_event_list.csv and market_data.pkl. It calculates abnormal returns around the announcement and effective dates of index additions and tests their statistical significance.
   • Key Outputs (in results/event_study/):
     • event_study_results.xlsx
     • Various AAR/CAR plots (e.g., aar_car_plot.png)
   • Depends on: Output from 01_data_acquisition_and_preparation.py.

3. python scripts/03_trading_strategy_backtesting.py

   • Purpose: Backtests several predefined trading strategies based on the index addition events. It uses the event list and market data to simulate trades and calculate performance metrics.
   • Key Outputs (in results/backtesting/):
     • backtest_results.xlsx
     • Trade logs (e.g., trade_log_announcement_day_strategy.csv)
     • Performance plots (e.g., equity_curve_announcement_day_strategy.png)
   • Depends on: Output from 01_data_acquisition_and_preparation.py.

Each script will print log messages to the console indicating its progress and any significant findings or errors. Results, including spreadsheets and plots, will be saved to the results/ directory.

Working with the Modules Directly

You can also use the modules directly in your own scripts:

from src.data.index_changes import process_index_changes
from src.event_study.event_study import create_event_windows, calculate_average_abnormal_returns
from src.backtesting.backtester import IndexAdditionBacktester

# Example: Process index changes
process_index_changes('path/to/input.csv', 'path/to/output.csv')

# ...and so on

Development

Running Tests

Execute tests using pytest:

# Run all tests
pytest

# Run specific test modules
pytest tests/data/test_clean_sp_data.py

Adding New Features

1. First implement core functionality in the appropriate module under src/
2. Update the corresponding __init__.py to expose the new functionality
3. Write tests for the new functionality
4. Integrate into the main workflow scripts as needed

License

MIT License

Acknowledgements

• This tool uses data from S&P Global for index change announcements
• Market data is sourced via the yfinance library

Understanding the Results

The project aims to identify statistically significant price patterns around S&P index additions and assess the viability of trading strategies based on these patterns. Here's how to interpret the outputs:

1. Data Acquisition and Preparation (scripts/01_data_acquisition_and_preparation.py)

• Key Output Files:
  • data/processed/master_event_list.csv: Confirms the events that will be analyzed. Check this file for the completeness and accuracy of event dates and company identifiers.
  • data/processed/market_data.pkl: Contains the raw price/volume data. Its presence indicates successful data fetching.
• What to look for:
  • Log messages indicating the number of events processed and the amount of market data fetched.
  • Ensure no major errors during data fetching or cleaning.
  • Visualizations generated by this script (e.g., distribution of events over time, if implemented) should appear reasonable.

2. Event Study Analysis (scripts/02_event_study_analysis.py)

• Key Output Files (in results/event_study/):
  • event_study_results.xlsx: This is the primary numerical output.
    • AAR (Average Abnormal Return): The average return on event days that cannot be explained by general market movements. Look for statistically significant AARs (e.g., p-value < 0.05) on and around the announcement and effective dates.
    • CAR (Cumulative Abnormal Return): The sum of AARs over a specific window. This shows the total abnormal price movement. Positive CAR leading up to the effective date might indicate pre-positioning by investors.
    • T-statistics and P-values: Indicate the statistical significance of AARs and CARs. Lower p-values (typically < 0.05 or < 0.01) suggest the observed returns are unlikely due to random chance.
    • Subperiod Analysis: Compares CARs across different time periods (e.g., early years vs. recent years) to see if the index effect has changed over time.
  • Plots (.png files):
    • aar_car_plot.png: Visualizes AAR and CAR over the event window. Look for trends and significant spikes.
    • subperiod_caar_comparison.png: Visually compares CARs across different subperiods.
• Interpretation:
  • The goal is to identify if there's a predictable pattern of abnormal returns. For example, a consistent positive CAR leading up to the effective date could suggest a profitable trading opportunity.
  • Pay attention to the economic significance (how large are the returns?) as well as statistical significance.
  • The analysis helps understand the "index effect" – how stock prices react to being added to an index.

3. Trading Strategy Backtesting (scripts/03_trading_strategy_backtesting.py)

• Key Output Files (in results/backtesting/):
  • backtest_results.xlsx: Contains performance metrics for each strategy.
    • Total Return / Annualized Return: Overall profitability.
    • Sharpe Ratio: Risk-adjusted return (higher is better). A Sharpe ratio > 1 is often considered good.
    • Sortino Ratio: Similar to Sharpe, but only considers downside volatility.
    • Max Drawdown: The largest peak-to-trough decline during a specific period, indicating risk.
    • Win Rate / Profit Factor: Measures of trade success.
  • trade_log_<strategy_name>.csv: Detailed list of all trades. Useful for debugging or in-depth analysis of strategy behavior.
  • Plots (.png files):
    • equity_curve_<strategy_name>.png: Shows the growth of portfolio value over time. An upward sloping curve is desired.
    • drawdown_plot_<strategy_name>.png: Visualizes periods of portfolio decline.
    • monthly_returns_heatmap_<strategy_name>.png: Shows return distribution by month and year.
• Interpretation:
  • Compare the performance of different strategies against a benchmark (e.g., Buy and Hold).
  • A successful strategy should demonstrate positive risk-adjusted returns after accounting for transaction costs.
  • Analyze trade logs to understand why a strategy performs well or poorly in certain conditions.
  • The backtest results provide evidence for whether the patterns identified in the event study can be practically exploited.

By examining these outputs, you can understand the typical market reaction to S&P index additions and evaluate the potential of different trading approaches to capitalize on these events.