How To: Adjusting NBA Teams' Offensive and Defensive Ratings using Strength of Schedule

This tutorial goes through my process of adjusting NBA Teams' Offensive and Defensive Ratings for strength of schedule (SoS). First read my blog post on the same topic, before going any further. The blog post explains the details including the math necessary for understanding the code in this tutorial.

The code for RAPM styled approach is adopted from Ryan Davis' RAPM Tutorial and I suggest you read that tutorial before continuing.

If you want to run the code yourself while reading the tutorial, you can find the notebook version of this tutorial on my github:

(https://github.com/sravanpannala/NBA-Tutorials/blob/main/sos_adjusted_ratings/how_to_adjust_nba_team_ratings_for_sos.ipynb

First let's import the necessary packages to run this code:

import pandas as pd  # for processing data
import numpy as np  # for numerical operations on arrays
from tqdm import tqdm  # gives up progress bar
import time  # for time related stuff
from sklearn.linear_model import RidgeCV

# don't raise warnings when chaining pandas operations
pd.options.mode.chained_assignment = None

Then we will load the team information as two variable. There are 30 teams in the NBA and each team has a name and a team ID

1. teams_list will have a list of all team IDs
2. teams_dict is a dictionary mapping the team IDs to the team names.

team_data = pd.read_csv("../data/NBA_teams_database.csv")
teams_list = team_data["TeamID"].tolist()
team_dict1 = team_data.to_dict(orient="records")
teams_dict = {team["TeamID"]: team["Team"] for team in team_dict1}

Scraping the Data Required

This section will cover the scraping part of the tutorial. You can skip the tutorial and go to the next section if you wish so. The data has already been scraped and is available for the 2023-24 season in the data folder.
We will be using the nba_api to get the necessary data. It should be installed already if you followed the instructions in Readme. The team ratings i.e. offensive, defensive and net ratings can be found for each game by using the boxscoreadvancedv3 endpoint. This endpoint needs needs the GameID to get the boxscores for both teams in that game. To get GameIDs for all games played in the 2023-24 season, we will use the leaguegamelog endpoint.

from nba_api.stats.endpoints import leaguegamelog, boxscoreadvancedv3

# for 2023-24 season
season = "2023"
# get the information
stats = leaguegamelog.LeagueGameLog(
    player_or_team_abbreviation="T",
    season=season,
    season_type_all_star="Regular Season",
)
# output the information as pandas dataframe
df = stats.get_data_frames()[0]
# get the GameIDs as a list
game_ids = df["GAME_ID"].tolist()
# GameIDs are repeated twich, once for home team and once for away team
# We can use numpy unique to remove the duplicates
game_ids = np.unique(game_ids)

Now we have a list of game_ids to use in boxscoreadvancedv3 endpoint. We just put the game_ids in a for loop to get the data for each game as a dataframe. We append the generated dataframe for each game to a list of dataframes dfa. Finally we can use pandas.concat to concatenate all the dataframes into a single dataframe for the season.
This process might take a while (10-20 minutes, depending on the number of games played), so grab a coffee or a snack and come back after some time. There is a small (maybe big) issue, if you just run a vanilla for loop. The stats.nba.com endpoint we use to scrape the data, times out when requested too many times in a short period of time and results in a error:

HTTPSConnectionPool(host='stats.nba.com', port=443): Read timed out. (read timeout=30)

Any error will stop the for loop and we have to repeat again. To prevent this issue, we wrap the call to the endpoint in try except blocks and retry the endpoint for that gameId till it succeeds.
I found an elegant solution for this issue while creating this tutorial which is to use the tenacity package.

1. We import the necessary modules from tenacity:
   1. retry: decorator to enable retries on the function
   2. stop_after_attempt: to define the maximum number of attempts. I set it as 5
   3. wait_fixed: to wait for a certain amount of fixed time before retrying. The number I use is 0.6 seconds as recommended by the authors of the nba_api

from tenacity import retry
from tenacity.stop import stop_after_attempt
from tenacity.wait import wait_fixed

2. We add the retry decorator with the necessary options to the get_boxscores function, which has the try except block to handle errors

@retry(stop=stop_after_attempt(5), wait=wait_fixed(0.6))
def get_boxscores(game_id):
    try:
        stats = boxscoreadvancedv3.BoxScoreAdvancedV3(game_id=game_id)
        df1 = stats.get_data_frames()[1]
    except Exception as error:
        print(error)
    return df1

3. Now we run the for loop with the decorated get_boxscores function. Finally, we save the scraped data as a csv file in the data folder.

dfa = []
for game_id in tqdm(game_ids):
    df1 = get_boxscores(game_id)
    dfa.append(df1)
df = pd.concat(dfa)
df.to_csv(f"./data/NBA_BoxScores_Adv_{season}.csv")

 13%|█▎        | 49/363 [00:59<06:26,  1.23s/it]

HTTPSConnectionPool(host='stats.nba.com', port=443): Max retries exceeded with url: /stats/boxscoreadvancedv3?EndPeriod=0&EndRange=0&GameID=0022300050&RangeType=0&StartPeriod=0&StartRange=0 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x00000207782F5090>, 'Connection to stats.nba.com timed out. (connect timeout=30)'))


100%|██████████| 363/363 [05:27<00:00,  1.11it/s]

Loading and Pre-Processing the Data

Now lets load the data. The data has a lot of columns we don't use. So to we import only the data necessary by using the usecols option in pandas.read_csv().

season = "2023"
cols = [
    "gameId",
    "teamName",
    "teamId",
    "offensiveRating",
    "defensiveRating",
    "netRating",
    "possessions",
]
df = pd.read_csv(f"./data/NBA_BoxScores_Adv_{season}.csv", usecols=cols)
cols = ["gameId", "tId", "team", "ORtg", "DRtg", "NRtg", "poss"]
df.columns = cols
df.head(4)

As you see the printed table, each gameId has two entries, one of each team in the game. Each row has only the information for that team. But what we need is a combined row entry with the opponent information also.
We will use pandas.groupby to achieve that. The variable to apply the operation will be gameId. This operation will create a groupby object, on which further operations can be run.

df1 = df.groupby("gameId")
df1

<pandas.core.groupby.generic.DataFrameGroupBy object at 0x00000207787B7710>

We then use the nth operation to get the 1st and 2nd rows of each game.

df1_1 = df1.nth(0)
df1_2 = df1.nth(1)
display(df1_1.head(2))
display(df1_2.head(2))

We can then rename the columns of the 1st dataframe, adding 1 to all its column names, except the gameId column (which is needed for the merging operation later). For the 2nd dataframe, similarly add 2 to the columns names.

df1_1.columns = ["gameId"] + [s + "1" for s in df1_1.columns if s != "gameId"]
df1_2.columns = ["gameId"] + [s + "2" for s in df1_2.columns if s != "gameId"]
display(df1_1.head(2))
display(df1_2.head(2))

We then merge the two dataframes df1_1 and df1_2 on the column gameId, generating the dataframe we need.

df1_3 = pd.merge(df1_1, df1_2, on="gameId")
display(df1_3.head(2))

One more step remaining. What we have right now is one row of each game. But, what we need is two rows for each game as described in my blog post. To get that dataframe, we repeat the process above, with 0 and 1 flipped when performing the nth operation. Finally we merge the two dataframes df1_3 and df1_6, to get the combined dataframe with two rows for each game.

df1_4 = df1.nth(1)
df1_5 = df1.nth(0)
df1_4.columns = ["gameId"] + [s + "1" for s in df1_4.columns if s != "gameId"]
df1_5.columns = ["gameId"] + [s + "2" for s in df1_5.columns if s != "gameId"]
df1_6 = pd.merge(df1_4, df1_5, on="gameId")
df2 = pd.concat([df1_3, df1_6]).sort_values(by="gameId").reset_index(drop=True)
data = df2.copy()
data.head(4)

Processing the Data

To process the data in a format required by the Ridge Regression algorithm RidgeCV, we define the following functions:

maps_teams()

1. Makes the matrix rows to be used in ridge regression
2. The weights for each team = 1/2
3. Equations per game are:
   $$\frac{1}{2}\hat{Team}^1_{OFF} + \frac{1}{2}\hat{Team}^2_{DEF} = Team^1_{OFF} $$ $$\frac{1}{2}\hat{Team}^2_{OFF} + \frac{1}{2}\hat{Team}^1_{DEF} = Team^2_{OFF} $$
4. The reason for doing this is that for unadjusted values of a game: $$ Team^1_{OFF} = Team^2_{DEF} $$
5. So, $$ Team^1_{OFF} = 0.5\times Team^1_{OFF} + 0.5\times Team^2_{DEF} $$
6. Therefore I use a similar structure for estimating adjusted ratings

def map_teams(row_in, teams, scale):
    t1 = row_in[0]
    t2 = row_in[1]

    rowOut = np.zeros([len(teams) * 2])
    rowOut[teams.index(t1)] = scale
    rowOut[teams.index(t2) + len(teams)] = scale

    return rowOut

convert_to_matrices()

1. Converts each row of data dataframe to x stints.
2. Then maps those rows using map_teams function to get matrix X rows
3. Gets Y rows. Here Y is ORtg1 i.e. we are trying to predict the offensive rating of the 1st team for every row

def convert_to_matricies(possessions, name, teams, scale=1):
    # extract only the columns we need
    # Convert the columns of player ids into a numpy matrix
    stints_x_base = possessions[["tId1", "tId2"]].to_numpy()
    # Apply our mapping function to the numpy matrix
    stint_X_rows = np.apply_along_axis(map_teams, 1, stints_x_base, teams, scale=scale)
    # Convert the column of target values into a numpy matrix
    stint_Y_rows = possessions[name].to_numpy()

    # return matricies and possessions series
    return stint_X_rows, stint_Y_rows

lambda_to_alpha()

• In stats world (R), glmnet() is used for Ridge Regression and uses the parameter $\lambda$. Most the NBA stats people use this parameter $\lambda$ for discussing the regularization parameter. But sklearn.linear_model.RidgeCV() has a parameter $\alpha$, which isn't the same.
• So we need to convert $\lambda$ to $\alpha$ needed for Ridge CV. More details here

def lambda_to_alpha(lambda_value, samples):
    return (lambda_value * samples) / 2.0

calculate_netrtg()

1. Converts lambdas to alphas using lambda_to_alpha function
2. Defines the ridge regression problem using scikit-learn's RidgeCV algorithm
3. cv=5 is chosen i.e. k-fold cross-validation splitting strategy using k=5
4. Intercept is set as true. This value is to be added later to our estimation results to get Offensive and Defensive ratings.
5. Gets coefficients and intercept
6. Add intercept to intercept to get adjusted ratings. Use adjusted off and def ratings to calculate adjusted net rating.
7. Create and return adjusted ratings dataframe

def calculate_netrtg(train_x, train_y, lambdas, teams_list):
    alphas = [lambda_to_alpha(l, train_x.shape[0]) for l in lambdas]
    # create a 5 fold CV ridgeCV model. Our target data is not centered at 0, so we want to fit to an intercept.
    clf = RidgeCV(alphas=alphas, cv=5, fit_intercept=True)

    # fit our training data
    model = clf.fit(
        train_x,
        train_y,
    )

    # convert our list of players into a mx1 matrix
    team_arr = np.transpose(np.array(teams_list).reshape(1, len(teams_list)))

    # extract our coefficients into the offensive and defensive parts
    coef_offensive_array = model.coef_[0 : len(teams_list)][np.newaxis].T
    coef_defensive_array = model.coef_[len(teams_list) : 2 * len(teams_list)][
        np.newaxis
    ].T
    # concatenate the offensive and defensive values with the playey ids into a mx3 matrix
    team_id_with_coef = np.concatenate(
        [team_arr, coef_offensive_array, coef_defensive_array], axis=1
    )
    # build a dataframe from our matrix
    teams_coef = pd.DataFrame(team_id_with_coef)
    intercept = model.intercept_
    teams_coef.columns = ["tId", "aOFF", "aDEF"]
    teams_coef["aNET"] = teams_coef["aOFF"] - teams_coef["aDEF"]
    teams_coef["aOFF"] = teams_coef["aOFF"] + intercept
    teams_coef["aDEF"] = teams_coef["aDEF"] + intercept
    teams_coef["Team"] = teams_coef["tId"].map(teams_dict)
    results = teams_coef[["tId", "Team", "aOFF", "aDEF", "aNET"]]
    results = results.sort_values(by=["aNET"], ascending=False).reset_index(drop=True)
    return results, model, intercept

Estimating Adjusted Ratings

Next, we run the functions defined above to generated the adjusted ratings

train_x, train_y = convert_to_matricies(data, "ORtg1", teams_list, scale=0.5)
lambdas_net = [0.015, 0.075, 0.15]
results_adj, model, intercept = calculate_netrtg(
    train_x, train_y, lambdas_net, teams_list
)
print(f"Intercept = {intercept}")

Intercept = 114.2197043446658

The intercept here can be interpreted as the league average offensive/defensive rating. Here are the adjusted ratings.

results_adj

Finishing Touches

We're not done yet. Now we need to compare the adjusted ratings with the unadjusted ones. But, we haven't calculated the unadjusted ratings yet. Let's do it now.

For a single game: $$ PTS_{OFF}*100 = ORtg^1 \times poss^1 $$ $$ PTS_{DEF}*100 = DRtg^1 \times poss^1 $$

Applying these operations on the data dataframe:

data["pts_off"] = data["ORtg1"] * data["poss1"]
data["pts_def"] = data["DRtg1"] * data["poss1"]

We have to use the groupby operation again, now on the tId1 column. After the groupby operation, we chain an agg (aggregate) operation, which applies a function on all rows of the group. The function we chose here is sum, which adds all the pts and and poss for a team.

off_p = data.groupby(["tId1"])[["poss1", "pts_off"]].agg("sum").reset_index()
def_p = data.groupby(["tId1"])[["poss1", "pts_def"]].agg("sum").reset_index()

The unadjusted team ratings would then be: $$ OFF = \frac{PTS_{OFF}^{Total}}{poss^{Total}} $$ $$ DEF = \frac{PTS_{DEF}^{Total}}{poss^{Total}} $$

off_p["OFF"] = off_p["pts_off"] / off_p["poss1"]
off_p = off_p[["tId1", "OFF"]]
def_p["DEF"] = def_p["pts_def"] / def_p["poss1"]
def_p = def_p[["tId1", "DEF"]]

We then merge these ratings to the results_adj dataframe

results_net = pd.merge(off_p, def_p, on=["tId1"])
results_net["NET"] = results_net["OFF"] - results_net["DEF"]
results_net.rename(columns={"tId1": "tId"}, inplace=True)
results_net = results_net.astype(float).round(2)
results_net["tId"] = results_net["tId"].astype(int)
results_adj["tId"] = results_adj["tId"].astype(int)
results_comb = pd.merge(results_net, results_adj, on=["tId"])
results_comb["aOFF"] = results_comb["aOFF"]
results_comb["aDEF"] = results_comb["aDEF"]
results_comb["oSOS"] = results_comb["aOFF"] - results_comb["OFF"]
results_comb["dSOS"] = results_comb["DEF"] - results_comb["aDEF"]
results_comb["SOS"] = results_comb["oSOS"] + results_comb["dSOS"]
results_comb.iloc[:, 1:] = results_comb.iloc[:, 1:].round(1)
results = results_comb[
    ["Team", "OFF", "oSOS", "aOFF", "DEF", "dSOS", "aDEF", "NET", "SOS", "aNET"]
]
results = results.sort_values(by="aNET", ascending=0).reset_index(drop=True)
results.index = results.index + 1

Reminder

You can find the notebook version of this tutorial on my github: (https://github.com/sravanpannala/NBA-Tutorials/blob/main/sos_adjusted_ratings/how_to_adjust_nba_team_ratings_for_sos.ipynb

Final Combined Data table:

You can save it as csv file and then you some fancy visualization tool to create a pretty looking table and/or efficiency landscape graph

results