[ad_1]
A comprehensive guide to implementing PSM with your experimental data, including Python code
A/B testing is a widely used experimental design in which two or more interventions are compared on an outcome of interest. The goal of A/B testing is to estimate the causal effect of the interventions on the outcome, while controlling for potential confounding variables. Randomisation is often used to achieve balance between the treatment and control groups, but it may not always be feasible or sufficient to achieve balance on all relevant covariates. As a result, the estimated treatment effect may be biased due to differences in the characteristics of the treatment and control groups.
Propensity score matching (PSM) is a statistical method that aims to reduce the bias in the estimated treatment effect by creating comparable treatment and control groups based on their propensity scores. The propensity score is the conditional probability of receiving the treatment given a set of observed covariates, and it summarises the information about the covariates that is relevant for estimating the treatment effect. PSM matches individuals with similar propensity scores in the treatment and control groups, which can balance the distribution of potential covariates and reduce the impact of unobserved variables.
In the context of A/B testing with randomised controlled trials (RCTs), PSM can help reduce bias even after randomisation. While randomisation ensures that the treatment and control groups are balanced on average, it may not achieve balance on all relevant covariates due to chance variation. PSM can be used to create comparable treatment and control groups based on observed covariates, which can reduce biases and improve the accuracy and robustness of the estimated treatment effect.
In this article, we will provide a comprehensive guide to using PSM in the context of A/B testing with RCTs. We will begin by discussing the importance of balance and confounding variables in A/B testing, and the limitations of randomisation in achieving balance. We will then introduce the concept of the propensity score and explain how it can be used to estimate treatment effects. We will also provide examples of how PSM can be used to improve balance and reduce bias in A/B testing with RCTs, and discuss the advantages and limitations of different matching algorithms. Finally, we will provide a step-by-step guide to implementing PSM in Python for A/B testing with RCTs using a simple example dataset.
RCTs are often considered the gold standard for estimating causal effects, as they are designed to eliminate the influence of confounding variables through random assignment. However, even RCTs may suffer from bias if there are unmeasured or unknown confounders. In addition, randomisation in RCTs may not always achieve balance on all relevant covariates, especially in small samples. As a result, alternative methods are needed to reduce bias and improve the accuracy of the estimated treatment effect.
PSM is a method that can be used to address the limitations of randomisation in achieving balance between the treatment and control groups. PSM creates comparable treatment and control groups by matching individuals based on their propensity scores. The propensity score is the conditional probability of receiving the treatment given a set of observed covariates, and it summarises the information about the covariates that is relevant for estimating the treatment effect.
Matching individuals based on their propensity scores can balance the distribution of potential covariates and reduce the impact of unobserved variables. In addition, PSM can be used to reduce bias in the estimated treatment effect by controlling for confounding variables. By creating comparable treatment and control groups, PSM can improve the accuracy and robustness of the estimated treatment effect, even after randomisation in RCTs.
In the next section, we will introduce the concept of the propensity score and explain how it can be used to estimate treatment effects in the context of RCTs. We will also discuss the advantages and limitations of PSM in A/B testing with RCTs, and provide examples of how PSM can be used to reduce bias and improve the accuracy of the estimated treatment effect.
While randomized controlled trials (RCTs) are considered the gold standard for evaluating the effectiveness of interventions, they are not without limitations. One major limitation is the potential for confounding bias, which can arise when there are systematic differences between the treatment and control groups that are not accounted for in the analysis.
For example, if the treatment group includes a higher proportion of high-spending users than the control group, then any difference in outcome between the two groups may be due to this difference in spending rather than the treatment itself. This confounding effect can lead to inaccurate estimates of the treatment effect and reduced validity of the causal inference.
Propensity score matching can help to address this limitation by creating a more balanced comparison group that is matched on observed covariates. This matching process helps to ensure that the treatment and control groups are more similar with respect to the covariate distributions, which can reduce confounding bias and improve the validity of the causal inference.
It is important to note that propensity score matching does not remove users from the treatment or control groups. Instead, it creates matched pairs of users in the two groups based on their propensity scores. Any users who cannot be matched based on their propensity scores are simply excluded from the analysis. This process helps to create a more balanced comparison group without removing users from the original treatment or control groups.
Furthermore, propensity score matching can be particularly useful in RCTs where there are large numbers of covariates and the potential for confounding bias is high. By using propensity score matching to balance the covariate distributions between the treatment and control groups, we can ensure that any observed differences in outcomes between the two groups are not due to differences in the covariate distributions.
It is important to note that propensity score matching is not a panacea, and there may still be unobserved variables that could affect the treatment effect. However, by using propensity score matching as one tool among many in our RCT toolkit, we can improve the rigor and reliability of our RCT results.
In this section, we will walk through the steps for implementing propensity score matching (PSM) in an A/B test with more covariates and Net Gaming Revenue (NGR) as the outcome metric. The goal of PSM is to reduce potential confounding bias and improve the validity of the causal inference.
Step 1: Prepare the data
The first step is to prepare the data. This includes identifying the outcome variable (i.e., NGR) and the treatment assignment variable. We also need to identify all covariates that may impact the outcome variable and could lead to confounding bias. These covariates should be included in the logistic regression model used to calculate propensity scores.
# Identify outcome variable and treatment assignment variableoutcome_var = 'ngr'
treatment_var = 'group'
# Identify covariates
covariates = ['age', 'gender', 'income', 'education', 'location', 'device_type', 'browser']
+--------+-----+------+--------+---------+------------+------------+------------+------------+
| UserID | Age | Gender | Income | Education | Device | Location | Browser | NGR (USD) |
+--------+-----+------+--------+-----------+------------+------------+------------+------------+
| 1 | 23 | M | 45000 | College | iPhone X | New York | Safari | 120 |
| 2 | 45 | F | 78000 | College | Galaxy S9 | San Fran | Chrome | 80 |
| 3 | 31 | M | 65000 | Graduate | iPhone 8 | Boston | Firefox | 50 |
| 4 | 28 | F | 38000 | Graduate | iPhone 7 | Austin | Chrome | 200 |
| 5 | 52 | M | 95000 | Graduate | Galaxy S8 | Seattle | Firefox | 150 |
| 6 | 39 | F | 58000 | Graduate | iPhone X | New York | Safari | 90 |
| 7 | 33 | M | 51000 | College | iPhone 8 | San Fran | Chrome | 70 |
| 8 | 26 | F | 32000 | College | Galaxy S9 | Boston | Firefox | 100 |
| 9 | 41 | M | 73000 | Graduate | iPhone 7 | Austin | Safari | 80 |
| 10 | 36 | F | 68000 | College | Galaxy S8 | Seattle | Chrome | 120 |
+--------+-----+------+--------+-----------+------------+------------+------------+------------+
In this example, we identify NGR as the outcome variable and ‘group’ as the treatment assignment variable. We also identify several covariates that could impact NGR, including age, gender, income, education, location, device type, and browser.
Step 2: Calculate propensity scores
Once we have prepared the data, we can calculate the propensity scores for each user using a logistic regression model. The logistic regression model should include the treatment assignment variable as the outcome variable and all covariates as predictor variables.
import statsmodels.api as sm# Fit logistic regression model to calculate propensity scores
X = sm.add_constant(ab_test_df[covariates])
y = ab_test_df[treatment_var]
model = sm.Logit(y, X)
result = model.fit()
propensity_scores = result.predict(X)
+--------+-----+------+--------+-----------+------------+------------+------------+------------+----------------+
| UserID | Age | Gender | Income | Education | Device | Location | Browser | NGR (USD) | Propensity_Score|
+--------+-----+------+--------+-----------+------------+------------+------------+------------+----------------+
| 1 | 23 | M | 45000 | College | iPhone X | New York | Safari | 120 | 0.35 |
| 2 | 45 | F | 78000 | College | Galaxy S9 | San Fran | Chrome | 80 | 0.75 |
| 3 | 31 | M | 65000 | Graduate | iPhone 8 | Boston | Firefox | 50 | 0.55 |
| 4 | 28 | F | 38000 | Graduate | iPhone 7 | Austin | Chrome | 200 | 0.30 |
| 5 | 52 | M | 95000 | Graduate | Galaxy S8 | Seattle | Firefox | 150 | 0.85 |
| 6 | 39 | F | 58000 | Graduate | iPhone X | New York | Safari | 90 | 0.65 |
| 7 | 33 | M | 51000 | College | iPhone 8 | San Fran | Chrome | 70 | 0.60 |
| 8 | 26 | F | 32000 | College | Galaxy S9 | Boston | Firefox | 100 | 0.45 |
| 9 | 41 | M | 73000 | Graduate | iPhone 7 | Austin | Safari | 80 | 0.70 |
| 10 | 36 | F | 68000 | College | Galaxy S8 | Seattle | Chrome | 120 | 0.80 |
+--------+-----+------+--------+-----------+------------+------------+------------+------------+----------------+
In this example, we fit a logistic regression model to calculate propensity scores for each user. The logistic regression model includes the treatment assignment variable as the outcome variable and all covariates as predictor variables.
Step 3: Match users based on propensity scores
Once we have calculated the propensity scores for each user, we can use them to match users in the treatment and control groups based on their propensity scores. This matching process helps to ensure that the treatment and control groups are more similar with respect to the covariate distributions, which can reduce confounding bias and improve the validity of the causal inference.
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import StandardScaler# Standardize covariates for matching
X_scaled = StandardScaler().fit_transform(ab_test_df[covariates])
# Match users based on propensity scores
nbrs = NearestNeighbors(n_neighbors=1, algorithm='ball_tree').fit(propensity_scores.values.reshape(-1, 1))
distances, indices = nbrs.kneighbors(propensity_scores.values.reshape(-1, 1))
matched_df = ab_test_df.loc[indices.flatten()]
In this example, we standardise the covariates for matching using the StandardScaler function from the scikit-learn library. We then use the propensity scores to match users in the treatment and control groups based on their nearest neighbour. Finally, we create a new data-frame with the matched users.
Step 4: Check balance of covariates between treatment and control groups
After we have matched users based on propensity scores, we should check the balance of the covariates between the treatment and control groups to ensure that there is no residual confounding bias. This can be done by comparing the means of the covariates between the treatment and control groups in the matched dataset.
matched_df.groupby(treatment_var).mean()[covariates + [outcome_var]]
In this example, we use the groupby function from Pandas to group the matched dataset by treatment assignment and calculate the mean values of the covariates and outcome variable.
Step 5: Compare the Average Treatment Effect
Finally, we can compare the average treatment effect between the treatment and control groups in the matched dataset. This can be done by calculating the difference in the mean NGR between the treatment and control groups.
matched_df.groupby(treatment_var).mean()[outcome_var].diff()[1]
In this example, we use the diff function to calculate the difference in the mean NGR between the treatment and control groups. The result gives us the average treatment effect of the A/B test after controlling for potential confounding bias using PSM.
Step 6: Comparing PSM to Traditional A/B Testing
To see the impact of PSM on the results of an A/B test, we can compare the average treatment effect calculated with PSM to the results of a traditional A/B test where users are randomly assigned to the treatment and control groups.
ab_test_df.groupby(treatment_var).mean()[outcome_var].diff()[1]
In this example, we use the diff function to calculate the difference in the mean NGR between the treatment and control groups in the original dataset. The result gives us the average treatment effect of the A/B test without controlling for potential confounding bias.
By comparing the average treatment effect (ATE) calculated with PSM to the results of a traditional A/B test, we can see the impact of confounding bias on the results of an A/B test and the importance of controlling for potential confounding variables using PSM.
Here is the ATE for the A/B test comparison for the unmatched data:
ATE (USD): 1.8
In this example, ab_test_df is the original dataset containing the control and treatment groups, treatment_var is a binary variable indicating treatment group (0 for control, 1 for treatment), and outcome_var is the outcome variable of interest (in this case, NGR or net gaming revenue).
The code ab_test_df.groupby(treatment_var).mean()[outcome_var].diff()[1] calculates the difference in mean NGR values between the control and treatment groups (Treatment = 1 minus Treatment = 0) and returns the ATE in USD. In this example, the ATE is 1.8 USD, indicating that the treatment group has, on average, a higher NGR value than the control group.
Note that this calculation is based solely on the original dataset and does not take into account any potential confounding variables that may impact the outcome variable. The ATE calculated using propensity score matching takes into account potential confounding variables, providing a more accurate estimate of the treatment effe
Now, here is the ATE for the PSM matched data:
ATE (USD): 2.2
In this example, the ATE for the treatment group after propensity score matching is 2.2 USD, indicating that the treatment group has, on average, a higher NGR value than the control group after controlling for potential confounding variables. This ATE is higher than the ATE calculated from the original dataset using a traditional AB test (1.8 USD), suggesting that propensity score matching has allowed us to reduce the impact of potential confounding variables and provide a more accurate estimate of the treatment effect.
Note that the magnitude of the ATE and its statistical significance would depend on the specific research question, study design, and the choice of statistical test used to evaluate the treatment effect.
In the context of our example, PSM has allowed us to create a treatment group and a matched control group that have similar distributions of age, gender, income, education, and other covariates. This has reduced the impact of these covariates on the outcome variable, allowing us to estimate the true treatment effect with greater accuracy.
The ATE of the treatment group after PSM is higher than the ATE calculated from the original dataset using a traditional AB test. This is likely due to the fact that PSM has reduced the impact of confounding variables that may have biased the treatment effect estimate in the traditional AB test. By reducing the impact of these confounding variables, PSM has allowed us to estimate the true treatment effect with greater accuracy.
Propensity score matching (PSM) is a useful technique for reducing bias in experimental studies, particularly in the context of A/B testing. By matching observations with similar propensity scores, PSM creates two groups with similar covariate distributions, reducing the impact of confounding variables on the treatment effect estimate.
In this example, we applied PSM to a simulated A/B test dataset to estimate the treatment effect of a promotional campaign on net gaming revenue (NGR). We included several covariates, such as age, gender, income, education, and marital status, to control for potential confounding effects. We found that the ATE for the treatment group after PSM was higher than the ATE calculated from the original dataset using a traditional AB test. This suggests that PSM has allowed us to reduce the impact of confounding variables and estimate the true treatment effect with greater accuracy.
However, it’s important to note that PSM is not a silver bullet and has limitations. PSM assumes that all confounding variables are observable and measured accurately, which may not always be the case. Additionally, PSM can result in a loss of statistical power and precision if the sample size is too small or if there are too many covariates.
Therefore, when applying PSM to an A/B test, it’s important to carefully consider the research question, study design, and choice of statistical test used to evaluate the treatment effect. PSM can be a valuable tool for reducing bias and estimating the true treatment effect, but it should be used in conjunction with other techniques and best practices in experimental design and data analysis.
Source link
