The purpose of this article is to provide some guide on how to conduct analysis of a sample scenario A/B test results using R, evaluate the results and draw conclusions based on the analysis.

本文的目的是提供一些指南，說明如何使用R對示例場景A / B測試結果進行分析，評估結果并根據分析得出結論。

Before we begin, let’s review what an A/B Test means and what it is typically used for and what are some of its limitations.

在開始之前，讓我們回顧一下A / B測試的含義，通常的用途以及它的一些局限性。

定義 (Definition)

A/B Testing also known as split testing is a general methodology used online when you want to test a new product or feature. The goal is to design an experiment that is robust and gives repeatable results so as to make an informed decision to launch or not.

A / B測試(也稱為拆分測試)是您要測試新產品或功能時在網上使用的一種通用方法。 目標是設計一個健壯的實驗，并給出可重復的結果，以便做出是否啟動的明智決定。

Typically, this test involves comparing two web pages by showing two variants A and B, to a similar number of visitors and the variant which gives better conversion rate wins.

通常，此測試涉及通過將兩個變體A和B展示給相似數量的訪問者來比較兩個網頁，并且變體可以帶來更好的轉化率。

It is mainly an experiment where two or more variations of the same webpage are compared against each other by displaying them to real-time visitors to determine which one performs better for a given goal. A/B testing is not limited by web pages only, you can A/B test your emails, popups, sign up forms, apps and more.

這主要是一項實驗，通過向實時訪問者顯示同一網頁的兩個或多個變體進行比較，以確定哪個變體在給定目標下效果更好。 A / B測試不僅限于網頁，您可以A / B測試電子郵件，彈出窗口，注冊表單，應用程序等。

Typically one of many reasons why an A/B test is done is to ensure new features planned to be introduced do have a measurable (positive) impact on the intended objective for wanting to introduce such feature(s).

通常，進行A / B測試的許多原因之一就是要確保計劃引入的新功能確實對想要引入此類功能的目標具有可衡量的(積極的)影響。

局限性 (Limitations)

A/B test is one of many tools for conversion optimization. It is not an independent solution. It won’t fix all your conversion issues. It can’t fix typical issues you get with messy data. You need to do more than just an A/B test to really improve on conversions

A / B測試是許多用于轉化優化的工具之一。 這不是一個獨立的解決方案。 它不能解決您所有的轉換問題。 它無法解決因凌亂的數據而引起的典型問題。 您要做的不只是A / B測試，還可以真正改善轉換率

Now, let’s take a dive into a practical example of a case study. The datasets and full analysis details can be found on my github If you need some help understanding and using Github, there is a useful link for beginners here.

現在，讓我們深入研究一個案例研究的實際示例。 該數據集和全面分析的細節上可以找到我的GitHub如果你需要一些幫助了解和使用Github上，有初學者有用的鏈接在這里 。

案例分析 (Case Study)

Imagine we have results of A/B tests from two hotel booking websites. (Note: data is made up to look like the real one.) We need to conduct A/B test analysis of the data, draw conclusions from the data and make recommendations to management or product teams.

想象一下，我們有兩個酒店預訂網站的A / B測試結果 。 (注意：數據組成看起來像真實的一樣。)我們需要對數據進行A / B測試分析，從數據中得出結論，并向管理層或產品團隊提出建議。

Note: The analysis below assumes some little knowledge of statistical concepts so as to make it concise and to the point.

注意：下面的分析假設您對統計概念有些了解，以使其簡潔明了。

數據集摘要 (Dataset Summary)

1. Variant A is the control group which depicts the existing products or features on the website.
   變體A是一個對照組，描述了網站上的現有產品或功能。
2. Variant B is the experimental group to experiment the new version of product or feature to see if users like it or if it increases bookings (conversions).
   變體B是一個實驗小組，用于測試產品或功能的新版本，以查看用戶是否喜歡它或它是否增加了預訂(轉化)。
3. Converted — Based on the given dataset, there are two categories defined by logical values (True or false):
   已轉換-根據給定的數據集，由邏輯值定義了兩個類別(“真”或“假”)：
4. Converted = True when customer successfully makes a booking
   當客戶成功進行預訂時，已轉換= True
5. Converted = False when customer visits the sites but does not make a booking
   當客戶訪問網站但未進行預訂時，Converted = False

檢驗假設 (Test Hypothesis)

零假設 (Null Hypothesis)

Both version A and B have the same probability of driving customer booking or conversion. In other words, there is no effect or no difference between version A and B.

版本A和版本B具有相同的驅動客戶預訂或轉化的可能性。 換句話說，版本A和B之間沒有影響或沒有區別。

替代假設 (Alternative Hypothesis)

Both version A and B have a different probability of driving customer booking or conversion. There is a difference between version A and B. Version B is better than A in driving customer bookings. PExp_B != Pcont_A

版本A和版本B具有不同的驅動客戶預訂或轉化的可能性。 版本A和版本B之間存在差異。在推動客戶預訂方面，版本B優于版本B。 PExp_B！= Pcont_A

分析 (Analysis)

Let’s do the analysis together. Download the dataset here. Please note you must have RStudio installed to do this.

讓我們一起做分析。 在此處下載數據集。 請注意，您必須安裝RStudio才能執行此操作。

1. Prepare the dataset and load the tidyverse library which contains the relevant packages used for the analysis.
   準備數據集并加載tidyverse庫，該庫包含用于分析的相關軟件包。

library(tidyverse)
setwd(“~egot_\\Projects\\ABTest”) #set up your own directory
ABTest <- read.csv("Website Results.csv", header = TRUE)#Using read.csv base R file import function so the data can be imported into a dataframe in RStudio.
save(ABTest, file = "~rda\\ABTest.rda")  #save in your own directory

2. Let’s filter conversions for variants A & B and compute their corresponding conversion rates.

2.讓我們過濾變體A和B的轉化并計算其相應的轉化率。

3. Let’s compute the relative uplift using conversion rates A & B. The uplift is a percentage of the increase.

3.讓我們使用轉化率A和B計算相對提升。提升是增量的百分比。

uplift <- (conv_rate_B - conv_rate_A)/ conv_rate_A * 100
uplift #82.72%
#B is better than A by 83%. This is high enough to decide a winner.

4. Let’s compute pooled probability, standard error , margin of error and difference in proportion (point estimate) for variants A & B.

4.讓我們計算變體A和B的合并概率，標準誤差，誤差容限和比例差異(點估計)。

5. Let’s compute the z-score.

5.讓我們計算z得分。

#5a. Compute the Z-score so we can determine the p-value
z_score <- d_hat/SE_poolprint(z_score) #2.249546

6. Using this z-score, we can quickly determine the p-value via a look-up table, or using the code below:

6.使用此z分數，我們可以通過查找表或以下代碼快速確定p值 ：

#5b.Let's compute p_value using the z_score value
p_value <- pnorm(q = -z_score, mean = 0, sd = 1) * 2print(p_value) #0.02447777

7. Let’s compute the confidence interval for the pool

7.讓我們計算池的置信區間

8. Let’s visualize the results computed so far in a dataframe (table):

8.讓我們在數據框(表)中可視化到目前為止所計算的結果：

模型 (Model)

Based on the computations made so far, i created a model for computing ABTests. You can simply throw in your file with corresponding ABTest results in exactly same format as the file i used and once you run this model your ABTest result will be computed and returned to you within seconds.

基于到目前為止的計算，我創建了一個用于計算ABTests的模型。 您可以簡單地將文件和相應的ABTest結果以與我使用的文件完全相同的格式放入文件中，一旦運行此模型，ABTest結果將被計算并在幾秒鐘內返回給您。

10. Finally let’s visualize the results of A & B using a normal distribution.

10.最后，讓我們使用正態分布可視化A和B的結果。

結論與建議 (Conclusions & Recommendation)

1. There are 721 hits and 20 conversions for variant A and 730 hits and 37 conversions for variant B.

1.變體A有721個匹配和20個轉換，變體B有730個匹配和37個轉換。

2. Relative uplift of 82.72% based on a Conversion rate for A = 2.77%, Conversion rate for B = 5.07%. Hence variant B is better than A by 82.72%.

2.基于A的轉化率= 2.77％，B的轉化率= 5.07％，相對提升為82.72％。 因此，變體B比A好82.72％。

3. P-value computed for this analysis was 0.02448 (p < 0.05 or p lower than 5% significance level). Hence, the tests results show strong statistical significance. You can be 95% confident that this result is a consequence of the changes made and not a result of random chance.

3.為此分析計算的P值為0.02448(P <0.05或P低于5％的顯著性水平)。 因此，測試結果顯示出很強的統計學意義。 您可以95％確信此結果是所做更改的結果，而不是隨機機會的結果。

4. Based on the aforementioned results that show strong statistical significance. you should reject the null hypothesis and proceed with launch.

4.基于上述結果，具有很強的統計學意義。 您應該拒絕零假設并繼續進行發布。

5. Hence, Accept variant “B” and roll it out for 100% of the users.

5.因此，接受變體“ B”并將其推廣給100％的用戶。

Note: Please check for the entire source code including the visualization on my Github repository.

注意：請檢查完整的源代碼，包括我Github存儲庫中的可視化文件。

Thanks for reading.

謝謝閱讀。