As a data science engineer, it’s imperative that the sample data set which you pick from the data is reliable, clean, and well tested for its usability in machine learning model building.

作為數據科學工程師，當務之急是從數據中挑選出的樣本數據集是可靠的，干凈的，并經過了良好的測試，以證明其在機器學習模型構建中的可用性。

So how do you do that?

那你該怎么做呢？

Well, we have multiple statistical techniques like descriptive statistics where we measure the data central value, how it is spread across the mean/median. Is it normally distributed or there is a skew in the data spread? Please refer to my previous article on the same for more clarity.

好吧，我們有多種統計技術，例如描述性統計，在其中我們測量數據中心值，其在均值/中位數之間的分布方式。 它是正態分布的還是數據散布有偏差？ 請參考我以前關于同一文章的更多信息。

As the first thing we do is to visualize the data using various data visualization techniques to make some early sense of any data skewness or discrepancies, to identify any kind of relationship between data set variables.

首先，我們要使用各種數據可視化技術來可視化數據，以便對任何數據偏斜或差異有早期的了解，以識別數據集變量之間的任何類型的關系。

Data has so much to say and we data engineer give it a voice to express and describe itself, using descriptive statistical techniques.

數據有這么多話要說，我們的數據工程師使用描述性統計技術表達和描述自己的聲音。

But to make any prediction or to infer something beyond the given data to find any hidden probability, we rely on inferential statistic techniques.

但是，要進行任何預測或推斷超出給定數據的內容以找到任何隱藏的概率，我們將依靠推斷統計技術。

Inferential statistics are concerned with making inferences based on relations found in the sample, to relations in the population. Inferential statistics help us decide, for example, whether the differences between groups that we see in our data are strong enough to provide support for our hypothesis that group differences exist in general, in the entire population.

推論統計涉及根據樣本中發現的關系對總體中的關系進行推論。 推論統計可以幫助我們確定，例如，我們在數據中看到的群體之間的差異是否足夠大，足以支持我們的假設，即整個人群中普遍存在群體差異。

Today we will cover one of the inferential statistical mechanisms to understand the concept of hypothesis testing using a popular Chi-Square test.

今天，我們將介紹一種推論統計機制，以了解使用流行的卡方檢驗進行假設檢驗的概念。

什么是卡方檢驗？ (What is the Chi-Square Test?)

Do remember that,

請記住，

It is an inferential statistical test that works on categorical data.

這是一種推論統計檢驗，適用于分類數據。

The Chi-Squared test is a statistical hypothesis test that assumes (the null hypothesis) that the observed frequencies for a categorical variable match the expected frequencies for the categorical variable. The test calculates a statistic that has a chi-squared distribution, named for the Greek capital letter Chi (X) pronounced “ki” as in kite.

卡方檢驗是一種統計假設檢驗，它假設(原假設)分類變量的觀測頻率與分類變量的預期頻率匹配。 該測試計算出具有卡方分布的統計量，該統計量以希臘大寫字母Chi(X)命名，在風箏中的發音為“ ki”。

We try to test the likelihood of test data(sample data) to find out whether the observed distribution of data set is a statistical fluke(due to chance ) or not. “Goodness of fit” statistic in the chi-square test, measures how well the observed distribution of data fits with the distribution that is expected if the variables are independent.

我們嘗試測試測試數據(樣本數據)的可能性，以發現觀察到的數據集分布是否為統計偶然(由于偶然)。 卡方檢驗中的“擬合優度”統計量用于衡量觀察到的數據分布與變量獨立的預期分布的吻合程度。

卡方如何工作？ (How Does Chi-Square Work?)

Generally, we try to establish a relationship between the given categorical variable in this test. Chi-square evaluates whether given variables in a data set(sample) are independent, called the Test of Independence. Chi-square tests are used for testing hypotheses about one or two categorical variables and are appropriate when the data can be summarized by counts in a table. The variables can have multiple categories.

通常，在此測試中，我們嘗試在給定的類別變量之間建立關系。 卡方可評估數據集(樣本)中的給定變量是否獨立，稱為獨立性測試。 卡方檢驗用于檢驗關于一個或兩個類別變量的假設，并且當可以通過表中的計數來匯總數據時，卡方檢驗是適用的。 變量可以具有多個類別。

卡方檢驗類型： (Type of Chi-Square Test:)

For One Categorical Variable, we perform

對于一個分類變量，我們執行

• Chi-Square Goodness-of-Fit Test

  卡方擬合優度檢驗

The chi-square goodness of fit test begins by hypothesizing that the distribution of a variable behaves in a particular manner. For example, in order to determine the daily staffing needs of a retail store, the manager may wish to know whether there is an equal number of customers each day of the week.

擬合檢驗的卡方檢驗的優劣始于假設變量的分布以特定方式表現。 例如，為了確定零售商店的日常人員需求，經理可能希望知道一周中的每一天是否有相等數量的顧客。

For, Two Categorical Variables, we perform

為此，我們執行兩個分類變量

• Chi-Square Test for Association

  卡方檢驗

Another way we can describe the Chi-square test is that:

我們可以描述卡方檢驗的另一種方式是：

It tests the null hypothesis that the variables are independent.

它測試變量是獨立的零假設。

The test compares the observed data to a model that distributes the data according to the expectation that the variables are independent. Wherever the observed data doesn’t fit the model, the likelihood that the variables are dependent becomes stronger, thus proving the null hypothesis incorrect!

該測試將觀察到的數據與一個模型進行比較，該模型根據變量是獨立的期望來分配數據。 無論在何處觀察到的數據都不適合模型，變量所依賴的可能性都會變得更大，從而證明原假設不正確！

卡方假設： (Hypothesis In Chi-Square:)

The first thing as a data engineer, you need to establish before performing any Inferential statistic test like Chi-Square, is to establish

作為數據工程師，在執行任何推理統計檢驗(如Chi-Square)之前，您需要建立的第一件事是建立

• H0: Null Hypothesis
  H0：零假設
• H1: Alternate Hypothesis
  H1：替代假設

對于一個分類變量： (For One Categorical Variable:)

• Null hypothesis: The proportions match an assumed set of proportions

  零假設 ：比例與假設的比例集匹配

• Alternative hypothesis: At least one category has a different proportion. ?

  替代假設 ：至少一個類別具有不同的比例。 ?

對于，兩個分類變量： (For, Two Categorical Variables:)

• Null hypothesis: There is no association between the two variables

  零假設 ：兩個變量之間沒有關聯

• Alternative hypothesis: There is an association between the two variable

  替代假設 ：兩個變量之間存在關聯

Before we jump into understanding how Chi-square works with an example, we need to understand what is Chi-square distribution & some other related concepts. This Chi-squared distribution is what we will analyze going forward in the chi-square or χ2 test.

在通過示例了解卡方的工作原理之前，我們需要了解什么是卡方分布及其他一些相關概念。 卡方分布是我們將在卡方 檢驗或χ2檢驗中繼續分析的結果。

什么是卡方分布？ (What Is Chi-Square Distribution?)

The chi-square distribution (also chi-squared or χ2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables.

具有 k 個自由度 的 卡方分布 (也 稱為 卡方 或 χ2分布 ) 是 k個獨立標準正態隨機變量 的平方和的分布 。

It is one of the most widely used probability distributions in inferential statistics, notably in hypothesis testing or in the construction of confidence intervals.

它是推論統計中使用最廣泛的概率分布之一，特別是在假設檢驗或構造置信區間中。

The primary reason that the chi-square distribution is used extensively in hypothesis testing is its relationship to the normal distribution. An additional reason that the chi-square distribution is widely used is that it is a member of the class of likelihood ratio tests (LRT).LRT’s have several desirable properties; in particular, LRT’s commonly provide the highest power to reject the null hypothesis.

在假設檢驗中廣泛使用卡方分布的主要原因是其與正態分布的關系。 卡方分布被廣泛使用的另一個原因是它是似然比檢驗(LRT)類的成員。 特別是LRT通常會提供最高的能力來拒絕原假設。

卡方分布的自由度： (Degree Of Freedom in Chi-Squared Distribution:)

The degrees of freedom in Chi-Squared distribution is equal to the number of standard normal deviates being summed. The mean of a Chi-square distribution is its degrees of freedom. A chi-square distribution constructed by squaring a single standard normal distribution is said to have 1 degree of freedom

卡方分布的自由度等于要求和的標準正態偏差的數量。 卡方分布的平均值是其自由度。 通過平方單個標準正態分布構造的卡方分布據說具有1個自由度

The degrees of freedom ( df or d) tell you how many numbers in your grid are actually independent. For a Chi-square grid, the degrees of freedom can be said to be the number of cells you need to fill in before, given the totals in the margins, you can fill in the rest of the grid using a formula.

自由度 ( df或d )告訴您網格中實際上有多少個獨立的數字。 對于卡方網格，自由度可以說是您之前需要填充的像元數，給定邊距的總數，則可以使用公式來填充網格的其余部分。

The degrees of freedom for a Chi-square grid is equal to the number of rows minus one times the number of columns minus one: that is, (R-1)*(C-1).

卡方網格的自由度等于行數減去列數減去一的一倍，即(R-1)*(C-1)。

Remember!

記得！

As the degree of freedom (df), increases the Chi-square distribution approaches a normal distribution

隨著自由度(df)的增加，卡方分布接近正態分布

卡方統計： (Chi-Square Statistic:)

The formula for the chi-square statistic used in the chi-square test is:

卡方檢驗中使用的卡方統計量公式為：

The subscript “c” here are the degrees of freedom. “O” is your observed value and E is your expected value. The summation symbol means that you’ll have to perform a calculation for every single data item in your data set.

下標“ c ”是自由度。 “ O ”是您的觀測值， E是您的期望值。 求和符號表示您必須對數據集中的每個數據項執行計算。

E=(row total×column total) / sample size

E =(行總數×列總數)/樣本量

The Chi-square statistic can only be used on the numbers. They can’t be used for percentages, proportions, means, or similar statistical value. For example, if you have 10 percent of 200 people, you would need to convert that to a number (20) before you can run a test statistic.

卡方統計只能用于數字。 它們不能用于百分比，比例，均值或類似的統計值。 例如，如果您有200個人中的10％，則需要先將其轉換為數字(20)，然后才能運行測試統計信息。

Chi-Square test involves calculating a metric called the Chi-square statistic mentioned above, which follows the Chi-square distribution.

卡方檢驗涉及計算遵循卡方分布的上述度量(稱為卡方統計量)。

Let’s see an example to get clarity on all the above-covered topics related to Chi-Square:

讓我們看一個例子，以使上面提到的所有與卡方相關的主題變得清晰：

P-Value:

P值：

The null hypothesis provides a probability framework against which to compare our data. Specifically, through the proposed statistical model, the null hypothesis can be represented by a probability distribution called P-value, which gives the probability of all possible outcomes if the null hypothesis is true;

零假設為比較我們的數據提供了一個概率框架。 具體來說，通過提出的統計模型，可以用稱為P值的概率分布表示零假設，如果零假設為真，則給出所有可能結果的概率。

It is a probabilistic representation of our expectations under the null hypothesis.

它是原假設下我們期望的概率表示。

Chi-Square Test Explained With Example:

卡方檢驗舉例說明：

We will cover the following important steps in our journey of the Chi_square test for Independence of two variables.

我們將在卡方檢驗的兩個變量獨立性測試過程中涵蓋以下重要步驟。

• State The Hypothesis

  陳述假設

• Formulate Data Analysis Plan

  制定數據分析計劃

• Analyze The Sample Data

  分析樣本數據

• Interpret The Outcome

  解釋結果

Problem: This problem has been sourced from starttrek

問題：此問題來自 starttrek

A public opinion poll surveyed a simple random sample of 1000 voters. Respondents were classified by gender (male or female) and by voting preference (Republican, Democrat, or Independent). The results are shown in the contingency table below.

一項民意調查對1000名選民進行了簡單隨機抽樣調查。 按性別(男性或女性)和投票偏好(共和黨，民主黨或獨立人士)對受訪者進行分類。 結果顯示在下面的列聯表中。

We have to infer, Is there a gender gap? Do the men’s voting preferences differ significantly from women’s preferences? Use a 0.05 level of significance.

我們必須推斷，是否存在性別差距？ 男性的投票偏好與女性的偏好有很大不同嗎？ 使用0.05的顯著性水平。

Let’s try to solve this problem using the Chi-Square test to find out the P-Value.

讓我們嘗試使用卡方檢驗來解決此問題，以找出P值。

Here test type which we will employ is :

我們將采用的測試類型為：

卡方檢驗是否具有獨立性。 (Chi-square test for independence.)

So let’s get started by first stating our hypothesis.

因此，讓我們從首先說明我們的假設開始。

Step 1: State The Hypothesis:

步驟1：陳述假設：

Here we need to start by establishing a null hypothesis and counter hypothesis(alternative hypothesis) as given below.

在這里，我們需要先建立一個零假設和反假設(替代假設)，如下所示。

Null Hypothesis:

零假設：

Ho: Gender and voting preferences are independent.

何：性別和投票偏好是獨立的。

Alternate Hypothesis:

替代假設：

H1: Gender and voting preferences are not independent.

假設1：性別和投票偏好不是獨立的。

Step 2: Let’s Build Our Data Analysis Plan :

步驟2：讓我們建立資料分析計劃：

Here we will try to find out P-Value and match it with the significance level. Let’s take the standard and accepted level of significance to be 0.05. Given the sample data in the table above, let’s try to employ Chi-Square test for independence and deduce the Probability value.

在這里，我們將嘗試找出P值并將其與顯著性水平匹配。 讓我們將標準和可接受的顯著性水平設為0.05。 給定上表中的樣本數據，讓我們嘗試采用卡方檢驗進行獨立性分析并推論出概率值。

Step 3: Let’s Do Sample Analysis:

步驟3：讓我們做樣本分析：

Here we will analyze the given sample data to compute

在這里，我們將分析給定的樣本數據以進行計算

• Degree of freedom

  自由度

• Expected Frequency Count of sample variable

  樣本變量的預期頻率計數

• Calculate Chi-Square test static value

  計算卡方檢驗靜態值

All the above values will help us find the P-value.

以上所有值將幫助我們找到P值 。

Degree Of Freedom Calculation: Let’s calculate df = (r — 1) * (c — 1), so in the given table, we have r(rows)= 2 and c(column) = 3

自由度計算：讓我們計算df =(r_1)*(c_1)，因此在給定的表中，我們的r(rows)= 2和c(column)= 3

df= (2–1)*(3–1) = 1*2= 2 ;

df =(2-1)*(3-1)= 1 * 2 = 2;

Expected Frequency Count Calculation:

預期的頻率計數計算：

Let Eij, represent expected values of the two variables are independent of one another.

令Eij代表兩個變量的期望值彼此獨立。

Eij = ith (row total X jth column total) / grand total

Eij = ith(行總數x第j列總數)/總計

Let’s calculate the expected value for each given row and column value by using the above mentioned formula, Let me copy the table image again below to help you make calculation easily,

讓我們使用上述公式為每個給定的行和列值計算期望值，讓我在下面再次復制表格圖片以幫助您輕松進行計算，

Here, Row 1 total value = 400, total value for column1 = 450, total sample size = 1000,

在這里，第1行的總值= 400，第1列的總值= 450，樣本總數= 1000，

So,

所以，

E1,1 = (400 * 450) / 1000 = 180000/1000 = 180

E1,1 =(400 * 450)/ 1000 = 180000/1000 = 180

Similarly, let's calculate other expected values as shown below,

同樣，讓我們??計算其他期望值，如下所示，

E1,2 = (400 * 450) / 1000 = 180000/1000 = 180E1,3 = (400 * 100) / 1000 = 40000/1000 = 40E2,1 = (600 * 450) / 1000 = 270000/1000 = 270E2,2 = (600 * 450) / 1000 = 270000/1000 = 270E2,3 = (600 * 100) / 1000 = 60000/1000 = 60

E1,2 =(400 * 450)/ 1000 = 180000/1000 = 180E1,3 =(400 * 100)/ 1000 = 40000/1000 = 40E2,1 =(600 * 450)/ 1000 = 270000/1000 = 270E2， 2 =(600 * 450)/ 1000 = 270000/1000 = 270E2,3 =(600 * 100)/ 1000 = 60000/1000 = 60

Time to calculate Chi-Squares for each calculated expected values above using the formula:

是時候使用以下公式為上述每個計算出的期望值計算Chi-Squares：

Calculating Chi-Square:

計算卡方：

As already discussed above, the formula for calculating chi-square statistic is

如上所述，用于計算卡方統計量的公式為

The subscript “c” here are the degrees of freedom. “O” is your observed value (actual values given in the table above)and E is your expected value(which we just calculated). The summation symbol means that you’ll have to perform a calculation for every single data item in your data set.

下標“ c ”是自由度。 “ O ”是您的觀測值(上表中給出的實際值)，而E是您的期望值(我們剛剛計算出)。 求和符號表示您必須對數據集中的每個數據項執行計算。

Χ2 = Σ [ (Oi,j — Ei,j)2 / Ei,j ]

Χ2=Σ[(Oi，j-Ei，j)2/ Ei，j]

Using the above formula our chi-square values comes out to be as given below,

使用上述公式，我們得出的卡方值如下所示：

Χ2 = (200–180)2/180 + (150–180)2/180 + (50–40)2/40 + (250–270)2/270 + (300–270)2/270 + (50–60)2/60Χ2 = 400/180 + 900/180 + 100/40 + 400/270 + 900/270 + 100/60

Χ2=(200–180)2/ 180 +(150–180)2/ 180 +(50–40)2/ 40 +(250–270)2/ 270 +(300–270)2/ 270 +(50– 60)2/60Χ2= 400/180 + 900/180 + 100/40 + 400/270 + 900/270 + 100/60

So our final chi-square statistic value ,

因此，我們的最終卡方統計值，

Χ2 = 2.22 + 5.00 + 2.50 + 1.48 + 3.33 + 1.67 = 16.2

Χ2= 2.22 + 5.00 + 2.50 + 1.48 + 3.33 + 1.67 = 16.2

Having calculated the chi-square value and degrees of freedom, we consult a chi-square table to check whether the chi-square statistic of 16.2 exceeds the critical value for the Chi-square distribution. The intent is to find P-value, which is is the probability that a chi-square statistic having 2 degrees of freedom is more extreme than 16.2.

計算出卡方值和自由度后，我們查閱卡方表以檢查16.2的卡方統計量是否超過卡方分布的臨界值。 目的是找到P值，這是具有2個自由度的卡方統計量比16.2極端的概率。

How to calculate P-value?

如何計算P值？

Given the degree of freedom = 2 & Chi-square statistic value = 16.2 , we can easily find P-value using this given

給定自由度= 2和卡方統計值= 16.2，我們可以使用給定的值輕松找到P值

Chi-Square Calculator link, simply enter the Chi-square statistic value & degree of freedom as an input, also keep your significance level as 0.05, you will find the result as given below,

卡方計算器鏈接，只需輸入卡方統計值和自由度作為輸入，并將您的顯著性水平保持為0.05，您將發現以下結果，

P-Value is =. 000304. The result is significant at p < .05.

P值為=。 000304。結果在p <.05時很顯著。

You can also find P-value using Chi-Square table given below, you can get this table from this source

您還可以使用下面給出的卡方表找到P值，您可以從此來源獲取此表

Having calculated the chi-square value to be 16.2 and degrees of freedom to be 2, we consult a chi-square table given above to check whether the chi-square statistic of 16.2 exceeds the critical value for the Chi-square distribution. The critical value for the alpha of .05 (95% confidence) for df=2 comes out to be 5.99

計算卡方值為16.2，自由度為2后，我們查閱上面給出的卡方表，檢查卡方統計量16.2是否超過卡方分布的臨界值。 df = 2的.05(95％置信度)的alpha的臨界值得出為5.99

Step 4: Interpreting the result

步驟4：解釋結果

A: Inference From The P-value:

答：從P值推斷：

Since we have got the P-value of 0.000304 we can interpret the result where it signifies that

由于我們獲得了0.000304的P值，因此我們可以在表示該值的地方解釋結果

As the P-value (0.000304) is less than the significance level (0.05),

由于P值(0.000304)小于顯著性水平(0.05)，

So we have to reject the below given

所以我們必須拒絕以下給出的

Null Hypothesis, which says, gender and voting preferences are independent.

零假設說， 性別和投票偏好是獨立的。

& accept Alternate Hypothesis:

并接受替代假設 ：

Which says, gender and voting preferences are not independent.

也就是說，性別和投票偏好不是獨立的。

Hence we can conclude that,

因此，我們可以得出結論，

There is a relationship between gender and voting preference.

性別與投票偏好之間存在聯系。

B：從卡方表解釋： (B: Interpreting from Chi-Square Table:)

Since the critical value for the alpha of .05 (95% confidence) for df=2 is 5.99 and our chi-square statistic value 16.3, is much larger than 5.99, we have sufficient evidence to reject our Null hypothesis which we covered above.

由于df = 2的.05的alpha的臨界值(95％的置信度)為5.99，而我們的卡方統計值16.3遠大于5.99，因此我們有足夠的證據拒絕我們上面討論的Null假設。

So we accept the Alternate Hypothesis:

因此，我們接受替代假設：

Which says, gender and voting preferences are not independent.

也就是說，性別和投票偏好不是獨立的。

Hence we conclude that,

因此，我們得出結論，

There is a relationship between gender and voting preference.

性別與投票偏好之間存在聯系。

下一步是什么？ (What’s Next?)

We will understand how to perform Chi-Square test using python & Jupyter notebook in part 2 of this series of Inferential Statistic: Hypothesis testing Using Chi-Square and will further explore

在本系列推論統計：使用卡方假設檢驗的系列文章的第2部分中，我們將了解如何使用python和Jupyter筆記本執行卡方檢驗。

• Normal Deviate Z Test:

  正常偏差Z測試：

• Two-Sample T-Test

  兩樣本T檢驗

• ANOVA Test

  方差檢驗

& also will introduce one of the key topic: “Power of Statistical Test “

＆還將介紹關鍵主題之一：“ 統計檢驗的力量”

The power of any test of statistical significance is defined as the probability that it will reject a false null hypothesis.

任何具有統計意義的檢驗的功效被定義為它將拒絕錯誤的虛假假設的概率。

總結這一部分，并提供一個非常有用的信息圖，它指導您選擇假設檢驗類型： (Summing up this part, with a very helpful infographic which guides you to choose your hypothesis test type:)

So choose your test data wisely and make sure you are interpreting sample data right, so that you can go ahead to design your ML models with required accuracy & confidence.

因此，請明智地選擇測試數據，并確保您正確解釋了樣本數據，以便您可以按要求的準確性和信心繼續設計ML模型。

Your ability to be an effective data scientist will largely become a reality only & only if you know how to analyze the given sample data with minimum deviation. The more you treat data with the required precision and clean them in the preliminary stage of EDA, the more reliable and productive your model building effort will become.

只有當您知道如何以最小的偏差分析給定的樣本數據時，您成為有效的數據科學家的能力才會在很大程度上變成現實。 在EDA的初期階段，您越以所需的精度處理數據并清理數據，建模工作就會變得更加可靠和高效。