By Stuart George, Executive Director of Design Technology at Method

Method設計技術執行總監Stuart George

When Andrew Mason, founder of Groupon, wanted to improve his email conversion metrics, he turned to data analysis. His team tested the impact of sending two emails per day instead of one, and found that, while more double-emailed customers tended to unsubscribe, the ones who stayed generated more revenue. Ignoring his intuition, he had his team switch to the two-a-day model.

Groupon的創始人安德魯·梅森(Andrew Mason)想改善電子郵件轉換指標時，便轉向數據分析。 他的團隊測試了每天發送兩封電子郵件而不是每天發送一封電子郵件的影響，并發現，盡管有更多重復發送電子郵件的客戶傾向于退訂，但留下的客戶卻產生了更多的收入。 無視他的直覺，他讓他的團隊改用每天兩次的模型。

This was not a good decision. Whereas there is no doubt that the Groupon data scientists achieved statistically significant results, they failed to consider the long-term effects of the change. Groupon became little more than a “marketplace of coupons,” Mason admits, eventually burning through the revenue potential of their dwindling market.

這不是一個好決定。 毫無疑問，Groupon數據科學家取得了統計上顯著的結果，但他們沒有考慮這一變化的長期影響。 梅森承認 ，Groupon僅僅是一個“優惠券市場”，最終燒毀了其日益萎縮的市場的潛在收入。

From this example, it would be easy to conclude that data-driven decision making is more trouble than it’s worth.

從這個例子中，很容易得出結論，數據驅動的決策比其價值更大。

Putting your faith in statistics and modeling alone can be quite risky, as data-driven analytics and insights are very prone to “crimes” of malpractice and misuse. But equally, an over-reliance on intuition can lead to suboptimal decision making, especially within teams.

僅僅依靠統計和建模可能會帶來很大的風險，因為數據驅動的分析和見解很容易發生瀆職和濫用的“罪行”。 但同樣，過分依賴直覺會導致決策不理想，尤其是在團隊內部。

When data is expensive or difficult to source, many companies use qualitative frameworks to synthesize ideas and opinions. While these can help simplify incredibly complex information, they often rely too heavily on intuition and fail to immunize teams from the perils of human bias.

當數據昂貴或難以獲取時，許多公司使用定性框架來綜合思想和觀點。 盡管這些可以幫助簡化極其復雜的信息，但它們通常過于依賴直覺，無法使團隊免受人為偏見的危害。

And we humans have a lot of biases to be wary of. Confirmation bias, availability bias, or representativeness bias, to name a few, can help us simplify decision making based on past experience — but this can often mean that we judge incorrectly. All these biases compound with other social biases in group decision making, creating minefields of judgement errors for project teams.

我們人類有很多偏見需要警惕。 確認偏見，可用性偏見或代表性偏見等可以幫助我們根據過去的經驗簡化決策，但這通常意味著我們判斷不正確。 所有這些偏見與集體決策中的其他社會偏見加在一起，為項目團隊創造了判斷錯誤的雷區。

So how might we balance intuition and data to make better group decisions? At Method we have a method. We call it Fact-based Hypothesis Testing, and it’s a way to help us make better decisions from qual and quant evidence and remove the bias that occurs in these decisions. When evidence for a particular hypothesis is mainly subjective or subjective and qualitative, Fact-Based Hypothesis Testing can make rigorous statistical analysis possible. This is achieved by asking team members questions about how artifacts, evidence and data acquired during the project affect the likelihood of each hypothesis being true. These answers are then analyzed and combined using Bayesian statistics. The system creates an audit trail of how the group considered evidence during the course of the project and how the group’s opinions change through the course of a project.

那么，我們如何平衡直覺和數據來做出更好的團隊決策呢？ 在Method中，有一個方法。 我們稱其為“基于事實的假設檢驗” ，這是一種幫助我們根據合格和定量證據做出更好的決策并消除這些決策中出現偏見的方法。 當特定假設的證據主要是主觀的或主觀的和定性的時，基于事實的假設檢驗可以使進行嚴格的統計分析成為可能。 這是通過向團隊成員詢問有關在項目期間獲取的工件，證據和數據如何影響每個假設為真的可能性來實現的。 然后使用貝葉斯統計分析和組合這些答案。 該系統會創建審核跟蹤，以了解小組在項目過程中如何考慮證據以及小組的意見在項目過程中如何變化。

To illustrate how the system works, consider the development of an energy usage app called EcoWatch. Your team is trying to determine if the product is desirable to 25- to 34-year-old first-time homeowners by evaluating a number of pieces of evidence. You could frame the project as a test of two hypotheses:

為了說明該系統如何工作，請考慮開發一個名為EcoWatch的能源使用應用程序。 您的團隊正在通過評估許多證據來確定25至34歲的首次購房者是否需要該產品。 您可以將項目設計為兩個假設的檢驗：

“Hypothesis A: EcoWatch is desirable to 25–34 year old first-time homeowners” “Hypothesis B: EcoWatch is not desirable to 25–34 year old first-time homeowners”

“假設A：25-34歲的首次購房者需要EcoWatch”“假設B：25-34歲的首次購房者不希望使用EcoWatch”

First, you would define your team’s “prior probability of a hypothesis.” Ask each team member to evaluate how likely they believe each hypothesis is to be true in qualitative terms (on a scale of impossible to extremely likely). The system then converts the designer’s evaluations into probabilities which are combined to produce a group likelihood of a hypothesis being true before evaluating evidence.

首先，您要定義團隊的“假設的先驗概率”。 要求每個團隊成員以定性的方式評估他們認為每個假設真實的可能性(以不可能到極有可能的程度)。 然后，系統將設計者的評估轉換為概率，在評估證據之前，將這些概率合并以產生假設為真的組似然。

Then you would evaluate the evidence from the project — in this case, evidence may look like the results of a survey or the synthesis of a user test. For each piece of evidence, the system asks two questions:

然后，您將評估項目中的證據-在這種情況下，證據可能看起來像調查結果或用戶測試的綜合結果。 對于每個證據，系統都會提出兩個問題：

“If Hypothesis A were 100 percent true, how likely is it that you would see this evidence?”

“如果假設A為100％正確，那么您看到該證據的可能性有多大？”

“If Hypothesis B were 100 percent false, how likely is it that you would see this evidence?”

“如果假設B為100％錯誤，那么您看到該證據的可能性有多大？”

If I were a team member, I may be inclined to say that the answer to the first question is “likely” and the second is “unlikely.” Using the theory of Bayesian statistics, we can combine all the team member’s answers to produce a group answer that fairly represents the group’s collective beliefs. This process continues as new evidence emerges or is added to the system, creating an audit trail of the likelihood of each hypothesis over time. By the end of the project, not only do we have the group’s preferred conclusion but also a rigorous and systematic way of understanding how the team arrived at its decision.

如果我是團隊成員，我可能會傾向于說，第一個問題的答案是“可能”，第二個問題的答案是“不太可能”。 使用貝葉斯統計理論，我們可以將所有團隊成員的答案結合起來，以得出公平地代表團隊集體信念的團隊答案。 隨著新證據的出現或將新證據添加到系統中，此過程將繼續進行，從而創建每個假設隨時間變化的可能性的審計線索。 到項目結束時，我們不僅獲得了小組的首選結論，而且以一種嚴格而系統的方式來了解團隊是如何做出決定的。

The Fact-Based Hypothesis Testing framework has four key features:

基于事實的假設檢驗框架具有四個關鍵特征：

1. Independent: each person evaluates the relevance of evidence independently from all other designers, helping to mitigate the effect of team groupthink.

   獨立性：每個人都獨立于其他所有設計師評估證據的相關性，從而有助于減輕團隊集體思考的影響。

2. Anonymous: each person’s answers are kept secret from the rest of the group, meaning there can be no finger-pointing if a person’s opinion dissents from the group.

   匿名 ：每個人的答案都與小組其他成員保密，這意味著如果一個人的意見與小組不同，就不會有指責。

3. Rigorous: the team’s answers are combined using a statistical procedure that avoids some of the pitfalls of simple aggregation techniques such as pooling or averaging.

   嚴謹：使用統計程序組合團隊的答案，避免了簡單的匯總技術(如合并或平均)的一些陷阱。

4. Calibrated: if the team leader believes a systematic bias could be at play in the group’s decision making, they can create fake evidence that, if true, would strongly confirm one hypothesis over all others. If the team members don’t evaluate this evidence in an appropriate fashion, the team leader can highlight the discrepancy to their team and address the bias.

   已校準：如果團隊負責人認為系統的偏見可能會影響團隊的決策，則他們可以創建虛假證據，如果為真，將強有力地證實一個假設高于所有其他假設。 如果團隊成員沒有以適當的方式評估此證據，則團隊負責人可以突出顯示與團隊的差異并解決偏見。

If Andrew Mason of Groupon had evaluated his email decision with Fact-based Hypothesis Testing, he may have found that keeping with one email made sense to decrease customer churn. He would have been able to balance the data against his intuition, without feeling the need to choose one over the other. And he could have brought that decision to his shareholders with an audit trail, giving them a window into what hypotheses were considered, what evidence was evaluated, and how his team’s opinion of the hypotheses changed over time.

如果Groupon的Andrew Mason使用基于事實的假設測試評估了他的電子郵件決定，他可能會發現保留一封電子郵件可以減少客戶流失。 他本來可以根據自己的直覺來平衡數據，而無需感覺需要選擇一個。 而且他本可以通過審計跟蹤將該決定帶給其股東，讓他們有機會了解所考慮的假設，所評估的證據以及其團隊對假設的看法隨時間變化的方式。

* * *

* * *

This article was written by Stuart George and edited by Erin Peace. Illustration by Claire Lorman. To learn more about our process, or understand how your teams might use Fact-based Hypothesis Testing, please get in touch.

本文由Stuart George撰寫，由Erin Peace編輯。 克萊爾·洛曼(Claire Lorman)的插圖。 要了解有關我們流程的更多信息，或了解您的團隊如何使用基于事實的假設檢驗，請 聯系 。