In predictive analytics, it can be a tricky thing to know when to stop.

在預測分析中，知道何時停止可能是一件棘手的事情。

Unlike many of life’s activities, there’s no definitive finishing line, after which you can say “tick, I’m done”. The possibility always remains that a little more work can yield an improvement to your model. With so many variables to tweak, it’s easy to end up obsessing over tenths of a percentage point, pouring huge amounts of effort into the details before looking up and wondering “Where did the time go?”.

與生活中的許多活動不同，沒有明確的終點線，之后您可以說“打勾，我完成了”。 總是有可能更多的工作可以改進您的模型。 由于要調整的變量太多，因此很容易最終迷上一個十分之一的百分點，在查尋細節之前投入了大量的精力，并想知道“時間花在哪里？”。

Iterating your model, via feature engineering, model selection and hyper-parameter tuning is a key skill of any data scientist. But knowing when to stop is something that rarely gets addressed, and can vastly alter the cost of model development and the ROI of a Data Science project.

通過特征工程，模型選擇和超參數調整來迭代模型是任何數據科學家的一項關鍵技能。 但是，知道何時停止是很少解決的事情，并且可以極大地改變模型開發的成本和數據科學項目的投資回報率。

I’m not talking about over vs under fitting here. Over-fitting is where your model is too closely fit to your training data and can be detected by comparing the training set error with a validation set error. There are many great tutorials on Medium and elsewhere which explain all this in much more detail.

我在這里談論的不是過度與不足。 過度擬合是模型過于適合訓練數據的地方，可以通過將訓練集誤差與驗證集誤差進行比較來檢測出。 在Medium和其他地方，有很多 很棒的 教程 ，它們對這一切進行了更詳細的解釋。

I’m referring to the time you spend working on the entire modelling pipeline, and how you quantify the rewards and justify the cost.

我指的是您花費在整個建模流程上的時間，以及如何量化收益并證明成本合理。

策略 (Strategies)

Some strategies that can help you decide when to wrap things up might be:

可以幫助您決定何時打包的一些策略可能是：

• Set a deadline — Parkinson’s law states that “work expands so as to fill the time available for its completion”. Having an open ended time-frame invites you to procrastinate by spending time on things that ultimately don’t provide much value to the end result. Setting yourself a deadline is a good way of keeping costs low and predictable by forcing you to prioritise effectively. The down-side is of course that if you set your deadline too aggressively, you may deliver a model that is of poor quality.

  設定最后期限-帕金森法則指出：“工作在擴大，以填補完成工作所需的時間”。 有一個開放式的時間框架會邀請您拖延時間，最終花費一些時間最終無法為最終結果提供太多價值。 為自己設定一個截止日期是一種有效的方法，它可以迫使您有效地確定優先級，從而將成本保持在較低水平且可預測。 不利的一面當然是，如果您過分地設置截止日期，則可能會提供質量較差的模型。

• Acceptable error rate — You could decide beforehand on an acceptable error rate and stop once you reach it. For example, a self-driving car might try to identify cyclists with a 99.99% level of accuracy. The difficulty of this approach is that before you start experimenting, it’s very hard to set expectations as to how accurate your model could be. Your desired accuracy rate might be impossible, given the level of irreducible error. On the other hand, you might stop prematurely whilst there is still room to easily improve your model.

  可接受的錯誤率-您可以預先確定可接受的錯誤率，并在達到錯誤率時停止。 例如，自動駕駛汽車可能會嘗試以99.99％的準確度識別騎自行車的人。 這種方法的困難在于，在您開始實驗之前，很難就模型的精確度設定期望。 鑒于無法減少的誤差水平，可能無法達到理想的準確率。 另一方面，您可能仍會過早停止，而仍有足夠的空間輕松改善模型。

• Value gradient method — By plotting the real-world cost of error in your model, vs the effort required to enhance it, you gain an understanding of what the return on investment is for each incremental improvement. This allows you to keep developing your model, only stopping when the predicted value of additional tuning fall below the value of your time.

  值梯度法-通過繪制模型中的實際錯誤成本與增強誤差所需的工作量，您可以了解每次增量改進的投資回報率。 這使您可以繼續開發模型，僅在其他調整的預測值低于您的時間值時才停止。

收益遞減法則 (The law of diminishing returns)

As you invest time into tweaking your model, you may find that your progress is fast in the beginning, but quickly plateaus. You’ll likely perform the most obvious improvements first, but as time goes by you’ll end up working harder and harder for smaller gains. Within the data itself, the balance between reducible and irreducible error puts an upper limit on the level of accuracy that your model can achieve.

當您花時間調整模型時，您可能會發現開始時進展很快，但很快就達到了平穩狀態。 您可能會首先執行最明顯的改進，但是隨著時間的流逝，您將越來越努力地爭取較小的收益。 在數據本身內，可減少的誤差與不可減少的誤差之間的平衡為模型可達到的精度水平設置了上限。

In a learning exercise or a Kaggle competition, you can iterate to your heart’s content, chasing those incremental improvements further and further down the decimal places.

在學習練習或Kaggle比賽中，您可以迭代自己的內心內容，將這些遞增的改進逐個追逐。

However, for a commercial project, the cost of tuning this model climbs linearly with respect to the amount of time you have invested. This means there comes a point where scraping out an extra 0.1% will not be worth the investment.

但是，對于商業項目，調整此模型的成本相對于您投入的時間呈線性增長。 這意味著有些時候刮掉額外的0.1％將不值得投資。

This varies from project to project. If you’re working with supermarket data, given the huge number of purchases on a daily basis, an additional hundredth of a percentage point of accuracy might be worth a lot of money. This puts a strong ROI on continuing efforts to improve your model. But for projects of more modest scale, you might have to draw the line a bit sooner.

這因項目而異。 如果您使用超市數據，由于每天都有大量購買，那么，精確度提高百分之一百分之百，可能是很多錢。 這為持續改進模型提供了可觀的投資回報率。 但是，對于規模較小的項目，您可能需要盡快畫線。

模型錯誤的實際成本 (The real-world cost of model error)

When tuning a model, the values you’re likely to be paying attention to are statistical in nature. MSE, % accuracy, R2 and AIC are defined by their mathematical formulae, and are indifferent to the real-world problem you’re attempting to solve.

調整模型時，您可能需要注意的值實際上是統計值。 MSE，％精度，R2和AIC由它們的數學公式定義，并且與您要解決的實際問題無關。

Rather than solely considering statistical measures of accuracy and error, these should be converted into something that can be weighed against the time investment you’re making, i.e. money.

不應只考慮準確性和錯誤的統計指標，而應將這些指標轉換為可以與您所花費的時間(即金錢)相權衡的指標。

Let’s say we run an ice-cream kiosk, and we’re trying to predict how many ice-creams we’ll sell on a daily basis, using variables like the weather, day of week, time of year etc.

假設我們經營一個冰淇淋亭，并嘗試使用天氣，星期幾，一年中的時間等變量來預測每天售出多少個冰淇淋。

No model we create will be perfect, and for any given day it will usually either;

我們創建的任何模型都不完美，并且在任何給定的一天通常都不會完美。

• overestimate — meaning we buy more ingredients than we need for the number of ice-creams sold.
  高估了-這意味著我們購買的食材比冰淇淋數量要多。
• underestimate — meaning we run out of stock and lose out on potential business.
  低估了-這意味著我們缺貨而失去了潛在業務。

Both of these types of error introduce a monetary cost to the business. If we run out of stock at midday, we’ve lost the margin on half a day’s sales. And if we overestimate, we may end up spending money on ingredients that end up being thrown away.

這兩種類型的錯誤都會給企業帶來金錢上的損失。 如果我們在中午缺貨，那么我們半天的銷售利潤就會損失。 而且，如果我們高估了價值，我們最終可能會花錢購買最終被扔掉的食材。

We can introduce business rules on top of our model to help reduce some of this loss. The cost of losing 1 ice-cream’s worth of sales is likely higher than the cost of throwing away 1 ice-cream’s worth of out-of-date milk (given we’re hopefully making a profit). Therefore, we’ll want to be biased in favour of over-stocking, for example by holding 20% more ingredients than suggested by the model’s prediction. This will greatly reduce the frequency and cost of stock outages, at the expense of having to throw out a few bottles of out-of-date milk.

我們可以在模型之上引入業務規則，以幫助減少部分損失。 損失1杯冰淇淋的銷售成本可能會比丟掉1杯冰淇淋的過期牛奶的成本高(假設我們希望獲利)。 因此，我們希望偏向于庫存過多，例如，持有比模型預測所建議的多20％的成分。 這將大大減少斷貨的頻率和成本，但以不得不扔掉幾瓶過期牛奶為代價。

Optimising this 20% rule, falls under the umbrella of Prescriptive Analytics. Using the training set, we can tweak this rule, until the average estimated real-world cost of the error in the model is at its lowest.

優化此20％規則屬于Prescriptive Analytics的保護范圍 。 使用訓練集，我們可以調整此規則，直到模型中錯誤的平均估計實際成本達到最低。

值梯度法 (The value gradient method)

Now that we have an estimated real-world cost for the accuracy of the model, we gain an idea of what the time we’re investing in the model is worth. With each iteration, we subtract the real-world cost from that of the previous version, to work out the value added by our extra effort. From there, we can extrapolate to a window of ROI.

現在，我們已經為模型的準確性估算了實際成本，現在我們了解了在模型上投資的時間是值得的。 在每次迭代中，我們都從上一版本中減去實際成本，以計算出我們付出的額外努力所帶來的價值。 從那里，我們可以推斷出一個投資回報率窗口。

For example, your validation set may contain 1,000 rows and your latest model saved $40 vs the previous iteration. If you are expecting to collect 100,000 data-points per year, then you can multiply the added value by 100 to get an annual rate. Therefore, the work you put in to produce the latest version of the model gives a return of $4,000 per year.

例如，您的驗證集可能包含1,000行，而您的最新模型與先前的迭代相比節省了40美元。 如果您希望每年收集100,000個數據點，則可以將增加值乘以100以獲得年費率。 因此，您投入到生產最新版本模型的工作中，每年可得到4,000美元的回報。

Comparing this to the cost of our time gives us an expected return on investment. E.g. if the above enhancement required a day’s work for someone earning $400 per day, it pays for itself very quickly.

將其與我們的時間成本進行比較，可以為我們帶來預期的投資回報。 例如，如果上述改進要求每天賺取$ 400的某人一天的工作，它會很快收回成本。

However, as the law of diminishing returns eats away at our rate of improvement, our margin will begin to fall. When it approaches zero, it’s time to take what we have and move on to the next stage in our project.

但是，隨著收益遞減法則的吞噬，我們的利潤率將開始下降。 當它接近零時，是時候利用我們所擁有的，并進入項目的下一個階段。

Of course, this is an inexact science. It assumes that improvements to our model will occur in a smooth and predictable way and that future gains will smaller than previous improvements. Whenever you call it a day, there will be the possibility that a significant breakthrough lies just around the corner.

當然，這是一門不精確的科學。 它假設對我們模型的改進將以一種平滑且可預測的方式進行，并且未來的收益將小于以前的改進。 每當您將其命名為“一天”時，都有可能出現重大突破。

But it’s always a good idea to keep a commercial eye on the time you’re investing in a model, allowing you to do more valuable work by keeping costs down and freeing up time to spend on the most important things.

但是，始終保持商業眼光投資模型的時間始終是一個好主意，通過降低成本并騰出時間花在最重要的事情上，從而使您能夠做更多有價值的工作。

Coming soon: A Python module which takes the predictions, actual values and a cost-function and outputs the expected ROI for any model — allowing you to integrate the above decision making into your model tuning process.

即將推出：一個Python模塊，它將獲取預測，實際值和成本函數，并輸出任何模型的預期ROI-使您可以將上述決策整合到模型調整過程中。