成功的機器學習/人工智能方法 (Methods for successful Machine learning / Artificial Intelligence)

It’s widely stated that data is the new oil, and like oil, data needs the right refinement to evolve to be utilised perfectly. The power of machine learning models will significantly depend on the quality of the data; I’m not saying anything new here.

人們普遍認為，數據是新的石油，就像石油一樣，數據需要進行適當的精煉才能發展以得到完美利用。 機器學習模型的功能將在很大程度上取決于數據的質量。 我不是在這里說新的話。

As AI development and its subsequent applications become even more pervasive, ML engineers everywhere are confronted with a grim reality. Once stakeholders overcome biases or skepticisms and finally buy-in, identify a use case with proven ROI, and now are eager to jump onto the AI ship, data curation is usually neglected and suffers from not attracting its due importance — often due to a quick win mentality and the fact it’s not sexy!

隨著AI開發及其后續應用變得越來越普遍，各地的ML工程師都面臨嚴峻的現實。 一旦利益相關者克服了偏見或懷疑并最終接受了投資，確定了具有良好ROI的用例，現在又急于跳入AI船上，數據管理通常會被忽略，并且由于無法快速獲得數據，因此無法發揮應有的重要性。贏得心態和事實，那就不是性感！

There are many assumptions even within technology groups, that AI only needs to be fed data collected and combined on a large measure; in most cases, this gravely backfires. Inaccurate datasets can come in many forms ranging from factually incorrect information to knowledge gaps to wrong guidelines. Among many other problems, an uncurated dataset can be:

即使在技術小組內部，也有許多假設，即只需要向AI提供大量收集和合并的數據即可。 在大多數情況下，這會適得其反。 不準確的數據集可能以多種形式出現，從事實不正確的信息到知識鴻溝再到錯誤的準則。 除許多其他問題外，未整理的數據集可能是：

• Biased: recently, several popular AI’s used for image recognition displayed disturbing gender and racial bias.

  偏見：最近，幾種流行的用于圖像識別的AI顯示出令人不安的性別和種族偏見。

• Inaccurate, unreliable or falsely represented

  不準確，不可靠或虛假陳述

• Error-ridden or ambiguous

  錯誤纏身或模棱兩可

The lack of using refined or curated raw datasets are universally known to decrease feature quality and limit the evaluation and applications of transfer tasks. So how should datasets be treated in a way that they serve the exact purpose ML needs to work, this is highly dependant on the use cases the ML engineers are trying to address.

眾所周知，缺乏使用精煉或精選原始數據集會降低要素質量并限制傳輸任務的評估和應用。 因此，應如何以滿足ML工作所需確切目的的方式對待數據集，這在很大程度上取決于ML工程師試圖解決的用例。

機器學習的數據集類型 (Types of Datasets for Machine Learning)

ML engineers depend on data throughout each step of their AI journey — from model choice, training, and testing. These datasets typically fall under three classifications:

機器學習工程師在AI歷程的每個步驟中都依賴于數據，包括模型選擇，培訓和測試。 這些數據集通常分為三類：

• Training sets

  訓練套

• Validation sets

  驗證集

• Testing sets.

  測試裝置。

Every ML project starts with two data set categories; the training data set and the testing data set.

每個ML項目都以兩個數據集類別開始； 訓練數據集和測試數據集。

• The training data set is used to train an algorithm, implement concepts, discover, and give results.
  訓練數據集用于訓練算法，實現概念，發現并給出結果。
• Testing data is used to examine the validity of the training data set. Training data is not used for testing because it will produce expected outputs.
  測試數據用于檢查訓練數據集的有效性。 訓練數據不用于測試，因為它將產生預期的輸出。

機器學習的數據需求 (Data needs for Machine Learning)

Data scientists collect data from various sources, integrate it into one form, validate, manipulate, archive, preserve, retrieve, and express it.

數據科學家從各種來源收集數據，將其集成為一種形式，然后進行驗證，操作，存檔，保存，檢索和表達。

The process of curating datasets for machine learning starts well before availing datasets.

整理用于機器學習的數據集的過程在使用數據集之前就已經開始了。

My suggestion:

我的建議：

• Identify the aim of the AI

  確定AI的目標

• Identify what dataset you will require to solve the problem

  確定解決問題所需的數據集

• Create a record of your hypotheses while selecting the Data

  選擇數據時創建假設記錄

• Strive for collecting assorted and meaningful data from both external and internal sources

  努力從外部和內部來源收集各種有意義的數據

• Create datasets that are hard for your competitors to copy (defendability)

  創建難以被競爭對手復制的數據集(可防御性)

If you have a small dataset, applying a model pre-trained on large datasets can be a great approach and use your small dataset to fine-tune.

如果您的數據集較小，則對大型數據集應用預訓練的模型可能是一種不錯的方法，并使用小型數據集進行微調。

Once you have accumulated the correct Data, you can progress with creating the training set. This step of putting data in the optimal format is called feature transformation, and it involves four stages:

一旦積累了正確的數據，就可以繼續創建訓練集。 將數據以最佳格式放置的這一步驟稱為特征轉換，它涉及四個階段：

Formatting: Data discovery is in different formats. Formatting will bring it together in one sheet. For example, consumer Data can come with different currencies, semantics and so on. These need to be compiled under one format for foundation uniformity.

格式：數據發現采用不同的格式。 格式化會將其合并到一張紙中。 例如，消費者數據可以帶有不同的幣種，語義等。 這些需要以一種格式進行編譯以實現基礎均勻性。

Labelling: Labelling ensures the Data set works for the specific model choice. For example, an autonomous car requires data labelled as images of cars, pedestrians, road signs, walkways.

貼標簽：貼標簽可確保數據集適用于特定的模型選擇。 例如，自動駕駛汽車需要標記為汽車，行人，道路標志，人行道圖像的數據。

Cleansing: Suboptimal characters need to be removed, and missing values are managed based on the weighting of need.

清理：需要刪除次優字符，并根據需要的權重來管理缺失值。

Extraction: Several features are examined and optimised — features that are essential for predictive capability and faster computation and less memory consumption.

提取：已檢查和優化了幾個功能-這些功能對于預測功能，更快的計算和更少的內存消耗至關重要。

底線 (The Bottom Line)

A dataset solely can ensure the success or failure of a machine learning model. Data curation is one of the fundamental aspects of machine learning, and if exercised correctly, it can unleash tremendous potential. The methods and subsequent processes can appear time-consuming; however, this will guarantee your dataset’s calibration with the goals of your machine learning at each step.

數據集僅可以確保機器學習模型的成功或失敗。 數據管理是機器學習的基本方面之一，如果正確執行，它可以釋放巨大的潛力。 方法和后續過程可能很耗時。 但是，這將確保您的數據集的校準符合每一步的機器學習目標。

Introducing data curation processes into your data team and the following procedures will appear time-consuming and expensive in the short term; therefore, organisations must carefully analyse current objectives and develop a strategy to support the relevance for curation-as-a-function. Managed services and Unsupervised methods trained on curated data are available and marketed by advisory and technology firms, be careful and choose carefully; this will play a key role in your AI future.

在您的數據團隊中引入數據管理流程，以下過程在短期內將顯得既耗時又昂貴。 因此，組織必須仔細分析當前的目標并制定策略，以支持與策展即功能有關。 咨詢和技術公司可以使用托管的服務和不受監管的方法進行策劃的數據培訓，并且要謹慎行事并謹慎選擇； 這將在您的AI未來中發揮關鍵作用。