The Essentials of Machine Learning Data Curation

ML engineers depend on data during each step of their AI journey – from model selection, training, and tuning to testing. These datasets usually fall under three categories:

1. Training sets
2. Testing sets
3. Validation sets

The training data set is used to train an algorithm, apply concepts, learn, and give results. Around 60 percent of data is training data.

Testing data is used to test the validity of the training data set. Training data is not used for testing because it will produce the expected output. The testing data set comprises of 20 percent of the total data.

Validation tests are used to identify and tune the ML model.

The process of curating datasets for machine learning starts well before availing datasets. Here’s what we suggest:

• Identify the goal of AI
• Identify what dataset you will need to solve the problem
• Make a record of your assumptions while selecting the data
• Aim for collecting diverse and meaningful data from both external and internal resources
• Build a dataset that is hard for your competitors to copy

If you have a small dataset, using a model pre-trained on large datasets can be a good idea. You can use your small dataset to fine-tune it.

1. Formatting: The data is spread in different formats. Formatting will bring it together in one sheet. For example, customer data can come with different currencies, languages, etc. These need to be compiled under one format.
2. Labeling: Labeling is done to ensure the data set works for your model. For example, a self-driving car will need data labeled as pictures of cars, pedestrians, street signs, footpaths etc.
3. Data Cleaning: Unwanted characters are removed and missing values are dealt with.
4. Feature extraction: A number of features are analyzed and optimized. Features that are important for prediction are selected for quicker computation and less memory consumption.