So you want to teach computers

Here’s where to look for a dataset:

• Dataset search engines: kaggle and google dataset search.
• Papers with similar problems: google scholar and papers with code (pwc is a surprise tool that will help us later).

If kaggle shrugged and google laughed at you, if none of the researchers made their datasets available, or if you’re a trailblazer thinking of using data no one has gathered before… Time to get your hands dirty. Set up some crawlers, scour google and youtube. And may the force be with you.

Preprocessing involves preparing the data, making sure it’s in the format the model requires.

If you’re working with images, always normalise!! This isn’t the first or the most important step, but it’s the one that always gets me. And use open CV to turn images into arrays.

If you’re not, I can’t help you here. Google and stack overflow will guide you.

Congratulations! Like Dante, you have muddled through the 9 rings of eternal damnation and survived to tell the tale. These are the pearly gates, this is the fun part.

Someone said once that there’s nothing new under the Sun, I think. And they were right. The first thing you should do is try and replicate models from papers with code. See, I told you it would help us.

Papers with code kind enough to sort papers by performance and provide github links. All you have to do is clone the repository and get the model working, try it out with your dataset, and adjust it as needed.

It’s time to train the model. Hopefully you set data aside for testing. My bad, I should have mentioned that earlier.

Usually we do 70% of the data for training, and 30% for testing. You can also have 20% for testing and 10% as a hold out.

Training can happen in seconds, or it can take days. It all depends on how complex your data and your model are.

This is the part where you sit back and relax. Grab some coffee, take a nap.

Hush, now! It’s okay. If it were perfect that would mean there’s no more fun to be had!

As far as metrics go, I like to go all out and catch them all. Keep in mind that if you have an unbalanced dataset (where one class has much less data) the accuracy will be misleading. Look at the sensitivity, specificity, all of those.

Check the training metrics.

Are they much better than the test metrics? That means the model learned a bit too well. It memorised the data and doesn’t perform as well on new information. You need more data, or a different model altogether.

Now if the training metrics suck? That means the model sucked. You need to tweak it. Add more epochs or layers, change the loss or activation function. If nothing works, it might be time to switch out the model.

This is called overfitting and underfitting, if you want to be fancy.

All jokes aside, the joyous world of Machine Learning is well within your reach. It’s not as hard as it seems, and it is just as fun as I made it seem. Pinky promise.

And you’re sure to have a large community of people banging their heads over the same exact problems you’re having.