For Beginners in Machine Learning II

Programming

In my view, studying machine learning involves three broad areas. The first is programming. I consider it essential: unless you are one of the very few exceptional scientists researching machine learning theory, it is hard to imagine studying machine learning without programming. Naturally, this raises the question of which language to learn. You can easily find debates about whether Python is currently popular, R is easier to learn, or C is needed for fast computation. But if you are new to programming, there is no need to agonize over the choice. Start by learning and using a language you like. At the beginning, trying is more important than what you start with. Once you understand the concepts, you will naturally discover which languages make your work easier or help you collaborate. I have worked mainly with people who use R, but recently I have increasingly worked with Python users and have gradually begun using Python more myself. Important programming concepts include data structures, control flow, and iteration; data structures deserve the most time. With further study, concepts such as file input/output and regular expressions are useful for data preprocessing.

Probability, statistics, and mathematics

The second area is probability, statistics, and mathematics. This may be the hardest part for people without a science background. Statistics takes a long time to learn, and mathematics probably does too. There are many relevant courses; among those I studied, I consider the following important.

• Probability and statistics
• Introduction to Statistics (first year): Learn the concept of randomness and the foundations of estimation and hypothesis testing.
• Concepts and Applications of Probability (second year): Learn about random variables, probability distributions, and expectations, along with the foundations of stochastic processes.
• Mathematical Statistics I and II (second–third year): Learn about likelihood functions and estimation methods.
• Bayesian Statistics (fourth year): Learn the philosophy of Bayesian statistics, modeling, and estimation.
• Multivariate Analysis (fourth year): Learn properties of the multivariate normal distribution and concepts in principal component and factor analysis. These may also provide insight into dimensionality reduction.
• Machine Learning (fourth year): Learn the foundations of machine learning methods, including how to model using an empirical risk function.
• Mathematics
• Calculus I (first year): Build a foundation in calculus.
• Linear Algebra (second year): Learn about vector spaces and how matrices express computations concisely. Matrix decompositions introduce fundamental ideas for extracting information from data.
• Introduction to Analysis (third year): Learn about limits.

These courses have a natural order, and I recommend following it as closely as possible. Otherwise, I think there is a substantial risk of giving up along the way. There are roughly ten courses, so they require considerable time. It is worth recognizing that doing something new in machine learning is very difficult without this knowledge.

Computation

The final area is computation. Once you can model with probability, statistics, and mathematics and express the model to a computer through programming, you need to compute. The work largely comes down to two problems: solving optimization problems that minimize an objective function subject to constraints, and using Monte Carlo simulation to draw random samples from complex probability distributions specified by a model. I recommend the following courses.

• Statistical Computing (third year): Learn basic numerical computation and methods for generating random samples.
• Optimization (fourth year): Learn algorithms for standard optimization problems and study the convergence of their solutions.

Illustration of a quadratic approximation algorithm: the X marks the optimal point, the light red dot is the current solution, the dashed lines are contours of the quadratic approximation at the current solution, and the dark red dot is the optimum of the approximation.

Listing these courses by year should give you a rough idea of how to plan your studies. Of all these subjects, linear algebra is particularly important. It shows how complex calculations can be expressed concisely and how much can be gained from doing so. If you are already studying machine learning and have missed any of these subjects, I recommend studying them separately.