此书尚未出版，该笔记仅供学习参考，原文见http://www.iro.umontreal.ca/~bengioy/dlbook/

Machine Learning Basics

本部分介绍机器学习的基本概念。

what is learning?

Definition of learning: “A computer program is said to learn from experience $E$ with respect to some class of task $T$ and proformance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$”

The task, $T$

Classification, Classification with missing input, Regression, Transcription, Density estimation, Anomaly detection, Synthesis and sampling, Imputation of missing values.

The proformance measure, $P$

accuracy of the model or the probability of the model to some examples.

Using test data to evaluate the proformance.

we always use loss to represent the cost associated with a particular event (such as a classification).

the objective of learning is then to minimize the loss

choice error as loss for classification tasks

the choice of performance measure depend on the application.

In order for a performance measure to be optimized directly. we almost always require a smooth signal — a gradient

in place of the natural loss functions, we often use surrogate performance measure (also called surrogate loss functions) that are amenable for direct use as the objective function optimized with respect to the model parameters.

The experience, $E$

example, label.

design matrix — in some case, feature have different size

Generalization, Capacity, Overfitting and Underfitting

Generalization

generalization error.

Formally, generalization performance is typically defined as the expected value of the chosen performance measure, taken over the probability distribution of interest.