Loss Functions

hinge loss

cross-entropy loss

The cross-entropy between a “true” distribution p and an estimated distribution q is defined as:

$$H(p, q)=-\sum_{x}p(x)\log q(x)$$

Possibly confusing naming conventions

To be precise, the SVM classifier uses the hinge loss, or also sometimes called the max-margin loss. The Softmax classifier uses the cross-entropy loss. The Softmax classifier gets its name from the softmax function, which is used to squash the raw class scores into normalized positive values that sum to one, so that the cross-entropy loss can be applied. In particular, note that technically it doesn’t make sense to talk about the “softmax loss”, since softmax is just the squashing function, but it is a relatively commonly used shorthand.

focal loss

read more

• CS231n - Linear Classification