Solution to Exercise 2
a) Softmax
For ,
Therefore,
The tool allows logits between and . With three classes and the Softmax function defined in the exercise without temperature scaling, one component is largest when its logit is and the two other logits are both :
The component can therefore approach very closely but cannot reach for finite logits. If the optional temperature mode is enabled at its minimum value , the result is even closer:
Technically, the tool also permits a single neuron. In that case, the sole Softmax value is necessarily ; however, this trivial case is not multiclass classification.
b) Comparing losses: CCE + Softmax versus MSE + Softmax
i) CCE with one-hot encoding
In single-label classification, the one-hot target vector has value exactly at the position of the correct class . All other components satisfy . Hence every term in the sum except the one for the correct class vanishes:
In this case, the CCE loss depends directly on the predicted probability of the correct class.
ii) Confidently incorrect prediction
Let be the correct class and an incorrect class. For a maximally confident incorrect prediction,
For categorical cross-entropy,
Thus, a confidently incorrect prediction is penalized without bound as the probability of the correct class approaches .
For MSE, the limiting case is and :
The MSE loss therefore remains bounded. With the tool's logit limits of and and three classes, for example,
which gives
As in the exercise, the natural logarithm is used for cross-entropy here.
c) Comparing error signals: CCE + Softmax versus MSE + Softmax
i) Total loss as a function of the logits
The formula can be derived in two ways.
Solution 1: Simplified form for single-label classification
For the correct class ,
Substitute this into the simplified form :
Since is a one-hot target vector with , . Therefore,
Solution 2: General categorical cross-entropy form
Substitute
into the general categorical cross-entropy formula:
Since a one-hot target vector satisfies ,
ii) Gradient of the loss
The gradient can likewise be derived from either form.
Solution 1: Simplified form for one-hot encoded targets
Start from
For the derivative of the first term,
The second term gives
Therefore,
Since is a one-hot target vector, this is componentwise. In vector form,
Solution 2: More general form
For any component , differentiate the two terms separately:
and
Thus, componentwise,
For the correct class , ; for all other classes, . Therefore,
In vector form,
iii) Examining the error signals
For the correct class , and therefore . For an incorrect class , and therefore .
| Case | Example for and | Meaning | |
|---|---|---|---|
| confidently correct | , | Almost no correction is required. | |
| confidently incorrect | , | The logit of the correct class is raised strongly, while that of the incorrectly selected class is lowered strongly. | |
| uncertain but correct | , | The correct class wins narrowly; the error signal is still substantial. | |
| uncertain and incorrect | , | The incorrect class wins narrowly; the correction has the same direction but is weaker than for a confidently incorrect prediction. |
The signs have a clear effect under gradient descent: a negative entry raises the corresponding logit, while a positive entry lowers it. This increases the probability of the correct class and decreases those of the incorrect classes.