Info

Task 3

In this worksheet, you will compare Categorical Cross Entropy (CCE) and Mean Squared Error (MSE) for classification with feedforward neural networks.

Exercise 1 – Experimental comparison of CCE and MSE

Open the FNN classification tool

Dataset: download synthetisch_ce_vs_mse_drei_klassen.csv

Step 1

Load and normalize the dataset

Load the sample dataset and normalize it to the interval .

Settings for normalizing the dataset

Step 2

Choose a suitable FNN

This two-dimensional dataset does not require deep feature abstractions. The main challenge is to place several local polygonal regions.

A wide, shallow network is therefore preferable:

  • Many neurons in one layer provide many possible separating lines.
  • The gradient only needs to propagate through a few layers.
  • There is less risk of inactive ReLU paths.

Model settings with a wide hidden layer and softmax output

a) Run and compare the classifications

Run the classification with multiclass cross entropy and with MSE , starting from the same “random” initial parameters with seed 42.

Compare the classification results in terms of:

  • total loss ,
  • accuracy,
  • the visual quality of the decision regions in the argmax heatmap, which is the lower heatmap.

b) Loss after reaching complete accuracy

Explain why the loss can continue to decrease even after 100% accuracy has been reached.

Exercise 2 – Mathematical comparison of CCE and MSE

Open the Softmax tool

We use the following definitions:

  • Categorical Cross Entropy (CCE):
  • Softmax:
  • Per-sample MSE loss:

a) Softmax

Calculate by hand and with the tool. What is the largest value of a Softmax component that can be produced with the app?

b) Comparing losses: CCE + Softmax versus MSE + Softmax

i) CCE with one-hot encoding

Explain why, for one-hot encoded target vectors in single-label classification, the error as a function of the Softmax outputs satisfies:

ii) Confidently incorrect prediction

Use the tool to compare and when the FNN makes an incorrect prediction with the greatest possible confidence. Inspect the calculations shown by the tool.

c) Comparing error signals: CCE + Softmax versus MSE + Softmax

i) Total loss as a function of the logits

Show by substitution that the total loss as a function of the logits and the true value is:

You may use the form already simplified for single-label classification,

or start from the general form

Hint

Property of the target vector

If you start from the general form, use .

ii) Gradient of the loss

Show by differentiation that

with the estimated probabilities

For the differentiation, you may either start from the simplified form for one-hot encoded targets

or alternatively use the general form

Use one-variable school calculus for the partial derivatives. Treat as a function of only one variable at a time,

and distinguish between the correct class and the remaining classes .

iii) Examine error signals

Examine the error signals for confidently correct, confidently incorrect, uncertain but correct, and uncertain and incorrect predictions. Substitute values of 0 or 1 for and in

View the solution to Exercise 2

Files, data source, and license

The penguin dataset is an adapted excerpt from palmerpenguins: Palmer Archipelago (Antarctica) penguin data by Horst, Hill, and Gorman (2020), Zenodo, DOI 10.5281/zenodo.3960218. The original dataset is licensed under CC0 1.0 Universal.