Inside the U-Net

A look inside a neural network while it reads a satellite image. A real Sentinel-2 image of glaciated terrain in British Columbia, taken on 28 September 2017, goes in at the top left, runs across the top of the U where it is turned into more and more abstract maps, turns at the bottleneck on the right and comes back along the bottom. At the end the answer of the network, a map of land-cover classes, appears next to the input.

One Sentinel-2 image at 10 m resolution, and the land-cover model published on Hugging Face (landscape-change-detection-unet).

The network really runs on the image, and the maps inside each stage are captured and turned into colours. Vertical links show the shortcuts between the two sides of the U.

The sound scans the network: the activity of each stage is read from left to right, with high notes for the top rows and low notes for the bottom ones. It ends on a chord whose notes are the predicted classes, as loud as the area they cover, which settles into tune.