CycleGAN Façade Art

Generative Adversarial Networks (GANs), like all machine learning models, are memetic. Few more so than CycleGANs, which learn a mapping between two modes: in this case, building façades and segmentation maps with windows, etc. labeled by different colored rectangles. One model, f, learns f(façade) = segmentation while the other, g, learns g(segmentation) = façade. These models are trained to convert back and forth from one mode to the other, so they ideally learn a perfect mapping, i.e. f(g(B)) = B and g(f(A)) = A.

In each of the following studies, mode A is interpreted as the Façade of a building, mode B is the segmentation map. real_A and real_B are the input images, which are unmodified. generated_B and generated_A are the corresponding examples in the other mode, i.e. f(real_A) and g(real_B). reconstruction_A and reconstruction_B are the reconstructions of real_A and real_B, i.e. g(f(real_A)) and f(g(real_B)).

CycleGAN

Unfortunately, like any transcription from one mode to another, the reconstruction process is imperfect and introduces artifacts. If repeated many times, those artifacts overwhelm the original signal.

Study 1

Study 2

Study 3: After Alvin Lucier

In studies 3-6, real_A and real_B are the same image. While that breaks the assumptions of the model, CycleGANs are robust and manage out-of-distribution data quite well. Notice the differences between f(g(x)) and g(f(x)).

Study 4: After Paul Klee

Study 5: After Piet Mondrian

Study 6: After Hundertwasserhaus

See this second page for some more CycleGAN Art!