Neural networks have become popular tools for transfering the style of one artistic work or photograph to another. Methods for performing style transfer are distinguished from one another by features of the neural network’s design as well as the loss function that guides network updates. In a recent project, I teamed up with Alessandro Zito to use a method based on the theory of optimal transport to transfer visual elements of musical instruments onto the forms of animals. The method was developed by Vince Marron and is available on his GitHub Page. The original photos used to produce the images in this post can be found at the links in the References section.
The artistic side of our work starts from a simple question: are musical instrument able to generate rhythm and motion even when they are not played? To test this, we fused the picture of several instruments with pictures of animals in resting poses (not caught during a particular motion).
Rather than simply combining the pictures, we wanted to recognize the existing patterns and shapes of the instruments and to use them to deconstruct and then reconstruct the resting animals. The decomposition gives a sense of motion that is typical of the paintings of Franz Marc and the Italian futurists (especially Umberto Boccioni), with an artistic process that is somehow similar to that of the early Piet Mondrian and, more broadly, Damien Hirst. All of these artists provided a vivid and dynamic image of movement that always came from the geometric segmentation of reality.
Our instruments are tools to set the animal into motion, to transfer momentum onto its soul. When every melody stops all that remains is the mood of the listener. Thus, we want the instruments, at the end, to almost disappear. All they have to provide is a spark of life.
CodeWritten on December 9th, 2019 by Jordan Bryan