How Should We Handle Subjective Problems in Computer Vision?

Over the past two years, I've been exploring a category of subjective computer vision problems and how we should approach them. In this post, I'd like to make the case for a more humanistic approach to these problems.

Models of the Self in Disco Elysium

In this post, I’d like to examine an unusual text, the label on a designer jacket produced by ZA/UM, the art collective and video game studio behind the hit 2019 game, Disco Elysium.

I Read 17 Old Web Design Books and You Will Not Believe What They Said!

Well, you might believe it. They mostly just gave advice on how to design websites. But the thing which I found absolutely fascinating in these books is the way they gave that advice. In fact, we found this style of writing so interesting, we wrote a paper about it, which is now available.

Tracing the Origins of Optimizationism

In mathematics, optimization is a technique for finding the highest or lowest value of a function. In machine learning we use optimization as a tool for fitting models to data, but it is more than that. In many ways, computer scientists engage with optimization more like an ideology.

Visualizing the Contour of an Artist's Career With Dynamic Programming

I made some visualizations of art images over time using an interesting algorithm! I think they have some provocative qualities which are worth discussing. My code is on Github so you can make plots like these too if you want.

The Problem With Beauty Equations

Over the past few months, I've been digging into a strange subfield of computer vision called "image aesthetic quality assessment. I've become absolutely infatuated with this research topic, not because I think that what they are doing is good or right, but because I think their work is a really good way to approach a difficult issue at the core of all the topics I study.

Sonifying the Melodramatic Life of my Roomba

I procedurally generated some musical accompaniment for a video taken from the perspective of my roomba!

So What Exactly Do You Do?

Hello again! I'm not sure who reads this blog (if anyone) but I've gotten the question a lot recently "so what exactly is it that you do?" and wanted to give a good answer, in full, somewhere.

History of Web Design Website

I made a little website on NeoCities to promote our recent CHI paper and wanted to post it here and point out some of the ideas behind my designs.

The Limits of AI Image Colorization: A Companion

I wrote a short article for hyperallergic.com. As it's a website for a general audience, it's a quite short article without technical details. As a companion here, I wanted to write out some more of the details about deep learning algorithms for image colorization.

A Short History of Web Design

Over the past two years, I've been studying the history of web design. I wrote an article about our work for The Conversation, and wrote a post on this blog about some of that work, but I haven't actually written about the history there yet. The following is my 1500 word summary of the history of web design.

Probablistic Models of Color Harmony

I did a generative art project based on research into color theory!

CycleGANs and Mimesis

In my free time, I made some art with GANs

Can Neural Networks See Cuteness?

I was wondering whether neural networks trained to detect cute things actually understand cuteness. I did some experiments and found that the answer was "yes…sort of?"

Talking Mathematically About Color Schemes (Part II)

In this section, we're going to start by looking at how to measure the difference between two colors, then define what a color scheme is in our quantitative framework and look at ways of extending our way of measuring difference to color schemes.

Talking Mathematically About Color Schemes (Part I)

I've been busy with grad school recently, but I thought work I've done recently merited a blog post. I've been looking at changes in web design over time using image analysis, and we want to find metrics which capture why websites look similar to one another, and use those to identify design trends. In this line of inquiry, I ran into an interesting question, how do you measure the difference between two color schemes?

User Interfaces, Metaphors and Midi Keyboards

I've had some free time lately (I recently left my job at RTI and am starting a PhD at Indiana University), so I've been reading and programming for fun again. Some of that reading and programming tied itself nicely into a coherent project that I've written up below. If you just want to skip to the programming, I've made a webapp available here.

Self-Similarity Matrices and Xenakis

Now that I'm updating my blog again, it occurred to me that I never posted about my musical studies capstone thesis. Rather than post a bloggy version of the things I was thinking about, like I did while working on my honors research, I'm going to keep things brief and only discuss one of the computational methods that I used, which is a neat data visualization technique.

Sonic Primitives

Last week, I discovered a wonderful twitter account that posts art generated from images using Primitive.lol by Michael Fogleman. Since following it, I have been bombarded with beautiful images of shapes arranged to resemble real things.

The Incredibles (continued)

A week ago, I spewed some of my thoughts about The Incredibles in a post. Now I've graduated college and am back home with more free time. Hopefully, in this post, I can go through the rest of my ideas about the first part of Giacchino's fantastic score.

The Incredibles

It's May now, and I'm done with classes! Commencement at Oberlin is next Monday, so I have some free time, and I decided to write a post. Rather than spend more time on my honors research, I figured I'd go into something else I've been thinking about and researching lately, which is Michael Giacchino's score to the Disney Pixar film The Incredibles. I wrote a paper for a music theory class on this topic, but I had a lot of thoughts about the music that didn't fit nicely into a paper, so I'm going to try to fit more content into a less structured blog post.

Survey Results

Well, it's been a month, and I finally have a free day to write up the answers to the survey! People had been asking whether they got the right answers, and since my results were anonymous, I couldn't just look it up. That said, I figured it would be fun to make a post with some graphs.

Final Thesis

Despite never getting the chance to post about generation schemes or survey design (which I plan to do retroactively at some point!), I was able to complete my research and sumbit my thesis on time.

DeepBach

As promised in a previous post, I wanted to write a summary of the DeepBach paper, published in December, which achieves a result similar to what I want to achieve with this project. This article saw some news coverage when it was published, so it may sound familiar. It is currently, as far as I know, the state-of-the-art model for generating Bach-like music.

Taking the Product

After finishing the individual expert models, I moved on to building a product model. The concept comes from Geoffrey Hinton and this particular application was my colleague from summer research, Daniel Johnson's idea. Basically, instead of training naively on untransposed pitches, preprocess that pitch data in a variety of different ways, then train several "expert" models on the preprocessed data, then take the output probability density functions outputted by each expert, multiply them together and renormalize.

Expert Models

The next step towards machine composition of four-part harmony, after working with a naive generative model, was to try to implement other models that better preprocessed training data. My ultimate goal with these is to apply Geoffry Hinton's Product of Experts and multiply together the output probability distributions of several models trained on different aspects of the training data to get a more nuanced result.

Naive Generative Model

Rather than dive immediately into complex neural network architectures, I wanted to create a first model that predicts the absolute pitches of one voice given the absolute pitches in other voices. My goal was not to create the best possible predictive model of music so much as it was to create a control against future experiments that pre-process the training data further. If the dataset I had found could be modeled with high accuracy without significant pre-processing, then any more sophisticated model would be unnecessary and I would know to find a less homogeneous dataset before proceeding.

Project Summary

My goal with this project is to investigate methods for procedurally generating counterpoint/voice leading by learning trends in existing music, rather than starting from pre-existing rules of counterpoint/voice leading. My reasons for this are two-fold. First, while undergrad composition and music theory classes in the conservatory model have strict composition rules as their bread and butter, mature composers of the past and present by and large compose music by recreating music they enjoy or find interesting, often by breaking the established rules of their chosen form. Second, there are many examples in the music composition literature of prescriptive, rule-based models for stochastic composition (see Hiller, Tenney, Xenakis, Eno, etc.). As far as I know, there are only a few people who have tried to do composition using machine learning techinques.