So what exactly do you do?
Hello again! I’m not sure who reads this blog (if anyone) but I’ve gotten the question a lot recently “so what exactly is it that you do?” and wanted to give a good answer, in full, somewhere.
What do you do?
I study the ways that machine learning bumps up against the humanities, especially history and aesthetics (sometimes also music, design and art). This means I study both the way that computer scientists and humanists think differently about similar problems, do my best to translate between their languages and make recommendations in each direction. For the computer scientists, this means reframing problems and restructuring datasets to avoid making horribly wrong claims. For the humanists, this means developing and introducing computational techniques that can help them both think about their evidence differently, and better search large databases.
I also study the culture of computing. I believe that for the most part, algorithms, data structures and design principles are invented and not discovered, and this means that the people who invent them have more control over the way computer science has been built than they would have us believe. Things like nerd masculinity and the culture of the early web are as important to understanding contemporary computer science as canonical topics like bitwise operators and complexity classes. While this sort of research seems quite distant from information retrieval and aesthetics, I argue that it is an essential part of any dialogue between computer science and the humanities.
What projects do you work on?
I’m not going to keep this list up to date, so some of these projects might have turned out to be dead ends. I’m definitely happy to talk about any of these ideas, though!
- The History of Web Design: We recently published based on this research! Check out our paper or the neat website I made for this project. The gist is that the Internet Archive is a huge database of websites, but it’s too large to study using conventional history techniques. Prior work in my group looked into ways that we can analyze websites automatically using computer vision. My work built on that to investigate the hypothesis that websites look more similar now than they used to. We used both image data as well as a series of interviews to dig into exactly how the web has changed over time.
- The Rhetoric of Dot-Com Era Web Design Books: As a follow-up to the history of web design, we noticed when conducting interviews that a lot of our participants mentioned a couple of design books from the dot-com era. I found and read those books and am working to analyze their rhetoric: how do they develop the authority and legitimacy of the author, and how do they construct web design as a discipline? Unsurprisingly, these questions have a lot to do with masculinity and nerd culture.
- Thinking Critically about Image Aesthetic Quality Assessment: There is a small research area within computer vision usually called either “computational image aesthetics” or “image aesthetic quality assessment” which seeks algorithms to measure whether photographs are beautiful. They develop machine learning algorithms based on a dataset of images with aesthetic quality scores to predict the scores for new images. I’m skeptical, though, that the quality they are measuring is really beauty. I also think this problem touches on a couple really interesting philosophical issues regarding computation and subjectivity.
- Color Scheme Modeling: Color scheme generators are a class of algorithms for randomly suggesting combinations of colors for use by designers. They tend to inherit ideas from the theory of color harmony, which asserts that some combinations of colors look better than others, solely based on their geometry on the color wheel. I’m interested in solving the opposite problem: can we build a statistical model from the color schemes which are actually in use? Will that model confirm or deny the theory of geometric color harmony? I’ve written about this project before on this blog.
- Egocentric Art Photography: A large part of photography involves staging and composing shots, but candid photography aims to capture a more natural view of the world (for some definition of natural). Given a sequence of video frames from an egocentric (body-worn) camera, can an algorithm select the frames it finds beautiful or interesting and edit them, either fully autonomously, or collaboratively with a human curator? Can we use this as a “test problem” to probe at the issues with aesthetic quality assessment?