Projects

Identifying concept libraries from language about object structure

Our understanding of the visual world goes beyond naming objects --- encompassing our ability to parse objects into meaningful parts, attributes, and relations. In this work, we leverage natural language descriptions for a diverse set of 2K procedurally generated objects to identify the parts people use and the principles leading these parts to be favored over others... (read full abstract below)

  Paper   Project Website

Training Transformers for Practical Drug Discovery with Tensor2Tensor

Despite the recent successes of Transformers in NLP, adoption of this architecture within cheminformatics has so far been limited to sequence-to-sequence based tasks like reaction prediction – just one class of tasks in the rapidly-growing problem space of molecular machine learning. This blog post for Reverie Labs (and accompanying Colab notebook) walks through a complete pipeline for training Transformer networks on a wide range of drug discovery tasks using Tensor2Tensor.

  Blog Post   Colab Notebook

Designing a User-Friendly ML Platform with Django

Building useful real-world machine learning systems requires more than just optimizing metrics. In an applied setting like computational chemistry, how can we make the latest ML research user-friendly and accessible for non-expert users? This blog post for Reverie Labs (and accompanying talk) outlines our Django-based production ML platform, which allows users to get predictions from state-of-the-art models with the ease of ordering online takeout.

  Blog Post   Talk Recording

Adversarial Regularization for Visual Question Answering

Visual question answering (VQA) models have been shown to over-rely on linguistic biases in VQA datasets, answering questions "blindly" without considering visual context. Adversarial regularization (AdvReg) aims to address this issue via an adversary sub-network that encourages the main model to learn a bias-free representation of the question. In this work, we investigate the strengths and shortcomings of AdvReg with the goal of better understanding how it affects inference in VQA models.

  Paper   Code

On the Flip Side

We explore a reformulation of the VQA task that challenges models to identify counterexamples: images that result in a different answer to the original question. We find that the multimodal representations learned by an existing state-of-the-art VQA model do not meaningfully contribute to performance on this task. These results call into question the assumption that successful performance on the VQA benchmark is indicative of general visual-semantic reasoning abilities.

  Paper   Code

Semantic Projection

We introduce a powerful, domain-general method for recovering human knowledge from word embeddings: "semantic projection" of word-vectors onto lines that represent various object features. Our results from a large-scale Mechanical Turk study show that this method recovers human judgments across a range of object categories and properties.

  Paper   Code

GAN You Spot the Difference?

We introduce a novel category learning paradigm called the “Art Show Game” that uses GANs to render naturalistic stimuli as an alternative to the contrived and often cartoonish stimuli typically employed in category learning studies. We define a sampling scheme that provides analogues to two key parameters for category learning stimuli: dimensionality and family resemblance. In an experiment with human learners, we closely replicate the findings of Vong, Hendrickson, Perfors, and Navarro (2016), which show that family resemblance mediates the curse of dimensionality's adverse effects on learning.

  Paper

Reinforcement Learning in Super Mario Bros.

We implemented a reinforcement learning agent to play Super Mario Bros. using the OpenAI Gym environment. We tested the performance of three RL algorithms: discrete Q learning, approximate Q learning, and approximate SARSA. While our agent is able to beat World 1-1, we discovered that SMB is a very difficult AI problem due to the extremely large state space. We discuss this problem and other challenges in our paper.

  Writeup   Code

Transfer Learning in CNNs

One key component of human perceptual learning is the ability to rapidly generalize acquired knowledge across categorical domains. In contrast, modern neural networks require thousands of training iterations on vast datasets. In this project, we conducted a systematic study of CNNs’ ability to generalize perceptual knowledge from one object classification task to another.

  Writeup   Code

Imitating Celebrity Tweets with AI

We developed a system that automatically extracts training data from a Twitter user's timeline to generate novel tweets in their style. Our goal was to explore the viability of using a K-dimensional Markov model to generate reasonable tweets.

  Writeup   Code