Hi, I'm Gabe.

I study brains, minds, and machines at Harvard and MIT.

Organizations

Harvard College

Class of 2018

I concentrate in Computer Science with a focus in Mind, Brain, and Behavior, a highly interdisciplinary program of study that draws on ideas and methods from computer science, cognitive science, neuroscience, psychology, linguistics, and philosophy.

Google AI

Google Research / Google Brain

For the past two summers, I interned at Google AI. In 2017, I worked on the AI Perception team, applying deep learning to the problem of multilingual character recognition. In 2018, I worked on the Tensorflow team within Google Brain, democratizing machine learning and deploying Tensorflow models across a wide range of Google products as part of the company's AI-first strategy.

Massachusetts Institute of Technology

Fedorenko Lab

I previously worked as a research assistant at the Fedorenko Lab at MIT. The Fedorenko Lab uses neuroimaging and behavioral methods to construct computational models for language processing in the brain. As part of my research, I developed an unsupervised ML algorithm to extract semantic information from GloVe vectors as part of the IARPA Knowledge Representation in Neural Systems (KRNS) project.

Cox Lab

Harvard Center for Brain Science

In 2016, I worked as a research assistant at the Cox Lab, which seeks to understand the computational underpinnings of vision cognition. I worked on a project to improve the performance of machine learning algorithms for image recognition and face detection using human behavioral data.

Center for Mind/Brain Sciences

University of Trento, Italy

I studied cognitive neuropsychology and social cognitive neuroscience at CIMeC during summer 2015 as part of the Harvard MBB study abroad program. Under the supervision of Paul Downing and Marius Peelen, I helped to design and execute an fMRI experiment that found differential activation of face (FFA/OFA) and body (EBA) regions when subjects viewed images of humans vs. wax figures of those same individuals.

Harvard Society for Mind, Brain, and Behavior (HSMBB)

HSMBB​ (hsmbb.org) is an undergraduate organization that regularly hosts events, talks, and symposia on a wide range of topics related to cognitive science. As a member and former chair of the HSMBB board, I coordinate with faculty and students, plan programming, promote events, and manage the group's budget.

Publications

On the flip side: identifying counterexamples in visual question answering

Grand, G., Szanto, A., Kim, Y., & Rush, A. (2018). Presented at KDD ’18 Deep Learning Day, Aug. 2018, London, UK. arXiv:1806.00857

Projects

On the Flip Side: Identifying Counterexamples in Visual Question Answering

Visual question answering (VQA) models respond to open-ended natural language questions about images. While VQA is an increasingly popular area of research, it is unclear to what extent current VQA architectures learn key semantic distinctions between visually-similar images. To investigate this question, we explore a reformulation of the VQA task that challenges models to identify counterexamples: images that result in a different answer to the original question. We introduce two methods for evaluating existing VQA models against a supervised counterexample prediction task, VQA-CX. While our models surpass existing benchmarks on VQA-CX, we find that the multimodal representations learned by an existing state-of-the-art VQA model do not meaningfully contribute to performance on this task. These results call into question the assumption that successful performance on the VQA benchmark is indicative of general visual-semantic reasoning abilities.

Semantic Projection: Recovering Human Knowledge of Multiple, Distinct Object Features from Word Embeddings

Word embeddings support measurement of the semantic similarity between objects via cosine similarity or euclidean distance. However, human judgments about object similarities are highly context-dependent and involve multiple, distinct semantic features. For example, dolphins and alligators appear similar in size, but differ in intelligence and aggressiveness. Could such context-dependent relationships be recovered from word embeddings? To address this issue, we introduce a powerful, domain-general solution: "semantic projection" of word-vectors onto lines that represent various object features. Our results from a large-scale Mechanical Turk study show that this method recovers human judgments across a range of object categories and properties.

Reinforcement Learning in Super Mario Bros.

We implemented a reinforcement learning agent to play Super Mario Bros. using the OpenAI Gym environment. We tested the performance of three RL algorithms: discrete Q learning, approximate Q learning, and approximate SARSA. While our agent is able to beat World 1-1, we discovered that SMB is a very difficult AI problem due to the extremely large state space. We discuss this problem and other challenges in our paper.

Generating Hallucinations with Deep Boltzmann Machines

In 1760, Swiss philosopher Charles Bonnet published a manuscript detailing the bizarre visual hallucinations his grandfather experienced after losing his sight. Charles Bonnet Syndrome occurs when the brain's homeostatic mechanisms attempt to compensate for a prolonged lack of visual input, producing spontaneous, complex hallucinations. In this project, we recapitulated a study by Reichert et al. (2013) that uses Deep Boltzmann Machines to produce a generative model of visual hallucinations.

Structure discovery with semantic vectors

Distributional semantic models represent word meanings in a high-dimensional vector space. However, because these vector spaces typically encompass several hundred dimensions, their structure remains an active area of research. In this project, undertaken at the Fedorenko Lab, we use unsupervised machine learning algorithm to automatically map and discover various category-based substructures within the GloVe vector space.

Perceptual annotation through eyetracking

Despite recent breakthroughs in deep learning, humans remain significantly better than machines for many problems in computer vision -- for instance, in identifying partially-obscured faces in images. The goal of perceptual annotation research, undertaken at the Cox Lab, is to measure human performance on various computer vision problems, and to use this psychophysical data to improve the performance of machine learning algorithms. By tracking participants' eye movements as they view images, we can better understand which features of images are most salient across different image identification tasks.

Transfer learning in convolutional neural networks

One key component of human perceptual learning is the ability to rapidly generalize acquired knowledge across categorical domains. In contrast, modern neural networks require thousands of training iterations on vast datasets. In this project, undertaken at Harvard, we conducted a systematic study of convolutional neural nets’ ability to generalize perceptual knowledge from one object classification task to another.

Imitating celebrity tweets with AI

We developed a software system that learns over a Twitter user's timeline to generate novel tweets in their style. Our goal was to explore the viability of using a K-dimensional Markov model to generate reasonable tweets.

Hyper-local social network for Harvard dining

Berg aims to create a better sense of community for the 1,700+ freshmen at Harvard by connecting students with their friends at mealtimes. Berg is available as an iOS and mobile web app.

Live crisis break feed for Model UN

Model UN crisis committees simulate international relations through fast-paced "crisis breaks." I developed an open-source Live Crisis Tool, based on the meteor.js framework, that allows directors to display crisis updates in real time to their committee members via a Twitter-style UI.

Representative Coursework

Computer Science

  • CS 20: Discrete Mathematics for Computer Science
  • CS 50: Introduction to Computer Science I
  • CS 51: Introduction to Computer Science II
  • CS 108: Intelligent Machines: Design and Ethical Challenges
  • CS 121: Introduction to the Theory of Computation
  • CS 179: Design of Useful and Usable Interactive Systems
  • CS 181: Machine Learning
  • CS 182: Artificial Intelligence
  • CS 287: Machine Learning for Natural Language
  • Math 21a: Multivariate Calculus
  • Math 21b: Linear Algebra
  • Stat 110: Probability Theory

Cognitive Science

  • FRSEMR 26K: Transformative Ideas in Brain Science and Neuroscience
  • MBB S-96: The Social Brain
  • MBB S-101: Windows into the Structure of the Mind and Brain
  • MBB 90R: Supervised Research in Mind/Brain/Behavior
  • MCB81: Fundamentals of Neuroscience
  • PSY 1057: Music and the Mind
  • PSY 1401: Computational Cognitive Neuroscience
  • SLS 20: Psychological Science

Current as of Spring 2018.

Languages

For computers

Python

MATLAB

HTML

CSS

Javascript

jQuery

PHP

mySQL

meteor.js

OCaml

TensorFlow

C

Objective C

Swift


For humans

English (native)

Spanish (fluent)

Mandarin Chinese (basic)

Contact