During learning, humans recruit multiple cognitive mechanisms, including value-based reinforcement learning and executive functions, like working memory. Recent research has begun to unmask connections between these two systems, proposing roles for attention and working memory in shaping underlying learning computations. Here, using a simple instrumental learning task, we provide evidence that working memory plays a role in establishing the correct state space that reinforcement learning operates over. We show that reinforcement learning is impaired when executive functioning is taxed by a secondary task and that this effect is especially pronounced when the two tasks are performed simultaneously rather than alternated. Computational modeling suggests that when the executive function is occupied, the reinforcement learning system forms policy over a confused state-space. This study adds to a growing body of research proposing a more fundamental role for high-level executive processes in low-level reinforcement learning computations.