Human beings rely on visual information to learn about the environment around them, construct representations of the world, and control their actions. By and large, humans are remarkably accurate when it comes to complex motor actions such as catching a baseball or hitting a target. In fact, the perceptual skills underlying such actions are not easily understood as they are far superior to any visual information processing capability of artificial systems constructed to date. In stark contrast to our excellent perception-action abilities, there are conditions under which humans make striking judgment errors that are at odds with the visual information experienced.. We will describe some examples of such errors in a large proportion of the population suggesting that knowledge of the physical world is represented poorly in the cognitive domain. We will discuss some explanations for this phenomenon, and explore the implications for a scientific study of visual representations and interpretations.