Images projected onto the retina of an animal eye are rarely still. Instead, they usually contain motion signals originating either from moving objects or from retinal slip caused by self-motion. Accordingly, motion signals tell the animal in which direction a predator, prey, or the animal itself is moving. At the neural level, visual motion detection has been proposed to extract directional information by a delay-and-compare mechanism, representing a classic example of neural computation. Neurons responding selectively to motion in one but not in the other direction have been identified in many systems, most prominently in the mammalian retina and the fly optic lobe. Technological advances have now allowed researchers to characterize these neurons’ upstream circuits in exquisite detail. Focusing on these upstream circuits, we review and compare recent progress in understanding the mechanisms that generate direction selectivity in the early visual system of mammals and flies.