What computation does the human brain perform when we experience ‘red’, ‘green’, ‘yellow’, or ‘blue’? Where in the visual pathway does the human visual system combine the retinal cone signals (L, M, S) to yield these fundamental colour sensations? Behavioural data show that the four unique hues (red, green, yellow, blue) do not map onto the cone-opponent mechanisms (i.e. L–M; S-(L+M)) found in the Lateral Geniculate Nucleus, a subcortical structure involved in early visual processing. The brain imaging experiment supports the behavioural result: using pattern classification algorithms applied to fMRI brain activation patterns we show that unique hues cannot be classified in the LGN, but we achieve above chance classification in primary visual cortex (V1). Our imaging data provide strong evidence that the unique hues do not originate in subcortical areas, but in the visual cortex, possibly as early as primary visual cortex.