Seeing, touching, sketching, exploring--throughout history these fundamental physical activities have defined our world, regularly supporting creativity and scientific discovery. In recent years, powerful scientific visualization tools have emerged but the potential to closely couple these techniques with natural, physical, spatial human-computer interfaces remains largely untapped. To address these issues we outline a forward-looking research agenda consisting of five major challenges for natural interfaces for visualization. The technological building blocks are now in place to address these challenges to enable an exciting future where natural interfaces powerfully strengthen and expand use of visualizations in science, engineering, art, and humanities.