Current virtual reality systems are more immersive than other traditional forms of digital entertainment, but the handheld controllers they use detract from what would otherwise be a truly immersive experience. This paper outlines a project implemented to overcome this issue by creating a spell-casting game in which everything is controlled by the user’s hand gestures. The implemented game made use of optical capture of the user’s hands using a Leap Motion controller attached to the front of a virtual reality headset. The captured gesture data was passed through a neural network for precise gesture classification in real-time, and the correct sequence of classified gestures created a spell effect in the game. The user was able to cast 10 different spells utilizing various combinations of a set of 14 different gestures.