In 2014 we ran a very successful machine learning challenge in High Ern-ergy physics attracting 1785 teams, which exposed the machine learning community for the first time to the problem of " learning to discover " (www.kaggle.com/c/higgs-boson). While physicists had the opportunity to improve on the state-of-the-art using " feature engineering " based on physics principles, this was not the determining factor in winning the challenge. Rather, the challenge revealed that the central difficulty of the problem is to develop a strategy to optimize directly the Approximate Median Significance (AMS) objective function, which is a particularly challenging and novel problem. This objective function aims at increasing the power of a statistical test. The top ranking learning machines span a variety of techniques including deep learning and gradient tree boosting. This paper presents the problem setting and analyzes the results.