Novelty Search (NS) is a unique approach towards search and optimization, where an explicit objective function is replaced by a measure of solution novelty. However, NS has been mostly used in evolutionary robotics, its usefulness in classic machine learning problems has been unexplored. This thesis presents a NS-based Genetic Programming (GP) algorithms for common machine learning problems, with the following contributions. It is shown that NS can solve real-world classification, clustering and symbolic regression tasks, validated on realworld benchmarks and synthetic problems. These results are made possible by using a domain-specific behavior descriptor, related to the concept of semantics in GP. Moreover, two new versions of the NS algorithm are proposed, Probabilistic NS (PNS) and a variant of Minimal Criteria NS (MCNS). The former models the behavior of each solution as a random vector and eliminates all the NS parameters while reducing the computational overhead of the NS algorithm; the latter uses a standard objective function to constrain and bias the search towards high performance solutions. The thesis also discusses the effects of NS on GP search dynamics and code growth. Results show that NS can be used as a realistic alternative for machine learning, and particularly for GP-based classification.