Computer games are big business, which is also reflected in the growing interest in competitive gaming, the so-called electronic sports. Multi-player online battle arena games are among the most successful games in this regard. In order to execute complex team-based strategies, players take on very specific roles within a team. This paper investigates the applicability of supervised machine learning to classifying player behavior in terms of specific and commonly accepted but not formally well-defined roles within a team of players of the game Dota 2. We provide an in-depth discussion and novel approaches for constructing complex attributes from low-level data extracted from replay files. Using attribute evaluation techniques, we are able to reduce a larger set of candidate attributes down to a manageable number. Based on this resulting set of attributes, we compare and discuss the performance of a variety of supervised classification algorithms. Our results with a data set of 708 labeled players see logistic regression as the overall most stable and best performing classifier.