Modelling epidemics using classical population-based models suffers from shortcomings that so-called individual-based models are able to overcome, as they are able to take into account heterogeneity features, such as super-spreaders, and describe the dynamics involved in small clusters. In return, such models often involve large graphs which are expensive to simulate and difficult to optimize, both in theory and in practice. By combining the reinforcement learning philosophy with reduced models, we propose a nu- merical approach to determine optimal health policies for a stochastic individual-based model taking into account heterogeneity in the population. More precisely, we introduce a deterministic reduced population-based model involving a neural network, designed to faithfully mimic the local dynamics of the more complex individual-based model. Then the optimal control is determined by sequentially training the network until an optimal strategy for the population-based model succeeds in also containing the epidemic when simulated on the individual-based model. After describing the practical implementation of the method, several numerical tests are pro- posed to demonstrate its ability to determine controls for models with contact heterogeneity.