Affordable and accessible screening tools are crucial to diagnose and, hence, effectively address vision impairment. We propose a novel pipeline based on photorefraction which complies with these requirements. It includes an innovative, easy-to-use, optical portable device combined with an image processing algorithm to automatically compute refraction errors from twelve jointly captured images. To do so, a supervised deep convolutional model is proposed, trained on synthetic images generated by a physical-aware simulator, as well as on real images carefully collected from a cohort of volunteers. We provide in this article an assessment of our approach accuracy and robustness on both simulated and real datasets. Our promising results pave the way for a practical application of the proposed device for an accurate vision loss diagnostic in a real setting.