In this paper we develop a theoretical analysis of the performance of sampling-based fitted value iteration (FVI) for solving large, or infinite state-space Markovian decision problems (MDP) with a generative model. Unlike most previous results, our theoretical guarantees apply to a large class of regressors (e.g. neural networks, adaptive regression trees, kernel machines, locally weighted learning). The bounds derived on the performance of sampling-based FVI make the dependence of loss explicit on the MDP's controllability and smoothness properties, the MDP's relation to the regressor employed and several other algorithmic choices. We discuss the relation of our results to previous results and illustrate some tradeoffs by means of a computer experiment.