Recent satellite missions launched in the past decades have led to a huge amount of earth observation data, most of them being freely available. In such context, satellite image time series have been use to study land used and land cover information. However, optical time series, like Sentinel-2 or Landsat ones, are provided with a irregular time sampling for different spatial location, and images contain clouds and shadows.Thus specific pre-processing techniques are required to properly classify such data. The proposed approach in the paper is able to deal with irregular temporal sampling and missing data directly in the classification process. It is based on Gaussian Process and allows to perform jointly the classification of the pixel labels as well as the reconstruction of the pixel time series. Its complexity scales linearly with the number of pixels, it is amenable in a large scale scenario. Experimental classification and reconstruction show that the method does not compete yet with state of the art classifier but allows for a robust reconstruction and does not need any temporal preprocessing