Covering more than 70% of Earth surface, oceans play a key role in climate regulation, are the main medium of world commercial trade and are a source of renewable energy, to cite few aspects. Despite its importance, ocean surface state reconstruction poses some challenges, due to its non-linear behavior and the heterogeneity of the spatio-temporal scales involved. State-of-the-art techniques for forecast and prediction involve numerical weather models, such as data assimilation approaches. Besides, remote sensing techniques deliver finer-grained information about the surface state. Among others, underwater passive acoustics uses the underwater soundscape to infer the above-surface atmospheric state. In this work, with a particular focus on the surface wind speed reconstruction, we propose a framework that bridges data assimilation and machine learning schemes, to exploit both the prior physical knowledge and the capability of machine learning modelling to take advantage of large data bases. Extensive numerical experiments show that this hybrid framework can outperform the state-of-the-art data-driven models with a relative gain up to 16% in terms of root mean squared error. Experiments also involve tests on multi-modal data, namely underwater passive acoustics and wind speed reanalyses, giving promising results.