We investigate different ways of generating approximate solutions to the inverse Ising problem. Our approach consists in to take as a starting point for further perturbation procedures, a Bethe mean-field solution obtained with a maximum spanning tree of pairwise mutual information which we refer to as the "Bethe reference point". We consider three different ways of following this idea: in the first one, we discuss a greedy procedure by which optimal links to be added starting from the Bethe reference point are selected and calibrated iteratively; the second one is based on the observation that the natural gradient can be computed analytically at the Bethe point; the last one deals with loop corrections to the Bethe point. Assuming no external field and using a dual transform we develop a dual loop joint model based on a well-chosen cycle basis. This leads us to identify a subclass of planar models, which we refer to as \emph{dual-loop-free models}, having possibly many loops, but characterized by a singly connected dual factor graph, for which the partition function and the linear response can be computed exactly in respectively O(N) and O(N^2) operations, thanks to a dual weight propagation message passing procedure that we set up. When restricted to this subclass of models, the inverse Ising problem being convex, becomes tractable at any temperature. Numerical experiments show that this can serve to some extent as a good approximation for models with dual loops.