Recently, due to the potential of machine learning approaches and the increased awareness of privacy risks, there is an increased interest in privacy-preserving machine learning. However, real-world medical applications are often complex. Our research, therefore, focuses on two objectives: we want to develop algorithms that make privacy-preserving machine learning more interpretable for non-experts and which automatically optimize the parameters and strategy of a machine learning solution to respect privacy requirements while optimizing utility, i.e., maximizing precision and minimizing cost. At its core, our methodology is based on a constraint programming approach. It is known that non-experts can relatively easily learn to express requirements in the form of constraints. At the same time, this approach allows us to use a wide range of publicly available constraint program solvers.