Solving Multi-Agent Reinforcement Learning Problems is a key issue. Indeed, the complexity of deriving multi-agent plans, especially when one uses an explicit model of the problem, is dramatically increasing with the number of agents. This papers introduces a general iterative heuristic: at each step one chooses a sub-group of agents and update their policies to optimize the task given the rest of agents have fixed plans. We analyse this process in a general purpose and show how it can be applied to Markov Decision Processes, Partially Observable Markov Decision Processes and Decentralized Partially Observable Markov Decision Processes.