We investigate the idea of having groups of agents coevolving in order to iteratively refine multi-agent plans. This idea we called coevolution is formalized and analyzed in a general purpose and applied to the stochastic control frameworks that use an explicit model of the world\,: coevolution can directly be adapted to the frameworks of Multi-Agent Markov Decision Processes (MMDP) and Multi-Agent Partially Observable MDP (MPOMDP). We also consider the decentralized version of MPOMDP (DEC-POMDP) which is known to be a difficult problem\,: we show that the coevolution approach can be applied if we restrict the search to memoryless policies. We evaluate our coevolutive approach experimentally on a typical multi-agent problem.