In recent years, several works have studied Multi-Agent Deep Reinforcement Learning for the Distributed Packet Routing problem, with promising results in various scenarios where network status changes dynamically, is uncertain, or is partially hidden (e.g., wireless ad hoc networks or wired multidomain networks). Unfortunately, these previous works focus on an ideal scenario where the impact of control signalling is neglected, and network simulation is tailored to simplistic assumptions. In this article, we present the first experimental investigation of control signalling mechanisms for distributed learning-based packet routing. We rely on PRISMA, our opensource simulation ns-3-based module. We formulate two signalling mechanisms between agents (value sharing and model sharing). We investigate the net gains considering in-band signalling and show that routing policies close to those provided by an oracle with full knowledge of traffic and network topology can be discovered with a control overhead of 150 % with respect to injected data packets, if neighboring agents share their Deep Neural Network models. We discuss the generality of our results to underline the importance of assessing net gains of Multi-Agent Deep Reinforcement Learning (MA-DRL)-based routing.