Song acquisition and production in songbirds is governed by a dedicated neural circuitry that involves two parallel pathways: a motor pathway for the production and a basal ganglia (BG) pathway for the acquisition. Juveniles learn by imitating adult vocalizations and proceed by trial and error, errors being conveyed by a dopaminergic signal. The complex nature of the relationship between neural control and syrinx musculature makes song learning a complicated problem to solve. Reinforcement learning (RL) has been widely hypothesized to underlie such sensorimotor learning even though this can lead to sub-optimal solutions under uneven contours in continuous action spaces. In this article, we propose to re-interpret the role of a dual pathway architecture, underlying avian vocal learning, that helps overcome these limitations. We posit that the BG pathway conducts exploration by inducing large daily shifts in the vocal production while the motor pathway gradually consolidates this exploration. This process can be understood as a modified form of a simulated annealing process. Simulations on Gaussian performance landscapes and a syrinx-based performance landscape are demonstrated and compared with standard approaches. Taking behavioral constraints into account (60 days of learning, 1000 trials per day), the model allows to reach the global optimum in complex landscapes and thus provides a sound insight into the role of the dual pathway architecture underlying vocal learning.