The power output of a wind farm is influenced by wake effects, a phenomenon in which upstream turbines facing the wind create sub-optimal conditions for turbines located downstream. Yaw misaligning strategies have been shown to increase total production. Yet designing efficient methods of cooperative control to find optimal yaw angles is a challenging task. Classical optimization methods become intractable as the size of the farm grows, do not recover from model inaccuracies and ignore the dynamic propagation of the wind inflow in real conditions. Reinforcement learning methods can provide a model-free alternative, but raise issues of scalability when the control is centralized. Existing decentralized RL methods have been shown to significantly increase power production under dynamic conditions, but relied on tabular methods with state and action space discretization. To accelerate convergence, we employ an actor-critic algorithm with linear function approximation for decentralized cooperative yaw control. We validate our method in dynamic simulators for wind farms with up to 32 turbines, and show empirically that, compared to previous tabular algorithms, our method is faster and scales to larger wind farms.