Echocardiography plays an important role in the diagnosis of cardiac dysfunction. In particular, motion estimation in echocardiography is a challenging task since ultrasound images suffer largely from low signal-to-noise ratio and out-of-view problems. Current deep learningbased models for cardiac motion estimation in the literature estimate the dense motion field with spatial regularization. However, the underlying spatial regularization can only cover a very small region in the neighborhood, which is not enough for a smooth and realistic motion field for the myocardium in echocardiography. In order to improve the performance and quality with deep learning networks, we propose applying polyaffine transformation for motion estimation, which intrinsically regularizes the myocardium motion to be polyaffine. Our thorough experiments demonstrate that the proposed method not only presents better evaluation metrics on the registration of cardiac structures but also shows great potential in abnormal wall motion detection.