Hierarchical B-splines that allow local refinement have become a promising tool for developing adaptive isogeometric methods. Unfortunately, similar to tensor-product B-splines, the computational cost required for assembling the system matrices in isogeometric analysis with hierarchical B-splines is also high, particularly if the spline degree is increased. To address this issue, we propose an efficient matrix assembly approach for bivariate hierarchical B-splines based on the previous work [42]. The new algorithm consists of three stages: approximating the integrals by quasi-interpolation, building three compact look-up tables and assembling the matrices via sum-factorization. A detailed analysis shows that the complexity of our method has the order O(N p 3) under a mild assumption about mesh admissibility, where N and p denote the number of degrees of freedom and spline degree respectively. Finally, several experimental results are demonstrated to verify the theoretical results and to show the performance of the proposed method.