Partial encryption is often used as a tradeoff between security and performance to protect scalable video data. In this paper, we argue that although partial encryption is strong enough for access control, it is not adequate for content confidentiality protection. We conduct experiments to show that partially encrypted H.264/SVC (scalable video coding) streams leak significant content information from the enhancement layers in all three scalability dimensions. Our analysis concludes that such leakage is caused by the underlying coding techniques used in H.264/SVC, and all layers should be encrypted to protect confidential video streams.