Low voltage operation and small device sizes reduce the critical charge stored in a SRAM cell making caches more susceptible to soft errors. To counter the problem, a plethora of protection schemes are being proposed to reduce the power and area overheads of cache protection. Accurate methods to measure and compare the effectiveness of such schemes are sorely needed. This paper introduces a unified reliability benchmarking framework with PARMA (Precise Analytical Reliability Model for Architecture). PARMA is a rigorous analytical framework to measure the failure rate in the presence of soft errors under any protection scheme. PARMA continuously and accurately accounts for the increase of failure density due to soft errors affecting program behavior during an execution. We have implemented the PARMA framework on top of a cycle-accurate simulator to benchmark Level-2 cache failure rates for a set of CPU 2000 benchmarks. Three protection schemes are compared: parity, word-level 1-bit ECC and block-level 1-bit ECC.