In this work we extend our recently proposed adaptive refinement strategy for $hp$-finite element approximations of elliptic problems by taking into account an inexact algebraic solver. Namely, on each level of refinement and on each iteration of an (arbitrary) iterative algebraic solver, we compute guaranteed a posteriori error bounds on the algebraic and the total errors in energy norm. For the algebraic error upper bound, we crucially exploit the nested hierarchy of $hp$-finite element spaces created throughout the adaptive algorithm, whereas the rest of the components of the total error upper and lower bounds are computed using the finest space only. These error bounds allow us to formulate adaptive stopping criteria for the algebraic solver ensuring that the algebraic error does not significantly contribute to the total error. Next, we use the total error bound to mark mesh vertices for refinement via D\"{o}rfler's bulk-chasing criterion. On patches associated with marked vertices only, we solve two separate primal finite element problems with homogeneous Dirichlet (Neumann) boundary conditions, which serve to decide between $h$-, $p$-, or $hp$-refinement. Altogether, we show that these ingredients lead to a computable guaranteed bound on the ratio of the total errors of the inexact approximations between successive refinements (the error reduction factor), when the stopping criteria are satisfied. Finally, in a series of numerical experiments, we investigate the practicality of the proposed adaptive solver, the accuracy of our bound on the reduction factor, and show that exponential convergence rates are also achieved even in the presence of an inexact algebraic solver.