In this work we extend our recently proposed adaptive refinement strategy for $hp$-finite element approximations of elliptic problems by taking into account an inexact algebraic solver. Namely, on each level of refinement and on each iteration of an (arbitrary) iterative algebraic solver, we compute guaranteed a posteriori error bounds on the algebraic and the total errors in energy norm. The algebraic error is the difference between the inexact discrete solution obtained by an iterative algebraic solver and the (unavailable) exact discrete solution. On the other hand, the total error stands for the difference between the inexact discrete solution and the (unavailable) exact solution of the partial differential equation. For the algebraic error upper bound, we crucially exploit the whole nested hierarchy of $hp$-finite element spaces created by the adaptive algorithm, whereas the remaining parts of the total error upper and lower bounds are computed using the finest space only. These error bounds allow us to formulate adaptive stopping criteria for the algebraic solver ensuring that the algebraic error does not significantly contribute to the total error. Next, we use the total error bound to mark mesh vertices for refinement via D\"orfler's bulk-chasing criterion. On patches associated with marked vertices only, we solve two separate primal finite element problems with homogeneous Dirichlet (Neumann) boundary conditions, which serve to decide between $h$-, $p$-, or $hp$-refinement. Altogether, we show that these ingredients lead to a computable guaranteed bound on the ratio of the total errors of the inexact approximations between successive refinements (the error reduction factor), when the stopping criteria are satisfied. Finally, in a series of numerical experiments, we investigate the practicality of the proposed adaptive solver, the accuracy of our bound on the reduction factor, and show that exponential convergence rates are achieved even in the presence of an inexact algebraic solver.