This paper describes an evaluation of a mobile exertion application, developed to promote physical activity among adult users. To extend previous work, we aimed to identify changes in walking behavior over time, and to determine the efficacy of non-visual cues presented via the phone. Data was gathered using three methods: automated logging, participant-maintained diaries and interviews. Findings revealed that participants were able to respond most effectively to multimodal icons to modify their pace over a two week period. However, their preferences were noted to change depending on the situational context. The study has also highlighted the complexities associated with data collection techniques for mobile evaluations.