This paper presents a novel set of design challenges for the development of mobile and wearable applications to be used at cultural heritage sites. These challenges were drawn out based on a user study that was carried out to evaluate a mobile application prototype, SmartC. SmartC was designed for supporting people in taking learning opportunities at sites whenever they need informally while they are on the move. Augmented reality and wearable computing, i.e. smart eye glasses, were used in this research with the aim of bringing the past to life, as well as enhancing visitors’ engagement. SmartC was evaluated by 26 participants, potential end-users, in the field. The evaluation study mainly focused on the interaction and usability aspects, which contribute to the field of HCI. The paper outlines several issues and challenges that were identified based on the evaluation study, summarised as: (1) interaction design related; (2) wearable computing related; (3) surroundings and environment related; (4) learner related; (5) context of use; and (6) technical issues. This paper also identifies aspects that relate to methods to be used and applied to such cases for evaluation studies.