This paper presents ACKTUS, a modeling tool for developing knowledge-based systems for the health domain, and an evaluation study of the system. The main purpose of the evaluation was to investigate whether the functionality and interaction design of ACKTUS was sufficiently intuitive for the domain experts to contribute with knowledge and to model the interaction design of the three end users’ applications. Another purpose was to evaluate the applicability of the activity assessment protocol AAIMA for analysis. The study design was qualitative and formative, using observations and interviews with users to collect data. Three medical experts and two experts in occupational therapy participated, providing expertise in four different domains. The participants increased their understanding and skills during the evaluation period leading to improved knowledge-based applications. The AAIMA protocol proved to be useful and the results are fed into ongoing development work on developing the adaptive functionality of the ACKTUS systems.