Technology is often not accessible to older adults, especially to those with low digital literacy or cognitive impairment. One premise of participatory design is that involving stakeholders including potential users during the whole process of design and development can result in solutions that are more accessible and make more sense to a target population. However, involving older adults in the design process is not straightforward, especially when they have little or no experience with information technology or some form of cognitive impairment, such as early stages of dementia. We investigate how to facilitate the participation of older adults with and without cognitive impairments in the phase of low-fidelity prototyping. We report on participatory design activities conducted in a non-governmental home for older adults with low socio-economic status and present lessons learned and challenges for planning and conducting participatory design that complement the literature in this subject area. For example, participants showed they are capable of some level of abstraction, although literature indicates that older adults with cognitive impairments have difficulties in abstract thinking.