The work presented outlines my doctoral research in emotion and action based human computer interfaces, with the approach to create an open source framework. The idea is to find the suitable abstractions and interfaces to integrate different approaches to process input data to interpret user’s emotions and actions. Fusing and comparing alternatives for processing will help to find the appropriate solution for a particular need for the next generation of interactive systems. Widening the sensing abilities of computers has the potential to add new degrees of freedom to the design of human computer interaction and ubiquitous entertainment computing.