We introduce a computational framework that allows a machine to bootstrap flexible autonomous learning of speech recognition skills. Technically, this framework shall en- able a robot to incrementally learn to recog- nize speech invariants from unsegmented au- dio streams and with no prior knowledge of phonetics. To achieve this, we import the bag-of-words/bag-of-features approach from recent research in computer vision, and adapt it to incremental developmental speech pro- cessing. We evaluate an implementation of this framework on a complex speech database.