An interesting way to evaluate the representations obtained with machine learning language models is to compared them with human brain recordings. Encoding models have been used to partially predict fMRI recordings of different areas given the features of language models (such as Transformers). However, these models still lack long-term cognitive plausibility as well as insights on the underlying neural substrate mechanisms: e.g. how their representations differ across model layer depth and longer contexts. We study the influence of context representations of different language models such as sequence-based models: Long short-term memory networks (LSTMs), ELMo, and a popular pretrained Transformer language model (Longformer). In particular, we study how the internal hidden representations of such models are aligned with the fMRI brain activity. We use fMRI recordings of subjects listening to narrative stories to interpret word and sequence embedding representations. We further investigate how the representations of language model layers reveal better semantic context during listening. One of the novelties is that we look at several hidden states of LSTMs: cell and output gate states. Our computational experiments provide the following cognitive insights: (i) LSTM cell states are better aligned with brain recordings than LSTM output gate states: the cell state activity can represent more long- term information; (ii) the representations of ELMo and Longformer display a good predictive performance across brain regions for listening stimuli; (iii) Posterior Medial Cortex (PMC), Temporo-Parieto-Occipital junction (TPOJ) and Dorsal Frontal Lobe (DFL) have higher correlation versus Early Auditory (EAC) and Auditory Association Cortex (AAC).