Introduction: A BCI technology can operate in 3 different modalities: online mode which requires analyzing the new real-time EEG data while acquiring it, offline mode where data is acquired and saved to a file and then analyzed afterwards (giving access to the data as a whole) and pseudo-online mode, which is a mix between the previous two modes, where stored acquired data is processed as if in online mode, but with the relaxation of the real time constraint. Currently, many studies concerning Brain Computer Interfaces (BCI) are tested in the offline mode. This thus leads to unrealistic performance compared to real-life online scenarios. The MOABB framework typically provides tools to evaluate algorithms in this offline mode. Other studies propose online algorithms evaluation, but often do not disclose the datasets and/or nor the code used for data analysis. There are other frameworks for online processing, but they do not focus on the statistical evaluation over several sessions/subjects as MOABB does. Material, Methods and Results: The objective of this research is to extend the current MOABB framework, which is currently limited to offline mode to allow comparison of different algorithms in a pseudo-online setting. We focus on asynchronous BCI where data is typically analyzed in overlapping sliding windows. This requires the addition of an idle state event to the datasets to mark signal pieces not related to an actual BCI task(s). Doing so generates datasets that are usually highly unbalanced in favor of this idle event, generating problems with some of the standard metrics used in BCI evaluation. We thus use the normalized Matthews Correlation Coefficient (nMCC) and the Information Transfer Rate (ITR). We applied this pseudo-online framework to evaluate the state-of-the-art algorithms over the last 15 years over several Motor Imagery (MI) datasets composed by several subjects. Discussion: Usually offline modality set an upper bound to the performances, while a online signal analysis approaches generally produce results that are less accurate but more representative of a therapeutic application usage. The pseudo-online implementation can be used as a methodology that best approximates the online process while still processing the data after complete recording. It still represents an upper bound on performance (as real time time is not required) but a more realistic one that can be reached with more powerful computing resources. Significance: The possibility of analyzing the performance of different algorithms first offline, followed by subsequent validation of performance in pseudo-online mode, will be enable more representative reports on the performance of classification algorithms for the BCI community.