RULKKG: Estimating User’s Knowledge Gain in Search-as-Learning Using Knowledge Graphs

In the context of search as learning, users engage in search sessions to fill their information gaps and achieve their learning goals. Tracking the user’s state of knowledge is therefore essential for estimating how close they are to achieve these learning goals. In this respect, we extend a recently proposed approach that uses the recognition of entities present in the text to track the user’s knowledge. Our approach introduces a more complete representation by considering both the entities and their relations. More precisely, we represent both the user’s knowledge and the user’s learning goals (or target knowledge) as knowledge graphs. We show that the proposed representation captures a complementary aspect of knowledge, thus helping to improve the user knowledge gain estimation when used in combination with other representations.


INTRODUCTION AND RELATED WORK
Understanding the reason(s) why a user is looking for information can help an Information Retrieval (IR) system to make the most appropriate answers to meet what is supposed to be the user's real need, especially in the context of Search-as-Learning (SAL).However, understanding the real needs of the user in search of information is not an easy task.Assuming that the user's query perfectly matches his or her needs is utopian.Belkin et al. in their seminal work [3] defined the concept of "Anomalous State of Knowledge (ASK)" as the state representing what is missing in the user's knowledge state that prevents them to be able to precisely formulate through a query, what they need.
Rose et al., who highlight the fact that search engines have focused much more on how the user looks for information (e.g.queries), than on the reasons why they look for it [29], proposed a framework to be used to understand the reasons for searching.The framework helps identifying and organizing the goals that users have in mind when they carry out a search.More recently, Shah et al. underlined the importance of the role a user's task plays in how and why they engage in a search, and what a search system should do to help them.
In particular, they stressed the fact that "the IR community has been too focused on query processing and assuming a search task to be a collection of user queries, often ignoring if or how such an assumption addresses the users accomplishing their tasks." They propose insights on understanding, extracting and addressing task-based search [30].Syed et al. who pointed out that Web search engines are best suited to provide results based on a generic concept of relevance, which is not suited to provide results for user's learning purposes.The optimization model they proposed, incorporates a cognitive learning model and the algorithm used proposes an approximate solution with results that represent the best "learning set" of pieces of information for an information-seeking user with the aim of learning [32].
We are convinced that it is possible to further improve the quality of the answers provided to the learning user.Indeed, in addition to knowing the reasons or goals of the user's search, it is also important to have an idea of the user's state of knowledge as well as the evolution of both the user's goals and knowledge state during the search process.Pirolli et al. introduced a personalized learning path model, which tracks the user's knowledge development through online interactions, using a social tagging strategy to predict relevant documents based on sets of recognized documents related to specific areas of interest [26].
The recent theoretical framework proposed by El Zein et al. in [18] uses a cognitive rule-based agent that adapts the document ranking based on both the user's current knowledge and search goal.Finally, the work on SAL presented by Gadiraju in [19], aims to improve the understanding of the knowledge acquired by users during the information search sessions.We share his point of view that the ability to estimate the state of the users' knowledge during their interactions with search systems can improve the users' learning experiences.However, tracking the learning process is not simple, as we need a representation of both the user's learning goals (or target knowledge) and state of knowledge, that captures the results of the user's interactions with the system.The recent work by El Zein et al. [6,17], represents a step forward in that direction.They consider a keyword-based representation of both the user's current state of knowledge as well as the user's learning goals, the aim of the user being to fill the gap between both through the interactions with the system.The framework then updates the user's knowledge state after the reading of each document, and estimates the similarity between the current state and the target knowledge [6].This work has been extended in [17] with the aim of capturing the information content from the named entities [5,25].However, the relations between those entities have not been considered, and as Shen et al. rightly pointed out in [31], considering the links between entities enables the entity mentioned in the text to be linked to the corresponding real-world entity in the existing knowledge base.
We fill this gap by proposing a knowledge graph representation of knowledge states.More precisely, we extend the named-entity based representation proposed in [17] and consider, in addition to the named entities, the relations between them.To validate our model, a dataset containing the users' actual knowledge gains and their interaction logs from an organized search session was used.The effectiveness of the model is demonstrated by the correlation measure between the model's estimated knowledge gain and the users' actual knowledge.
The paper is organized as follows.Section 2 presents the existing RULK version that we will extend in this paper.Section 3 describes the novel graph-based implementation of RULK, while Section 4 presents and discusses the experiments and results.Finally, Section 5 concludes the paper.

BACKGROUND
In this section, we introduce the RULK framework, explaining the primary functions of its three components in tracking the user's knowledge state throughout a search session with the aim of estimating their knowledge gain.Additionally, we provide a brief overview of the operation of its existing implementations.

Representing User Knowledge in
Search-As-Learning (RULK) The estimation of the user's knowledge gain has a significant importance in the Search as Learning domain.We share the point of view according to which such an estimation can help increasing the effectiveness of information retrieval systems by illustrating the value and impact of the retrieved information on users' understanding and learning.Indeed, by quantifying the knowledge gain, we can evaluate how well the system supports users in acquiring new knowledge and filling information gaps.This valuable information opens the opportunity to design personalized information retrieval systems that employ knowledge-adaptive approaches, thereby enhancing learning outcomes.RULK [6] is one of the most recently proposed frameworks that estimates the user's knowledge gain during a learning process.Its main goal is to estimate the user's knowledge by comparing two representations of knowledge: the user's target knowledge and the user's current knowledge.On the one hand, the target knowledge refers to the specific knowledge that a user aims to acquire or obtain through an information retrieval process.As in [6], we suppose that that the user's target knowledge, i.e., the user's actual need for each search topic, is the referenced Wikipedia document.On the other hand, it is assumed that the user's current knowledge is updated after reading documents, except the Wikipedia page, on a specific topic during the search session.RULK [6] consists of three main components: • The Feature Extractor (), which encodes any given document  into another representation, mainly a fixed-length vector.
Both the Wikipedia document and the clicked document are encoded by  in the same way.• The Updater (): the state of the user's knowledge is represented by a vector − →   .When a user clicks on a document, the  component combines − →   with − →   to obtain the updated • The Estimator ( ), estimates the degree of achievement of the user's learning goal as the similarity between the user's current knowledge − →   and the target knowledge from which the estimate of the user's knowledge gain can be computed.

RULK Versions
There are 3 versions of the RULK framework, each one using a different technique to encode the document into an embedding vector.
The first one,    , is based on a keyword and vocabulary learning model, the second one,    , is a transformer-based model, and finally, the last model,     , uses the Named Entity Recognition method.
adopts the vocabulary learning model [1].This model considers that a user achieves their need to learn a topic  when they learn a set of related vocabulary keywords.The keywords to be learned are defined using the Yet Another Keyword Extraction (YAKE) method [8].This method selects the most important keywords of a text relying on statistical feature extraction.
executes two natural language processing tasks, namely Named Entity Recognition and Entity Linking, to extract a set of entities, denoted as   , from the target knowledge document and another set of entities, denoted as   , from the clicked document.The  then transforms the target knowledge into a vector, denoted as −−→   , by counting the occurrences of the 10 most common entities in   [17].Similarly, the clicked document  is encoded as a vector by counting the occurrences of the top-10 entities in   within .
In all versions, the  updates the current knowledge state with a simple addition of two count vectors.Finally, the  uses the cosine similarity measure to compute the similarity between two vectors representing the current and target knowledge states.

A KNOWLEDGE-GRAPH-BASED IMPLEMENTATION OF RULK
We propose a novel instantiation of RULK, named RULK KG , using knowledge graphs (KG) to represent both the user and the target knowledge states.Knowledge graphs are large networks of entities, their semantic types, properties, and relationships between entities [21].The entities (concepts and instances) are the vertices of the graph; the relations between entities, which are the arcs of the graph, can be represented as triples (, , ), where  and  are entities and  is the name of a relation or property.
RULK KG is a knowledge-graph-based extension of the RULK NE [17] in the sense that, besides considering named entities, it also considers the relations between those entities.The purpose of Named Entity Recognition (NER) is to automatically extract and label specific named entities mentioned in a document.On the other hand, Relation Extraction (RE), focuses on identifying and extracting relationships or associations between these named entities.

Entity Recognition and Relation Extraction
Named Entity Recognition and Relation Extraction are crucial challenges for natural language processing research in general, and information extraction in particular.A named entity can be defined as a linguistic expression often associated, for example, with people (personalities), places and organizations.Named entity recognition plays a very important role in many information retrieval tasks [5,14,15,25,34].In particular, the pre-trained REBEL [5] model has demonstrated its prowess, by outperforming several times the state-of-the-art in the field.
REBEL is an end-to-end relation extraction model based on BART [22] and uses a sequence-to-sequence (Seq2Seq) architecture to extract over 200 distinct relation types.Its primary objective is identifying and extracting triplets (subject, predicate, object) from raw text.We use REBEL to implement the NER and RE tasks.Thus, our Feature Extractor component represents a document as a KG.The KG that represents the user's current knowledge is updated by the Updater each time the user clicks on a document by adding the triplets provided by the newly clicked document.

Similarity Measure
As we have already pointed out, the primary function of the Estimator component is to compute the similarity between two representations: the Target and the Current Knowledge State of the user.In the previous 3 versions of RULK, similarity computation relied on cosine similarity, wherein documents were transformed into vector representations.In our case, a document is represented by a KG.Therefore we consider three Graph-based measure methods to compute the similarity.

3.2.1
Graph-based measure.Measuring the similarity between two graphs refers to calculating the resemblance between their structural representations.It is a numerical indicator of how similar two graphs are.Several measures are commonly used for this purpose, and the choice often depends on the specific application.
We adopted one of the most commonly used graph-similarity measures, the Maximum Common Subgraph (MCS) [4,10,16,23].It relies on the largest shared structural information between two graphs.Suppose we have two graphs,  1 and  2 , each representing a network of interconnected nodes and edges.To retrieve their MCS, we need to identify the largest set of nodes and edges that are present in both.The MCS-based similarity between  1 and  2 is calculated using the following formula: where | ( 1 ,  2 )| is the number of nodes belonging to the maximum common subgraph,| 1 | and | 2 | are the number of nodes in graphs  1 and  2 respectively.The larger the common subgraph, the higher the similarity score.
Daoud et al. [9] used the WebJaccard (  ) similarity to compare two graphs, representing the user's query and profile respectively, in an attempt to detect the user's search boundary.Their application was based on the number of common concepts between both graphs.The WebJaccard similarity is in line with the perception of the "graph union" distance [33], which also involves the concept of intersection and union operations.
We use the   measure to quantify the similarity between the user's target and current knowledge based on the number of entities belonging to the triplets intersection between the addressed KGs representations.
Given two knowledge graphs  1 and  2 , the WebJaccard similarity is calculated as follows: where  ( 1 ∩ 2 ) is the number of entities belonging to the triplets in the intersection between  1 and  2 and  ( 1 ∪ 2 ) is the number of entities belonging to the triplets in the union between the two graphs.A similarity value of 1 means complete identity, while a similarity value of 0 means that there are no common elements between the two graphs.
Finally, we propose a triplet-based similarity measure.We define the target knowledge state as the repository of information relevant to the topic the user would like to learn about-it essentially represents the user's learning goal.The idea is that to reach the target knowledge, the users should click on documents allowing them to capture the content of the target knowledge.We would like to stress that once a triplet from the target knowledge is identical to a triplet in the current knowledge, we consider that the user successfully acquired this particular piece of the target knowledge.

𝑇 𝑟𝑖𝑝𝑙𝑒𝑡𝑆𝑖𝑚(𝐺
where   and   are, respectively, the target and the current knowledge graphs,   (  ,  ) represents the number of the common identical triplets shared between   and   , and    represents the total number of triplets in   .

3.2.2
Isomorphism vs Semantic Similarity.Two objects are considered to be isomorphic when they share exactly the same structure and possess identical properties.However, it is important to note that similarity between two graphs does not always require isomorphism, as it is the case for exact graph matching.Two graphs with only certain points in common are considered somehow similar, and this does not imply absolute identity in terms of structure and properties-inexact graph matching [13].Bunke et al. [4] share the same perspective.They doubt the possibility of finding a perfect correspondence between graphs and support the idea of using fault-tolerant algorithms to calculate graph similarity.The same view is also shared by Riesen et al. [28], who propose a comparative study of exact and inexact graph matching methods.
In this paper, we consider that each user aims to acquire as much information as possible on the target knowledge document relating to a specific topic.Users search to acquire this knowledge by reading various documents semantically related to it,but whose precise content is not necessarily identical.The idea is that documents containing different entities, synonyms, and phrases may nevertheless contain the same information.
Based on that concept, we extended our similarity measures to consider not only the identical information matching but also the semantic relatedness matching.In both MCS and WJ, we consider that any triplet in the current knowledge, containing a node or entity strongly semantically related to the search topic (i.e., with a semantic similarity of 80% or higher), should be considered part of the target knowledge (because it will help reach the target knowledge that represents the topic).We measure such a similarity using spaCy [12], a Python open-source natural language processing library providing a built-in method to compute the semantic similarity between tokens.The resulting semantic similarity score varies between 0 and 1, where 1 represents full synonymy.
The components of both formulas remain unchanged; however, we have introduced a new variable  _ in the numerator.This variable represents the count of semantically related entities to the search topic extracted from the user's current knowledge.Also, we added the  _ variable to the triplet-based similarity measure (Equation 9), which represents the semantically related triplets between KGs.

𝑇𝑟𝑖𝑝𝑙𝑒𝑡𝑆𝑖𝑚(𝐺
(9) To our knowledge, there is no direct approach in the literature for comparing the similarity between two triplets from different graphs.To fill this gap, we transform triplets into sentences.For example, the triplet "entity:Great Recession; relation:has cause; entity:Subprime Mortgage Crisis" is transformed into "The Great Recession has cause the Subprime Mortgage Crisis".Then, to measure the semantic similarity between two sentences representing two triplets, we used SpaCy.An 80% of semantic similarity threshold is adopted in our experiment.

EXPERIMENTS AND RESULTS
Our model was evaluated using the same dataset utilized in evaluating prior versions of RULK [7].Three implementations of our model were employed:    ( ) ,    ( ) , and    (  ) using triplet-based, , and   similarity measures, respectively, to estimate the knowledge gain.

Dataset
The dataset was obtained from a study focusing on search-aslearning sessions, involving the participation of 126 crowd-workers tasked with conducting searches about 7 chosen topics belonging to different domains.The search system they utilized in that study was built on top of SearchX [27], an Interactive Information Retrieval research framework.This dataset incorporates also the user's behavioral features such as query length, query count, click count, and session duration.Additionally, it includes details on the actual knowledge gain of each user, as assessed through two topic-related tests one before starting the search note   and the other one after finishing, note   .The difference between these knowledge gain measurements serves to quantify both Absolute Learning Gain  and Realized Potential Learning .Aligning with the methodologies of previous RULK experiments, we used  and  values to illustrate the actual knowledge gain of the user.Details of how to calculate these metrics can be found in [7].

Model Comparison
To validate the effectiveness of the framework, we examine the correlation between the estimated gain Ǧ and the user's actual knowledge quantified as  and .The outcomes presented in Table 1 indicate that our three implementations have outperformed the     version.That confirms our viewpoint, that is, adding the relation extraction task to named entity recognition enriches the knowledge representation, thus leading to better and improved estimation results.   ( ) exhibits slightly better correlation values than    (  ) , standing at 0.18 for  and . and   demonstrate similar behavior in considering common entities between two graphs.Notably,   incorporates the size of the union of the two graphs in the denominator, enabling both graphs' sizes to influence the measure.   ( ) exhibits the best result among our three implementations, showing a correlation score of 0.265 for  and 0.236 for .That reveals the significance of involving the entire structure of a graph's triplet in the similarity metric rather than prioritizing nodes over edges.
In general, the correlation results suggest that other factors not considered in the model may also contribute to the actual knowledge gain.For example, the study made by Yu et al. considered the users' behavioral features in order to predict their knowledge gain [35].

Model Combinations
Multiple implementations of RULK can be combined as an interpolated estimator where each  is a distinct implementation of RULK and the weights   , such that    = 1, are set so as to optimize the result.As suggested by previous work, combining knowledge-state representation models of different natures can boost the accuracy of knowledge gain prediction.That happens because each model covers a different and complementary aspect of knowledge.For instance, [17] pointed out that even though     is by far the worst in terms of gain prediction accuracy when used alone, the combinations using it outperform the combinations not using it.This observation underlines that the real interest of each model is its ability to capture aspects of knowledge that the other models fail to capture.
In this spirit, we now assess all potential model combinations, in order to determine to which degree the model we are proposing is capable of capturing an aspect of knowledge that the others do not, namely what we might call the relational aspect.We use the    ( ) implementation, which exhibits the best result among our three implementations.
Table 2 shows the ALG and RPL measures for all possible combinations of models.Although    ranks only third out of four when used solo, we can observe that: • the pair combinations can be ranked as • the all-model combination clearly outperforms all the other combinations and solo models, according to both measures, confirming the hypotheses that the four models are indeed complementary.

CONCLUSION
In this paper, we introduced    , a graph-based representation of the RULK framework designed for tracking a user's knowledge state during a web search session, with the aim of estimating the knowledge gain.When assessing the effectiveness of our model by measuring the correlation between the estimated and actual knowledge gain, it shows a higher correlation value compared to     .In addition, the results of the model combinations show that the knowledge-graph representation is complementary to the others and, in combination with them, helps obtain better correlations.
A limitation of our framework to be pointed out is the assumption that the whole document is considered when a user clicks on it, which ignores the fact that users often only read parts of documents.One solution to this problem could be to take into account the user's eye movements [2,20].
Future work might include identifying a method to learn the user's target knowledge during a search session, providing the opportunity to evaluate   framework against other models across diverse datasets that encompass different types of knowledge assessment tests, such as [24].

Table 1 :
Pearson correlation between the estimated knowledge gain of RULK versions and the actual knowledge gain of the users.

Table 2 :
Pearson correlation between the estimated knowledge gain of different RULK model combinations and the actual knowledge gain of the users.The combinations are sorted by RPL correlation.    0.3228 0.3309    +     0.3184 0.3333    +    +    0.3332 0.3333    +    +     0.3378 0.3490    +     +    0.35020 0.34752    +    +     0.35465 0.3555    +    +     +    0.36044 0.36163 that KG tends to perform better in combination with LM or KW than in combination with NE; indeed, NE and KG focus on similar aspects of knowledge; • the trio combinations can be ranked as KW + KG + NE > LM + NE + KG > KW + LM + NE > KW + LM + KG according to  and as KW + KG + NE > KW + LM + NE > LM + NE + KG > KW + LM + KG according to ; both measures agree in ranking the combination of KG with KW and NE first; LM + KG > LM + NE > KW + NE > KW + KG > KW + LM > KG + NE according to  and as KW + NE > LM + NE > KW + KG > KW + LM > LM + KG > KG + NE according to , which suggests