Whittle Index Policy for Crawling Ephemeral Content - Inria - Institut national de recherche en sciences et technologies du numérique
Article Dans Une Revue IEEE Transactions on Control of Network Systems Année : 2018

Whittle Index Policy for Crawling Ephemeral Content

Résumé

We consider the task of scheduling a crawler to retrieve from several sites their ephemeral content. This is content, such as news or posts at social network groups, for which a user typically loses interest after some days or hours. Thus development of a timely crawling policy for ephemeral information sources is very important. We first formulate this problem as an optimal control problem with average reward. The reward can be measured in terms of the number of clicks or relevant search requests. The problem in its exact formulation suffers from the curse of dimensionality and quickly becomes intractable even with moderate number of information sources. Fortunately, this problem admits a Whittle index, a celebrated heuristics which leads to problem decomposition and to a very simple and efficient crawling policy. We derive the Whittle index for a simple deterministic model and provide its theoretical justification. We also outline an extension to a fully stochastic model.
Fichier principal
Vignette du fichier
root.pdf (373.02 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01937994 , version 1 (28-11-2018)

Identifiants

Citer

Konstantin Avrachenkov, Vivek S. Borkar. Whittle Index Policy for Crawling Ephemeral Content. IEEE Transactions on Control of Network Systems, 2018, 5 (1), pp.446 - 455. ⟨10.1109/TCNS.2016.2619066⟩. ⟨hal-01937994⟩
86 Consultations
129 Téléchargements

Altmetric

Partager

More