Not So Creepy Crawler: Crawling the Web with Xquery

Presented at: 3rd Future Internet Symposium (FIS2010)

by Franziska von Dem Bussche, Klara Weiand, Benedikt Linse, Tim Furche, Fran├žois Bry

Web crawlers are increasingly used for focused tasks such as the extraction of data from Wikipedia or the analysis of social networks like In these cases, pages are far more uniformly structured than in the general Web and thus crawlers can use the structure of Web pages for more precise data extraction and more expressive analysis. In this demonstration, we present a focused, structure-based crawler generator, the "Not so Creepy Crawler" (nc2). What sets nc2 apart, is that all analysis and decision tasks of the crawling process are delegated to an (arbitrary) XML query engine such as XQuery or Xcerpt. Customizing crawlers just means writing (declarative) XML queries that can access the currently crawled document as well as the metadata of the crawl process. We identify four types of queries that together suffice to realize a wide variety of focused crawlers.

Keywords: Semantic Web

Resource URI on the dog food server:

Explore this resource elsewhere: