edu.uci.ics.crawler4j.crawler

Class WebCrawler

java.lang.Object

edu.uci.ics.crawler4j.crawler.WebCrawler

All Implemented Interfaces:

Runnable

public class WebCrawler
extends Object
implements Runnable

WebCrawler class in the Runnable class that is executed by each crawler thread.

Author:

Yasser Ganjisaffar

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	logger
protected CrawlController	myController The controller instance that has created this crawler thread.
protected int	myId The id associated to the crawler thread running this instance

Constructor Summary

Constructors

Constructor and Description
WebCrawler()

Method Summary

Modifier and Type	Method and Description
CrawlController	getMyController()
int	getMyId() Get the id of the current crawler instance
Object	getMyLocalData() The CrawlController instance that has created this crawler instance will call this function just before terminating this crawler thread.
Thread	getThread()
protected void	handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) This function is called once the header of a page is fetched.
protected WebURL	handleUrlBeforeProcess(WebURL curURL) This function is called before processing of the page's URL It can be overridden by subclasses for tweaking of the url before processing it.
void	init(int id, CrawlController crawlController) Initializes the current instance of the crawler
boolean	isNotWaitingForNewURLs()
void	onBeforeExit() This function is called just before the termination of the current crawler instance.
protected void	onContentFetchError(WebURL webUrl) This function is called if the content of a url could not be fetched.
protected void	onPageBiggerThanMaxSize(String urlStr, long pageSize) This function is called if the content of a url is bigger than allowed size.
protected void	onParseError(WebURL webUrl) This function is called if there has been an error in parsing the content.
protected void	onRedirectedStatusCode(Page page) This function is called if the crawler encounters a page with a 3xx status code
void	onStart() This function is called just before starting the crawl by this crawler instance.
protected void	onUnexpectedStatusCode(String urlStr, int statusCode, String contentType, String description) This function is called if the crawler encountered an unexpected http status code ( a status code other than 3xx)
protected void	onUnhandledException(WebURL webUrl, Throwable e) This function is called when a unhandled exception was encountered during fetching
void	run()
void	setThread(Thread myThread)
protected boolean	shouldFollowLinksIn(WebURL url) Determine whether links found at the given URL should be added to the queue for crawling.
boolean	shouldVisit(Page referringPage, WebURL url) Classes that extends WebCrawler should overwrite this function to tell the crawler whether the given url should be crawled or not.
void	visit(Page page) Classes that extends WebCrawler should overwrite this function to process the content of the fetched and parsed page.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

myId

protected int myId

The id associated to the crawler thread running this instance

myController

protected CrawlController myController

The controller instance that has created this crawler thread. This reference to the controller can be used for getting configurations of the current crawl or adding new seeds during runtime.

Constructor Detail

WebCrawler

public WebCrawler()

Method Detail

init

public void init(int id,
                 CrawlController crawlController)
          throws InstantiationException,
                 IllegalAccessException

Initializes the current instance of the crawler

Parameters:

id - the id of this crawler instance

crawlController - the controller that manages this crawling session

Throws:

IllegalAccessException

InstantiationException

getMyId

public int getMyId()

Get the id of the current crawler instance

Returns:

the id of the current crawler instance

getMyController

public CrawlController getMyController()

onStart

public void onStart()

This function is called just before starting the crawl by this crawler instance. It can be used for setting up the data structures or initializations needed by this crawler instance.

onBeforeExit

public void onBeforeExit()

This function is called just before the termination of the current crawler instance. It can be used for persisting in-memory data or other finalization tasks.

handlePageStatusCode

protected void handlePageStatusCode(WebURL webUrl,
                                    int statusCode,
                                    String statusDescription)

This function is called once the header of a page is fetched. It can be overridden by sub-classes to perform custom logic for different status codes. For example, 404 pages can be logged, etc.

Parameters:

webUrl - WebUrl containing the statusCode

statusCode - Html Status Code number

statusDescription - Html Status COde description

handleUrlBeforeProcess

protected WebURL handleUrlBeforeProcess(WebURL curURL)

This function is called before processing of the page's URL It can be overridden by subclasses for tweaking of the url before processing it. For example, http://abc.com/def?a=123 - http://abc.com/def

Parameters:

curURL - current URL which can be tweaked before processing

Returns:

tweaked WebURL

onPageBiggerThanMaxSize

protected void onPageBiggerThanMaxSize(String urlStr,
                                       long pageSize)

This function is called if the content of a url is bigger than allowed size.

Parameters:

urlStr - - The URL which it's content is bigger than allowed size

onRedirectedStatusCode

protected void onRedirectedStatusCode(Page page)

This function is called if the crawler encounters a page with a 3xx status code

Parameters:

page - Partial page object

onUnexpectedStatusCode

protected void onUnexpectedStatusCode(String urlStr,
                                      int statusCode,
                                      String contentType,
                                      String description)

This function is called if the crawler encountered an unexpected http status code ( a status code other than 3xx)

Parameters:

urlStr - URL in which an unexpected error was encountered while crawling

statusCode - Html StatusCode

contentType - Type of Content

description - Error Description

onContentFetchError

protected void onContentFetchError(WebURL webUrl)

This function is called if the content of a url could not be fetched.

Parameters:

webUrl - URL which content failed to be fetched

onUnhandledException

protected void onUnhandledException(WebURL webUrl,
                                    Throwable e)

This function is called when a unhandled exception was encountered during fetching

Parameters:

webUrl - URL where a unhandled exception occured

onParseError

protected void onParseError(WebURL webUrl)

This function is called if there has been an error in parsing the content.

Parameters:

webUrl - URL which failed on parsing

getMyLocalData

public Object getMyLocalData()

The CrawlController instance that has created this crawler instance will call this function just before terminating this crawler thread. Classes that extend WebCrawler can override this function to pass their local data to their controller. The controller then puts these local data in a List that can then be used for processing the local data of crawlers (if needed).

Returns:

currently NULL

run

public void run()

Specified by:

run in interface Runnable

shouldVisit

public boolean shouldVisit(Page referringPage,
                           WebURL url)

Classes that extends WebCrawler should overwrite this function to tell the crawler whether the given url should be crawled or not. The following default implementation indicates that all urls should be included in the crawl.

Parameters:

url - the url which we are interested to know whether it should be included in the crawl or not.

referringPage - The Page in which this url was found.

Returns:

if the url should be included in the crawl it returns true, otherwise false is returned.

shouldFollowLinksIn

protected boolean shouldFollowLinksIn(WebURL url)

Determine whether links found at the given URL should be added to the queue for crawling. By default this method returns true always, but classes that extend WebCrawler can override it in order to implement particular policies about which pages should be mined for outgoing links and which should not. If links from the URL are not being followed, then we are not operating as a web crawler and need not check robots.txt before fetching the single URL. (see definition at http://www.robotstxt.org/faq/what.html). Thus URLs that return false from this method will not be subject to robots.txt filtering.

Parameters:

url - the URL of the page under consideration

Returns:

true if outgoing links from this page should be added to the queue.

visit

public void visit(Page page)

Classes that extends WebCrawler should overwrite this function to process the content of the fetched and parsed page.

Parameters:

page - the page object that is just fetched and parsed.

getThread

public Thread getThread()

setThread

public void setThread(Thread myThread)

isNotWaitingForNewURLs

public boolean isNotWaitingForNewURLs()