edu.uci.ics.crawler4j.crawler

Class CrawlController

java.lang.Object

edu.uci.ics.crawler4j.crawler.Configurable

edu.uci.ics.crawler4j.crawler.CrawlController

public class CrawlController
extends Configurable

The controller that manages a crawling session. This class creates the crawler threads and monitors their progress.

Author:

Yasser Ganjisaffar

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	CrawlController.WebCrawlerFactory<T extends WebCrawler>

Field Summary

Fields

Modifier and Type	Field and Description
protected List<Object>	crawlersLocalData Once the crawling session finishes the controller collects the local data of the crawler threads and stores them in this List.
protected Object	customData The 'customData' object can be used for passing custom crawl-related configurations to different components of the crawler.
protected DocIDServer	docIdServer
protected com.sleepycat.je.Environment	env
protected boolean	finished Is the crawling of this session finished?
protected Frontier	frontier
protected PageFetcher	pageFetcher
protected RobotstxtServer	robotstxtServer
protected boolean	shuttingDown Is the crawling session set to 'shutdown'.
protected Object	waitingLock

Fields inherited from class edu.uci.ics.crawler4j.crawler.Configurable

config

Constructor Summary

Constructors

Constructor and Description
CrawlController(CrawlConfig config, PageFetcher pageFetcher, RobotstxtServer robotstxtServer)

Method Summary

Modifier and Type	Method and Description
void	addSeed(String pageUrl) Adds a new seed URL.
void	addSeed(String pageUrl, int docId) Adds a new seed URL.
void	addSeenUrl(String url, int docId) This function can called to assign a specific document id to a url.
List<Object>	getCrawlersLocalData() Once the crawling session finishes the controller collects the local data of the crawler threads and stores them in a List.
Object	getCustomData()
DocIDServer	getDocIdServer()
Frontier	getFrontier()
PageFetcher	getPageFetcher()
RobotstxtServer	getRobotstxtServer()
boolean	isFinished()
boolean	isShuttingDown()
void	setCustomData(Object customData)
void	setDocIdServer(DocIDServer docIdServer)
void	setFrontier(Frontier frontier)
void	setPageFetcher(PageFetcher pageFetcher)
void	setRobotstxtServer(RobotstxtServer robotstxtServer)
void	shutdown() Set the current crawling session set to 'shutdown'.
protected static void	sleep(int seconds)
<T extends WebCrawler> void	start(Class<T> clazz, int numberOfCrawlers) Start the crawling session and wait for it to finish.
<T extends WebCrawler> void	start(CrawlController.WebCrawlerFactory<T> crawlerFactory, int numberOfCrawlers) Start the crawling session and wait for it to finish.
protected <T extends WebCrawler> void	start(CrawlController.WebCrawlerFactory<T> crawlerFactory, int numberOfCrawlers, boolean isBlocking)
<T extends WebCrawler> void	startNonBlocking(Class<T> clazz, int numberOfCrawlers) Start the crawling session and return immediately.
<T extends WebCrawler> void	startNonBlocking(CrawlController.WebCrawlerFactory<T> crawlerFactory, int numberOfCrawlers) Start the crawling session and return immediately.
void	waitUntilFinish() Wait until this crawling session finishes.

Methods inherited from class edu.uci.ics.crawler4j.crawler.Configurable

getConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

customData

protected Object customData

The 'customData' object can be used for passing custom crawl-related configurations to different components of the crawler.

crawlersLocalData

protected List<Object> crawlersLocalData

Once the crawling session finishes the controller collects the local data of the crawler threads and stores them in this List.

finished

protected boolean finished

Is the crawling of this session finished?

shuttingDown

protected boolean shuttingDown

Is the crawling session set to 'shutdown'. Crawler threads monitor this flag and when it is set they will no longer process new pages.

pageFetcher

protected PageFetcher pageFetcher

robotstxtServer

protected RobotstxtServer robotstxtServer

frontier

protected Frontier frontier

docIdServer

protected DocIDServer docIdServer

waitingLock

protected final Object waitingLock

env

protected final com.sleepycat.je.Environment env

Constructor Detail

CrawlController

public CrawlController(CrawlConfig config,
                       PageFetcher pageFetcher,
                       RobotstxtServer robotstxtServer)
                throws Exception

Throws:

Exception

Method Detail

start

public <T extends WebCrawler> void start(Class<T> clazz,
                                         int numberOfCrawlers)

Start the crawling session and wait for it to finish. This method utilizes default crawler factory that creates new crawler using Java reflection

Type Parameters:

T - Your class extending WebCrawler

Parameters:

clazz - the class that implements the logic for crawler threads

numberOfCrawlers - the number of concurrent threads that will be contributing in this crawling session.

start

public <T extends WebCrawler> void start(CrawlController.WebCrawlerFactory<T> crawlerFactory,
                                         int numberOfCrawlers)

Start the crawling session and wait for it to finish.

Type Parameters:

T - Your class extending WebCrawler

Parameters:

crawlerFactory - factory to create crawlers on demand for each thread

numberOfCrawlers - the number of concurrent threads that will be contributing in this crawling session.

startNonBlocking

public <T extends WebCrawler> void startNonBlocking(CrawlController.WebCrawlerFactory<T> crawlerFactory,
                                                    int numberOfCrawlers)

Start the crawling session and return immediately.

Type Parameters:

T - Your class extending WebCrawler

Parameters:

crawlerFactory - factory to create crawlers on demand for each thread

numberOfCrawlers - the number of concurrent threads that will be contributing in this crawling session.

startNonBlocking

public <T extends WebCrawler> void startNonBlocking(Class<T> clazz,
                                                    int numberOfCrawlers)

Start the crawling session and return immediately. This method utilizes default crawler factory that creates new crawler using Java reflection

Type Parameters:

T - Your class extending WebCrawler

Parameters:

clazz - the class that implements the logic for crawler threads

numberOfCrawlers - the number of concurrent threads that will be contributing in this crawling session.

start

protected <T extends WebCrawler> void start(CrawlController.WebCrawlerFactory<T> crawlerFactory,
                                            int numberOfCrawlers,
                                            boolean isBlocking)

waitUntilFinish

public void waitUntilFinish()

Wait until this crawling session finishes.

getCrawlersLocalData

public List<Object> getCrawlersLocalData()

Once the crawling session finishes the controller collects the local data of the crawler threads and stores them in a List. This function returns the reference to this list.

Returns:

List of Objects which are your local data

sleep

protected static void sleep(int seconds)

addSeed

public void addSeed(String pageUrl)

Adds a new seed URL. A seed URL is a URL that is fetched by the crawler to extract new URLs in it and follow them for crawling.

Parameters:

pageUrl - the URL of the seed

addSeed

public void addSeed(String pageUrl,
                    int docId)

Adds a new seed URL. A seed URL is a URL that is fetched by the crawler to extract new URLs in it and follow them for crawling. You can also specify a specific document id to be assigned to this seed URL. This document id needs to be unique. Also, note that if you add three seeds with document ids 1,2, and 7. Then the next URL that is found during the crawl will get a doc id of 8. Also you need to ensure to add seeds in increasing order of document ids. Specifying doc ids is mainly useful when you have had a previous crawl and have stored the results and want to start a new crawl with seeds which get the same document ids as the previous crawl.

Parameters:

pageUrl - the URL of the seed

docId - the document id that you want to be assigned to this seed URL.

addSeenUrl

public void addSeenUrl(String url,
                       int docId)

This function can called to assign a specific document id to a url. This feature is useful when you have had a previous crawl and have stored the Urls and their associated document ids and want to have a new crawl which is aware of the previously seen Urls and won't re-crawl them. Note that if you add three seen Urls with document ids 1,2, and 7. Then the next URL that is found during the crawl will get a doc id of 8. Also you need to ensure to add seen Urls in increasing order of document ids.

Parameters:

url - the URL of the page

docId - the document id that you want to be assigned to this URL.

getPageFetcher

public PageFetcher getPageFetcher()

setPageFetcher

public void setPageFetcher(PageFetcher pageFetcher)

getRobotstxtServer

public RobotstxtServer getRobotstxtServer()

setRobotstxtServer

public void setRobotstxtServer(RobotstxtServer robotstxtServer)

getFrontier

public Frontier getFrontier()

setFrontier

public void setFrontier(Frontier frontier)

getDocIdServer

public DocIDServer getDocIdServer()

setDocIdServer

public void setDocIdServer(DocIDServer docIdServer)

getCustomData

public Object getCustomData()

setCustomData

public void setCustomData(Object customData)

isFinished

public boolean isFinished()

isShuttingDown

public boolean isShuttingDown()

shutdown

public void shutdown()

Set the current crawling session set to 'shutdown'. Crawler threads monitor the shutdown flag and when it is set to true, they will no longer process new pages.