edu.uci.ics.crawler4j.robotstxt

Class RobotstxtServer

java.lang.Object

edu.uci.ics.crawler4j.robotstxt.RobotstxtServer

public class RobotstxtServer
extends Object

Author:

Yasser Ganjisaffar

Field Summary

Fields

Modifier and Type	Field and Description
protected RobotstxtConfig	config
protected Map<String,HostDirectives>	host2directivesCache
protected PageFetcher	pageFetcher

Constructor Summary

Constructors

Constructor and Description
RobotstxtServer(RobotstxtConfig config, PageFetcher pageFetcher)
RobotstxtServer(RobotstxtConfig config, PageFetcher pageFetcher, int maxBytes)

Method Summary

Modifier and Type	Method and Description
boolean	allows(WebURL webURL) Please note that in the case of a bad URL, TRUE will be returned

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected RobotstxtConfig config

host2directivesCache

protected final Map<String,HostDirectives> host2directivesCache

pageFetcher

protected PageFetcher pageFetcher

Constructor Detail

RobotstxtServer

public RobotstxtServer(RobotstxtConfig config,
                       PageFetcher pageFetcher)

RobotstxtServer

public RobotstxtServer(RobotstxtConfig config,
                       PageFetcher pageFetcher,
                       int maxBytes)

Method Detail

allows

public boolean allows(WebURL webURL)

Please note that in the case of a bad URL, TRUE will be returned