edu.uci.ics.crawler4j.robotstxt

Class HostDirectives

java.lang.Object

edu.uci.ics.crawler4j.robotstxt.HostDirectives

public class HostDirectives
extends Object

Author:

Yasser Ganjisaffar

Field Summary

Fields

Modifier and Type	Field and Description
static int	ALLOWED
static int	DISALLOWED
static int	UNDEFINED

Constructor Summary

Constructors

Constructor and Description
HostDirectives(RobotstxtConfig configuration)

Method Summary

Modifier and Type	Method and Description
void	addDirectives(UserAgentDirectives directives) Store set of directives
boolean	allows(String path) Check if the host directives allows visiting path.
int	checkAccess(String path) Check if any of the rules say anything about the specified path
boolean	disallows(String path) Check if the host directives explicitly disallow visiting path.
long	getLastAccessTime()
boolean	needsRefetch()
void	setUserAgent(String userAgent) Change the user agent string used to crawl after initialization.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ALLOWED

public static final int ALLOWED

See Also:

Constant Field Values

DISALLOWED

public static final int DISALLOWED

See Also:

Constant Field Values

UNDEFINED

public static final int UNDEFINED

See Also:

Constant Field Values

Constructor Detail

HostDirectives

public HostDirectives(RobotstxtConfig configuration)

Method Detail

needsRefetch

public boolean needsRefetch()

allows

public boolean allows(String path)

Check if the host directives allows visiting path.

Parameters:

path - The path to check

Returns:

True if the path is not disallowed, false if it is

setUserAgent

public void setUserAgent(String userAgent)

Change the user agent string used to crawl after initialization. This will reorder (recreate) the list of user agent directives for this host.

Parameters:

userAgent - The new user agent to use.

disallows

public boolean disallows(String path)

Check if the host directives explicitly disallow visiting path.

Parameters:

path - The path to check

Returns:

True if the path is explicity disallowed, false otherwise

checkAccess

public int checkAccess(String path)

Check if any of the rules say anything about the specified path

Parameters:

path - The path to check

Returns:

One of ALLOWED, DISALLOWED or UNDEFINED

addDirectives

public void addDirectives(UserAgentDirectives directives)

Store set of directives

Parameters:

directives - The set of directives to add to this host

getLastAccessTime

public long getLastAccessTime()