edu.uci.ics.crawler4j.robotstxt

Class UserAgentDirectives

java.lang.Object

edu.uci.ics.crawler4j.robotstxt.UserAgentDirectives

public class UserAgentDirectives
extends Object

The UserAgentDirectives class stores the configuration for a single user agent as defined in the robots.txt. The user agent string used depends on the most recent User-agent: definition in the robots.txt file.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	UserAgentDirectives.UserAgentComparator

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	logger
Set<String>	userAgents

Constructor Summary

Constructors

Constructor and Description
UserAgentDirectives(Set<String> userAgents) Create a UserAgentDirectives clause

Method Summary

Modifier and Type	Method and Description
void	add(String rule, String value) Add a rule to the list of rules for this user agent.
int	checkAccess(String path, String userAgent)
Double	getCrawlDelay() Return the configured crawl delay in seconds
String	getPreferredHost() Return the specified preferred host name in robots.txt.
List<String>	getSitemap() Return the listed sitemaps, or null if none was specified
boolean	isEmpty()
boolean	isWildcard()
int	match(String userAgent) Match the current user agent directive set with the given user agent.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

public static final org.slf4j.Logger logger

userAgents

public Set<String> userAgents

Constructor Detail

UserAgentDirectives

public UserAgentDirectives(Set<String> userAgents)

Create a UserAgentDirectives clause

Parameters:

userAgents - The list user agents for this rule

Method Detail

match

public int match(String userAgent)

Match the current user agent directive set with the given user agent. The returned value will be the maximum match length of any user agent.

Parameters:

userAgent - The user agent used by the crawler

Returns:

The maximum length of a matching user agent in this set of directives

isWildcard

public boolean isWildcard()

isEmpty

public boolean isEmpty()

checkAccess

public int checkAccess(String path,
                       String userAgent)

add

public void add(String rule,
                String value)

Add a rule to the list of rules for this user agent. Valid rules are: sitemap, crawl-delay, host, allow and disallow. These are based on the wikipedia article at: https://en.wikipedia.org/wiki/Robots_exclusion_standard and the Google documentation at: https://support.google.com/webmasters/answer/6062596

Parameters:

rule - The name of the rule

value - The value of the rule

getCrawlDelay

public Double getCrawlDelay()

Return the configured crawl delay in seconds

Returns:

The configured crawl delay, or null if none was specified

getPreferredHost

public String getPreferredHost()

Return the specified preferred host name in robots.txt.

Returns:

The specified hostname, or null if it was not specified

getSitemap

public List<String> getSitemap()

Return the listed sitemaps, or null if none was specified

Returns:

The list of sitemap-links specified in robots.txt