edu.uci.ics.crawler4j.robotstxt

Class PathRule

java.lang.Object

edu.uci.ics.crawler4j.robotstxt.PathRule

public class PathRule
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	logger
Pattern	pattern
int	type

Constructor Summary

Constructors

Constructor and Description
PathRule(int type, String pattern) Create a new path rule, based on the specified pattern

Method Summary

Modifier and Type	Method and Description
boolean	matches(String path) Check if the specified path matches this rule
static boolean	matchesRobotsPattern(String pattern, String path) Check if the specified path matches a robots.txt pattern
static Pattern	robotsPatternToRegexp(String pattern) Match a pattern defined in a robots.txt file to a path Following the pattern definition as stated on: https://support.google.com/webmasters/answer/6062596?hl=en&ref_topic=6061961 This page defines the following items: * matches any sequence of characters, including / $ matches the end of the line

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

type

public int type

pattern

public Pattern pattern

Constructor Detail

PathRule

public PathRule(int type,
                String pattern)

Create a new path rule, based on the specified pattern

Parameters:

type - Either HostDirectives.ALLOWS or HostDirectives.DISALLOWS

pattern - The pattern for this rule

Method Detail

robotsPatternToRegexp

public static Pattern robotsPatternToRegexp(String pattern)

Match a pattern defined in a robots.txt file to a path Following the pattern definition as stated on: https://support.google.com/webmasters/answer/6062596?hl=en&ref_topic=6061961 This page defines the following items: * matches any sequence of characters, including / $ matches the end of the line

Parameters:

pattern - The pattern to convert

Returns:

The compiled regexp pattern created from the robots.txt pattern

matchesRobotsPattern

public static boolean matchesRobotsPattern(String pattern,
                                           String path)

Check if the specified path matches a robots.txt pattern

Parameters:

pattern - The pattern to match

path - The path to match with the pattern

Returns:

True when the pattern matches, false if it does not

matches

public boolean matches(String path)

Check if the specified path matches this rule

Parameters:

path - The path to match with this pattern

Returns:

True when the path matches, false when it does not