net.fortytwo.linkeddata.util

Class HTTPUtils

java.lang.Object

net.fortytwo.linkeddata.util.HTTPUtils

public class HTTPUtils
extends java.lang.Object

Author:

Joshua Shinavier (http://fortytwo.net)

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ACCEPT
static java.lang.String	BODY
static java.lang.String	CONTENT_TYPE
static java.lang.String	SPARQL_QUERY
static java.lang.String	USER_AGENT

Constructor Summary

Constructors

Constructor and Description
HTTPUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.http.client.HttpClient	createClient(boolean autoRedirect)
static org.apache.http.client.methods.HttpGet	createGetMethod(java.lang.String url)
static void	setAcceptHeader(org.apache.http.HttpRequest method, java.lang.String value)
static long	throttleHttpRequest(org.apache.http.HttpRequest method) Enforces crawler etiquette with respect to timing of HTTP requests.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ACCEPT

public static final java.lang.String ACCEPT

See Also:

Constant Field Values

BODY

public static final java.lang.String BODY

See Also:

Constant Field Values

CONTENT_TYPE

public static final java.lang.String CONTENT_TYPE

See Also:

Constant Field Values

SPARQL_QUERY

public static final java.lang.String SPARQL_QUERY

See Also:

Constant Field Values

USER_AGENT

public static final java.lang.String USER_AGENT

See Also:

Constant Field Values

Constructor Detail

HTTPUtils

public HTTPUtils()

Method Detail

createClient

public static org.apache.http.client.HttpClient createClient(boolean autoRedirect)

createGetMethod

public static org.apache.http.client.methods.HttpGet createGetMethod(java.lang.String url)

setAcceptHeader

public static void setAcceptHeader(org.apache.http.HttpRequest method,
                                   java.lang.String value)

throttleHttpRequest

public static long throttleHttpRequest(org.apache.http.HttpRequest method)

Enforces crawler etiquette with respect to timing of HTTP requests. That is, it avoids the Ripple client making a nuisance of itself by making too many requests, too quickly, of the same host. TODO: request and respect robots.txt, if present.

Returns:

the amount of time, in milliseconds, that is spent idling for the sake of crawler etiquette