edu.uci.ics.crawler4j.crawler

Class Page

java.lang.Object

edu.uci.ics.crawler4j.crawler.Page

public class Page
extends Object

This class contains the data for a fetched and parsed page.

Author:

Yasser Ganjisaffar

Field Summary

Fields

Modifier and Type	Field and Description
protected String	contentCharset The charset of the content.
protected byte[]	contentData The content of this page in binary format.
protected String	contentEncoding The encoding of the content.
protected String	contentType The ContentType of this page.
protected org.apache.http.Header[]	fetchResponseHeaders Headers which were present in the response of the fetch request
protected ParseData	parseData The parsed data populated by parsers
protected boolean	redirect Redirection flag
protected String	redirectedToUrl The URL to which this page will be redirected to
protected int	statusCode Status of the page
protected boolean	truncated Whether the content was truncated because the received data exceeded the imposed maximum
protected WebURL	url The URL of this page.

Constructor Summary

Constructors

Constructor and Description
Page(WebURL url)

Method Summary

Modifier and Type	Method and Description
String	getContentCharset()
byte[]	getContentData()
String	getContentEncoding()
String	getContentType()
org.apache.http.Header[]	getFetchResponseHeaders() Returns headers which were present in the response of the fetch request
String	getLanguage()
ParseData	getParseData()
String	getRedirectedToUrl()
int	getStatusCode()
WebURL	getWebURL()
boolean	isRedirect()
boolean	isTruncated()
void	load(org.apache.http.HttpEntity entity, int maxBytes) Loads the content of this page from a fetched HttpEntity.
void	setContentCharset(String contentCharset)
void	setContentData(byte[] contentData)
void	setContentEncoding(String contentEncoding)
void	setContentType(String contentType)
void	setFetchResponseHeaders(org.apache.http.Header[] headers)
void	setLanguage(String language)
void	setParseData(ParseData parseData)
void	setRedirect(boolean redirect)
void	setRedirectedToUrl(String redirectedToUrl)
void	setStatusCode(int statusCode)
void	setWebURL(WebURL url)
protected byte[]	toByteArray(org.apache.http.HttpEntity entity, int maxBytes) Read contents from an entity, with a specified maximum.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

url

protected WebURL url

The URL of this page.

redirect

protected boolean redirect

Redirection flag

redirectedToUrl

protected String redirectedToUrl

The URL to which this page will be redirected to

statusCode

protected int statusCode

Status of the page

contentData

protected byte[] contentData

The content of this page in binary format.

contentType

protected String contentType

The ContentType of this page. For example: "text/html; charset=UTF-8"

contentEncoding

protected String contentEncoding

The encoding of the content. For example: "gzip"

contentCharset

protected String contentCharset

The charset of the content. For example: "UTF-8"

fetchResponseHeaders

protected org.apache.http.Header[] fetchResponseHeaders

Headers which were present in the response of the fetch request

parseData

protected ParseData parseData

The parsed data populated by parsers

truncated

protected boolean truncated

Whether the content was truncated because the received data exceeded the imposed maximum

Constructor Detail

Page

public Page(WebURL url)

Method Detail

load

public void load(org.apache.http.HttpEntity entity,
                 int maxBytes)
          throws Exception

Loads the content of this page from a fetched HttpEntity.

Parameters:

entity - HttpEntity

maxBytes - The maximum number of bytes to read

Throws:

Exception - when load fails

toByteArray

protected byte[] toByteArray(org.apache.http.HttpEntity entity,
                             int maxBytes)
                      throws IOException

Read contents from an entity, with a specified maximum. This is a replacement of EntityUtils.toByteArray because that function does not impose a maximum size.

Parameters:

entity - The entity from which to read

maxBytes - The maximum number of bytes to read

Returns:

A byte array containing maxBytes or fewer bytes read from the entity

Throws:

IOException - Thrown when reading fails for any reason

getWebURL

public WebURL getWebURL()

setWebURL

public void setWebURL(WebURL url)

isRedirect

public boolean isRedirect()

setRedirect

public void setRedirect(boolean redirect)

getRedirectedToUrl

public String getRedirectedToUrl()

setRedirectedToUrl

public void setRedirectedToUrl(String redirectedToUrl)

getStatusCode

public int getStatusCode()

setStatusCode

public void setStatusCode(int statusCode)

getFetchResponseHeaders

public org.apache.http.Header[] getFetchResponseHeaders()

Returns headers which were present in the response of the fetch request

Returns:

Header Array, the response headers

setFetchResponseHeaders

public void setFetchResponseHeaders(org.apache.http.Header[] headers)

getParseData

public ParseData getParseData()

Returns:

parsed data generated for this page by parsers

setParseData

public void setParseData(ParseData parseData)

getContentData

public byte[] getContentData()

Returns:

content of this page in binary format.

setContentData

public void setContentData(byte[] contentData)

getContentType

public String getContentType()

Returns:

ContentType of this page. For example: "text/html; charset=UTF-8"

setContentType

public void setContentType(String contentType)

getContentEncoding

public String getContentEncoding()

Returns:

encoding of the content. For example: "gzip"

setContentEncoding

public void setContentEncoding(String contentEncoding)

getContentCharset

public String getContentCharset()

Returns:

charset of the content. For example: "UTF-8"

setContentCharset

public void setContentCharset(String contentCharset)

getLanguage

public String getLanguage()

Returns:

Language

setLanguage

public void setLanguage(String language)

isTruncated

public boolean isTruncated()