io.earcam.utilitarian.site.search.offline

Class Crawler

java.lang.Object

io.earcam.utilitarian.site.search.offline.Crawler

public class Crawler
extends Object

Pipeline needs to be built according to definition, with names driven via SPI, e.g. configuration is just a Map passed to each

 
    
         default-regex
         
          regex
          regex
       
    
    
       default-html
    
    
       default-pdf
    
    
    
       default-tokenizer
    
 
 

Therefore Filter and Processor both need to extend 'Component' Component{ String id; void configure(Map) } Filter imps Predicate Processor{ process(Document); } HtmlContentParser{ }

Constructor Summary

Constructors

Constructor and Description
Crawler()

Method Summary

Modifier and Type	Method and Description
static Crawler	crawler(Map<Path,URI> directories)
Stream<Document>	documents()
Crawler	filter(Predicate<Document> filter)
Crawler	processor(Processor processor)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Crawler

public Crawler()

Method Detail

crawler

public static Crawler crawler(Map<Path,URI> directories)

filter

public Crawler filter(Predicate<Document> filter)

processor

public Crawler processor(Processor processor)

documents

public Stream<Document> documents()