Package com.apple.foundationdb.record.lucene.directory

Class FDBDirectory

java.lang.Object
org.apache.lucene.store.Directory
com.apple.foundationdb.record.lucene.directory.FDBDirectory
All Implemented Interfaces:
Closeable, AutoCloseable
@API(EXPERIMENTAL) @NotThreadSafe public class FDBDirectory extends org.apache.lucene.store.Directory
Directory implementation backed by FDB which attempts to model a file system on top of FoundationDB. A few interesting details:
  • Each segment written in Lucene once flushed is immutable.
  • As more data is written into Lucene, files are continually merged via a MergePolicy
See Also:

  • Field Details

    • DEFAULT_BLOCK_SIZE

      public static final int DEFAULT_BLOCK_SIZE
      See Also:

    • DEFAULT_BLOCK_CACHE_MAXIMUM_SIZE

      public static final int DEFAULT_BLOCK_CACHE_MAXIMUM_SIZE
      See Also:

    • DEFAULT_CONCURRENCY_LEVEL

      public static final int DEFAULT_CONCURRENCY_LEVEL
      See Also:

    • DEFAULT_INITIAL_CAPACITY

      public static final int DEFAULT_INITIAL_CAPACITY
      See Also:

    • DATA_SUBSPACE

      public static final int DATA_SUBSPACE
      See Also:

    • FILE_LOCK_SUBSPACE

      public static final int FILE_LOCK_SUBSPACE
      See Also:

    • storedFieldsSubspace

      protected final Subspace storedFieldsSubspace

  • Constructor Details

  • Method Details

    • getIncrement

      public long getIncrement() throws IOException
      Sets increment if not set yet and waits till completed to return Returns and increments the increment if its already set.
      Returns:
      current increment value
      Throws:
      IOException

    • getFDBLuceneFileReferenceAsync

      @API(INTERNAL) @Nonnull public CompletableFuture<FDBLuceneFileReference> getFDBLuceneFileReferenceAsync(@Nonnull String name)
      Checks the cache for the file reference. If the file is in the cache it returns the cached value. If not there then it checks the subspace for it. If its there the file is added to the cache and returned to caller If the file doesn't exist in the subspace it returns null.
      Parameters:
      name - name for the file reference
      Returns:
      FDBLuceneFileReference

    • getFDBLuceneFileReference

      @API(INTERNAL) @Nullable public FDBLuceneFileReference getFDBLuceneFileReference(@Nonnull String name)

    • getFieldInfosStorage

      public FieldInfosStorage getFieldInfosStorage()

    • setFieldInfoId

      public void setFieldInfoId(String filename, long id, ByteString bitSet)

    • getFieldInfosCount

      public CompletableFuture<Integer> getFieldInfosCount()

    • isSegmentInfo

      public static boolean isSegmentInfo(String name)

    • isCompoundFile

      public static boolean isCompoundFile(String name)

    • isEntriesFile

      public static boolean isEntriesFile(String name)

    • isFieldInfoFile

      public static boolean isFieldInfoFile(String name)

    • isStoredFieldsFile

      public static boolean isStoredFieldsFile(String name)

    • writeFDBLuceneFileReference

      public void writeFDBLuceneFileReference(@Nonnull String name, @Nonnull FDBLuceneFileReference reference)
      Puts a file reference in the meta subspace and in the cache under the given name.
      Parameters:
      name - name for the file reference
      reference - the file reference being inserted

    • writeData

      public int writeData(long id, int block, @Nonnull byte[] value)
      Writes data to the given block under the given id.
      Parameters:
      id - id for the data
      block - block for the data to be stored in
      value - the data to be stored
      Returns:
      the actual data size written to database with potential compression and encryption applied

    • writeStoredFields

      public void writeStoredFields(@Nonnull String segmentName, int docID, @Nonnull byte[] value)
      Write stored fields document to the DB.
      Parameters:
      segmentName - the segment name writing to
      docID - the document ID to write
      value - the bytes value of the stored fields

    • deleteStoredFields

      public void deleteStoredFields(@Nonnull String segmentName) throws IOException
      Delete stored fields data from the DB.
      Parameters:
      segmentName - the segment name to delete the fields from (all docs in the segment will be deleted)
      Throws:
      IOException - if there is an issue reading metadata to do the delete

    • readBlock

      @API(INTERNAL) @Nonnull public CompletableFuture<byte[]> readBlock(@Nonnull org.apache.lucene.store.IndexInput requestingInput, @Nonnull String fileName, @Nonnull CompletableFuture<FDBLuceneFileReference> referenceFuture, int block)
      Reads known data from the directory.
      Parameters:
      requestingInput - the FDBIndexInput requesting the block; used for logging
      fileName - Description should be non-null, opaque string describing this resource; used for logging
      referenceFuture - the reference where the data supposedly lives
      block - the block where the data is stored
      Returns:
      Completable future of the data returned
      Throws:
      RecordCoreException - if blockCache fails to get the data from the block
      RecordCoreArgumentException - if a reference with that id hasn't been written yet.

    • readStoredFields

      @Nonnull public byte[] readStoredFields(String segmentName, int docId) throws IOException
      Throws:
      IOException

    • readAllStoredFields

      @Nonnull public List<KeyValue> readAllStoredFields(String segmentName)

    • listAll

      @Nonnull public String[] listAll() throws IOException
      Lists all file names in the subspace. Puts all references in the cache. Logs the count of references, and the total size of the data. All references are put into the cache because each composite file will be required to serve a query and the file references are small.
      Specified by:
      listAll in class org.apache.lucene.store.Directory
      Returns:
      String list of names of lucene file references
      Throws:
      IOException

    • listAllAsync

      public CompletableFuture<Collection<String>> listAllAsync()

    • getAllAsync

      public CompletableFuture<Map<String,FDBLuceneFileReference>> getAllAsync()
      Return a CompletableFuture to a map of file names to file references. This is a copy of the file reference map that exists in the cache (and the subspace).
      Returns:
      a future to a map of file names to LuceneFileSystemProto.LuceneFileReferences

    • scanStoredFields

      public CompletableFuture<List<KeyValue>> scanStoredFields(String segmentName)

    • getFileReferenceCacheAsync

      @Nonnull public CompletableFuture<Map<String,FDBLuceneFileReference>> getFileReferenceCacheAsync()

    • deleteFile

      public void deleteFile(@Nonnull String name) throws IOException
      deletes the file reference under the provided name.
      Specified by:
      deleteFile in class org.apache.lucene.store.Directory
      Parameters:
      name - the name for the file reference
      Throws:
      IOException

    • deleteFileInternal

      protected boolean deleteFileInternal(@Nonnull Map<String,FDBLuceneFileReference> cache, @Nonnull String name) throws IOException
      Throws:
      IOException

    • usesOptimizedStoredFields

      public boolean usesOptimizedStoredFields()

    • fileLength

      public long fileLength(@Nonnull String name) throws IOException
      Returns the size of the given file under the given name.
      Specified by:
      fileLength in class org.apache.lucene.store.Directory
      Parameters:
      name - the name of the file reference
      Returns:
      long value of the size of the file
      Throws:
      NoSuchFileException - if the file reference doesn't exist.
      IOException

    • createOutput

      @Nonnull public org.apache.lucene.store.IndexOutput createOutput(@Nonnull String name, @Nullable org.apache.lucene.store.IOContext ioContext) throws IOException
      Create new output for a file.
      Specified by:
      createOutput in class org.apache.lucene.store.Directory
      Parameters:
      name - the filename to create
      ioContext - the IOContext from Lucene
      Returns:
      IndexOutput FDB Backed Index Output FDBIndexOutput
      Throws:
      IOException

    • createTempOutput

      @Nonnull public org.apache.lucene.store.IndexOutput createTempOutput(@Nonnull String prefix, @Nonnull String suffix, @Nonnull org.apache.lucene.store.IOContext ioContext) throws IOException
      Lucene uses temporary files internally that are actually serialized onto disk. Once on disk, the files can be renamed.
      Specified by:
      createTempOutput in class org.apache.lucene.store.Directory
      Parameters:
      prefix - prefix
      suffix - suffix
      ioContext - ioContext
      Returns:
      IndexOutput
      Throws:
      IOException

    • getTempFileName

      @Nonnull protected static String getTempFileName(@Nonnull String prefix, @Nonnull String suffix, long counter)

    • sync

      public void sync(@Nonnull Collection<String> collection)
      Specified by:
      sync in class org.apache.lucene.store.Directory

    • syncMetaData

      public void syncMetaData() throws IOException
      Specified by:
      syncMetaData in class org.apache.lucene.store.Directory
      Throws:
      IOException

    • rename

      public void rename(@Nonnull String source, @Nonnull String dest) throws IOException
      It is the caller's responsibility to make the dest (destination) does not exist.
      Specified by:
      rename in class org.apache.lucene.store.Directory
      Parameters:
      source - source
      dest - desc
      Throws:
      IOException

    • openInput

      @Nonnull public org.apache.lucene.store.IndexInput openInput(@Nonnull String name, @Nonnull org.apache.lucene.store.IOContext ioContext) throws IOException
      Specified by:
      openInput in class org.apache.lucene.store.Directory
      Throws:
      IOException

    • obtainLock

      @Nonnull public org.apache.lucene.store.Lock obtainLock(@Nonnull String lockName) throws IOException
      Specified by:
      obtainLock in class org.apache.lucene.store.Directory
      Throws:
      IOException

    • close

      public void close() throws IOException
      No Op that reports in debug mode the block and file reference cache stats.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in class org.apache.lucene.store.Directory
      Throws:
      IOException

    • getPendingDeletions

      @Nonnull public Set<String> getPendingDeletions()
      We delete inline vs. batching them together.
      Specified by:
      getPendingDeletions in class org.apache.lucene.store.Directory
      Returns:
      Emtpy set of strings

    • getBlockSize

      public int getBlockSize()

    • getAgilityContext

      public AgilityContext getAgilityContext()

    • asyncToSync

      @Nullable public <T> T asyncToSync(@Nonnull StoreTimer.Wait event, @Nonnull CompletableFuture<T> async)

    • getSerializer

      @Nullable public LuceneSerializer getSerializer()

    • getSubspace

      public Subspace getSubspace()

    • openChecksumInput

      public org.apache.lucene.store.ChecksumIndexInput openChecksumInput(String name, org.apache.lucene.store.IOContext context) throws IOException
      Places a prefetchable buffer over the checksum input in an attempt to pipeline reads when an FDBIndexOutput performs a copyBytes operation.
      Overrides:
      openChecksumInput in class org.apache.lucene.store.Directory
      Parameters:
      name - file name
      context - io context
      Returns:
      ChecksumIndexInput
      Throws:
      IOException - ioexception

    • getPrimaryKeySegmentIndex

      @Nullable public LucenePrimaryKeySegmentIndex getPrimaryKeySegmentIndex()
      Get a primary key segment index if enabled.
      Returns:
      index or null if not enabled

    • primaryKeySegmentId

      public long primaryKeySegmentId(@Nonnull String segmentName, boolean create) throws IOException
      Throws:
      IOException

    • primaryKeySegmentName

      @Nullable public String primaryKeySegmentName(long segmentId)

    • getBooleanIndexOption

      public boolean getBooleanIndexOption(@Nonnull String key, boolean defaultValue)
      Convenience methods to get index options from the directory's index.
      Parameters:
      key - the option key
      defaultValue - the value to use when the option is not set
      Returns:
      the index option value, or the default value if not found

    • getIndexOption

      @Nullable public String getIndexOption(@Nonnull String key)
      Convenience methods to get index options from the directory's index.
      Parameters:
      key - the option key
      Returns:
      the index option value, null if not found