Package io.openlineage.spark.shade.extension.v1

Interface LineageRelationProvider

public interface LineageRelationProvider
Interface for classes implementing `org.apache.spark.sql.sources.RelationProvider`. `RelationProvider` implements `createRelation` with `SQLContext` and `parameters` as arguments. We want this package to not depend on Spark's code which may be different across Spark versions.

We're aiming to have arguments of `getLineageDataset` the same as arguments of `createRelation` within `RelationProvider`. When implementing this method, one can provide two implementations: one with arguments exactly the same as with `RelationProvider`, and other throwing an exception which should never be called.

  • Method Summary

    Modifier and Type
    Method
    Description
    io.openlineage.client.utils.DatasetIdentifier
    getLineageDatasetIdentifier(String sparkListenerEventName, io.openlineage.client.OpenLineage openLineage, Object sqlContext, Object parameters)
     

  • Method Details

    • getLineageDatasetIdentifier

      io.openlineage.client.utils.DatasetIdentifier getLineageDatasetIdentifier(String sparkListenerEventName, io.openlineage.client.OpenLineage openLineage, Object sqlContext, Object parameters)