io.ray.streaming.python.stream

Class PythonDataStream

java.lang.Object

io.ray.streaming.api.stream.Stream<PythonDataStream,Object>

io.ray.streaming.python.stream.PythonDataStream

All Implemented Interfaces:

PythonStream, Serializable

Direct Known Subclasses:

PythonKeyDataStream, PythonStreamSource, PythonUnionStream

public class PythonDataStream
extends Stream<PythonDataStream,Object>
implements PythonStream

Represents a stream of data whose transformations will be executed in python.

See Also:

Serialized Form

Constructor Summary

Constructors

Modifier	Constructor and Description
	PythonDataStream(DataStream referencedStream) Create a python stream that reference passed java stream.
	PythonDataStream(PythonDataStream input, PythonOperator pythonOperator)
	PythonDataStream(PythonDataStream input, PythonOperator pythonOperator, Partition<Object> partition)
protected	PythonDataStream(StreamingContext streamingContext, PythonOperator pythonOperator)
protected	PythonDataStream(StreamingContext streamingContext, PythonOperator pythonOperator, Partition<Object> partition)

Method Summary

Modifier and Type	Method and Description
DataStream<Object>	asJavaStream() Convert this stream as a java stream.
PythonDataStream	broadcast() Apply broadcast to this stream.
PythonDataStream	filter(PythonFunction func) Apply a filter function to this stream.
PythonDataStream	filter(String moduleName, String funcName)
PythonDataStream	flatMap(PythonFunction func) Apply a flat-map function to this stream.
PythonDataStream	flatMap(String moduleName, String funcName)
Language	getLanguage()
PythonKeyDataStream	keyBy(PythonFunction func) Apply a key-by function to this stream.
PythonKeyDataStream	keyBy(String moduleName, String funcName)
PythonDataStream	map(PythonFunction func) Apply a map function to this stream.
PythonDataStream	map(String moduleName, String funcName)
PythonDataStream	partitionBy(PythonPartition partition) Apply a partition to this stream.
PythonStreamSink	sink(PythonFunction func) Apply a sink function and get a StreamSink.
PythonStreamSink	sink(String moduleName, String funcName)
PythonDataStream	union(List<PythonDataStream> streams) Apply union transformations to this stream by merging PythonDataStream outputs of the same type with each other.
PythonDataStream	union(PythonDataStream stream, PythonDataStream... others) Apply union transformations to this stream by merging PythonDataStream outputs of the same type with each other.

Methods inherited from class io.ray.streaming.api.stream.Stream

disableChain, forward, getConfig, getId, getInputStream, getOperator, getOriginalStream, getParallelism, getPartition, getStreamingContext, isProxyStream, setParallelism, setPartition, withChainStrategy, withConfig, withConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PythonDataStream

protected PythonDataStream(StreamingContext streamingContext,
                           PythonOperator pythonOperator)

PythonDataStream

protected PythonDataStream(StreamingContext streamingContext,
                           PythonOperator pythonOperator,
                           Partition<Object> partition)

PythonDataStream

public PythonDataStream(PythonDataStream input,
                        PythonOperator pythonOperator)

PythonDataStream

public PythonDataStream(PythonDataStream input,
                        PythonOperator pythonOperator,
                        Partition<Object> partition)

PythonDataStream

public PythonDataStream(DataStream referencedStream)

Create a python stream that reference passed java stream. Changes in new stream will be reflected in referenced stream and vice versa

Method Detail

map

public PythonDataStream map(String moduleName,
                            String funcName)

map

public PythonDataStream map(PythonFunction func)

Apply a map function to this stream.

Parameters:

func - The python MapFunction.

Returns:

A new PythonDataStream.

flatMap

public PythonDataStream flatMap(String moduleName,
                                String funcName)

flatMap

public PythonDataStream flatMap(PythonFunction func)

Apply a flat-map function to this stream.

Parameters:

func - The python FlapMapFunction.

Returns:

A new PythonDataStream

filter

public PythonDataStream filter(String moduleName,
                               String funcName)

filter

public PythonDataStream filter(PythonFunction func)

Apply a filter function to this stream.

Parameters:

func - The python FilterFunction.

Returns:

A new PythonDataStream that contains only the elements satisfying the given filter predicate.

union

public final PythonDataStream union(PythonDataStream stream,
                                    PythonDataStream... others)

Apply union transformations to this stream by merging PythonDataStream outputs of the same type with each other.

Parameters:

stream - The DataStream to union output with.

others - The other DataStreams to union output with.

Returns:

A new UnionStream.

union

public final PythonDataStream union(List<PythonDataStream> streams)

Apply union transformations to this stream by merging PythonDataStream outputs of the same type with each other.

Parameters:

streams - The DataStreams to union output with.

Returns:

A new UnionStream.

sink

public PythonStreamSink sink(String moduleName,
                             String funcName)

sink

public PythonStreamSink sink(PythonFunction func)

Apply a sink function and get a StreamSink.

Parameters:

func - The python SinkFunction.

Returns:

A new StreamSink.

keyBy

public PythonKeyDataStream keyBy(String moduleName,
                                 String funcName)

keyBy

public PythonKeyDataStream keyBy(PythonFunction func)

Apply a key-by function to this stream.

Parameters:

func - the python keyFunction.

Returns:

A new KeyDataStream.

broadcast

public PythonDataStream broadcast()

Apply broadcast to this stream.

Returns:

This stream.

partitionBy

public PythonDataStream partitionBy(PythonPartition partition)

Apply a partition to this stream.

Parameters:

partition - The partitioning strategy.

Returns:

This stream.

asJavaStream

public DataStream<Object> asJavaStream()

Convert this stream as a java stream. The converted stream and this stream are the same logical stream, which has same stream id. Changes in converted stream will be reflected in this stream and vice versa.

getLanguage

public Language getLanguage()

Specified by:

getLanguage in class Stream<PythonDataStream,Object>