io.ray.streaming.api.stream

Class DataStream<T>

java.lang.Object

io.ray.streaming.api.stream.Stream<DataStream<T>,T>

io.ray.streaming.api.stream.DataStream<T>

Type Parameters:

T - Type of data in the stream.

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

DataStreamSource, JoinStream, KeyDataStream, UnionStream

public class DataStream<T>
extends Stream<DataStream<T>,T>

Represents a stream of data.

This class defines all the streaming operations.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DataStream(DataStream<R> input, StreamOperator streamOperator)
DataStream(DataStream<R> input, StreamOperator streamOperator, Partition<T> partition)
DataStream(PythonDataStream referencedStream) Create a java stream that reference passed python stream.
DataStream(StreamingContext streamingContext, StreamOperator streamOperator)
DataStream(StreamingContext streamingContext, StreamOperator streamOperator, Partition<T> partition)

Method Summary

Modifier and Type	Method and Description
PythonDataStream	asPythonStream() Convert this stream as a python stream.
DataStream<T>	broadcast() Apply broadcast to this stream.
DataStream<T>	filter(FilterFunction<T> filterFunction)
<R> DataStream<R>	flatMap(FlatMapFunction<T,R> flatMapFunction) Apply a flat-map function to this stream.
Language	getLanguage()
<O,R> JoinStream<T,O,R>	join(DataStream<O> other) Apply a join transformation to this stream, with another stream.
<K> KeyDataStream<K,T>	keyBy(KeyFunction<T,K> keyFunction) Apply a key-by function to this stream.
<R> DataStream<R>	map(MapFunction<T,R> mapFunction) Apply a map function to this stream.
DataStream<T>	partitionBy(Partition<T> partition) Apply a partition to this stream.
<R> DataStream<R>	process()
DataStreamSink<T>	sink(SinkFunction<T> sinkFunction) Apply a sink function and get a StreamSink.
DataStream<T>	union(DataStream<T> stream, DataStream<T>... others) Apply union transformations to this stream by merging DataStream outputs of the same type with each other.
DataStream<T>	union(List<DataStream<T>> streams) Apply union transformations to this stream by merging DataStream outputs of the same type with each other.

Methods inherited from class io.ray.streaming.api.stream.Stream

disableChain, forward, getConfig, getId, getInputStream, getOperator, getOriginalStream, getParallelism, getPartition, getStreamingContext, isProxyStream, setParallelism, setPartition, withChainStrategy, withConfig, withConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataStream

public DataStream(StreamingContext streamingContext,
                  StreamOperator streamOperator)

DataStream

public DataStream(StreamingContext streamingContext,
                  StreamOperator streamOperator,
                  Partition<T> partition)

DataStream

public DataStream(DataStream<R> input,
                  StreamOperator streamOperator)

DataStream

public DataStream(DataStream<R> input,
                  StreamOperator streamOperator,
                  Partition<T> partition)

DataStream

public DataStream(PythonDataStream referencedStream)

Create a java stream that reference passed python stream. Changes in new stream will be reflected in referenced stream and vice versa

Method Detail

map

public <R> DataStream<R> map(MapFunction<T,R> mapFunction)

Apply a map function to this stream.

Type Parameters:

R - Type of data returned by the map function.

Parameters:

mapFunction - The map function.

Returns:

A new DataStream.

flatMap

public <R> DataStream<R> flatMap(FlatMapFunction<T,R> flatMapFunction)

Apply a flat-map function to this stream.

Type Parameters:

R - Type of data returned by the flatmap function.

Parameters:

flatMapFunction - The FlatMapFunction

Returns:

A new DataStream

filter

public DataStream<T> filter(FilterFunction<T> filterFunction)

union

@SafeVarargs
public final DataStream<T> union(DataStream<T> stream,
                                              DataStream<T>... others)

Apply union transformations to this stream by merging DataStream outputs of the same type with each other.

Parameters:

stream - The DataStream to union output with.

others - The other DataStreams to union output with.

Returns:

A new UnionStream.

union

public final DataStream<T> union(List<DataStream<T>> streams)

Apply union transformations to this stream by merging DataStream outputs of the same type with each other.

Parameters:

streams - The DataStreams to union output with.

Returns:

A new UnionStream.

join

public <O,R> JoinStream<T,O,R> join(DataStream<O> other)

Apply a join transformation to this stream, with another stream.

Type Parameters:

O - The type of the other stream data.

R - The type of the data in the joined stream.

Parameters:

other - Another stream.

Returns:

A new JoinStream.

process

public <R> DataStream<R> process()

sink

public DataStreamSink<T> sink(SinkFunction<T> sinkFunction)

Apply a sink function and get a StreamSink.

Parameters:

sinkFunction - The sink function.

Returns:

A new StreamSink.

keyBy

public <K> KeyDataStream<K,T> keyBy(KeyFunction<T,K> keyFunction)

Apply a key-by function to this stream.

Type Parameters:

K - The type of the key.

Parameters:

keyFunction - the key function.

Returns:

A new KeyDataStream.

broadcast

public DataStream<T> broadcast()

Apply broadcast to this stream.

Returns:

This stream.

partitionBy

public DataStream<T> partitionBy(Partition<T> partition)

Apply a partition to this stream.

Parameters:

partition - The partitioning strategy.

Returns:

This stream.

asPythonStream

public PythonDataStream asPythonStream()

Convert this stream as a python stream. The converted stream and this stream are the same logical stream, which has same stream id. Changes in converted stream will be reflected in this stream and vice versa.

getLanguage

public Language getLanguage()

Specified by:

getLanguage in class Stream<DataStream<T>,T>