Package dev.vortex.spark.io

Class VortexIo

java.lang.Object

dev.vortex.spark.io.VortexIo

All Implemented Interfaces:

Serializable

public final class VortexIo
extends Object
implements Serializable

The Hadoop configuration this Spark job reaches storage with, and the read settings that go with it.

File contents are read and written through Hadoop streams, so the connector sees the same schemes and credential providers as Spark's file index and commit protocol. Reads open a HadoopReadable; writes go through HadoopWritable on the task path Spark's commit protocol assigns.

Built on the driver and shipped to executors, so the configuration travels with it, or is broadcast to them.

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static final String	READ_CONCURRENCY_OPTION	Option bounding how many concurrent read upcalls the native reader issues against one file.

Method Summary

Modifier and Type	Method	Description
static VortexIo	broadcast(VortexOptions options, org.apache.hadoop.conf.Configuration hadoopConf, org.apache.spark.SparkContext context)	Like create(dev.vortex.spark.VortexOptions, org.apache.hadoop.conf.Configuration), but broadcasts hadoopConf through context.
static VortexIo	create(VortexOptions options, org.apache.hadoop.conf.Configuration hadoopConf)	Captures hadoopConf and the read settings found in the format options.
static VortexIo	defaults()	Hadoop I/O over a default configuration, for callers that have no Spark session to draw one from.
org.apache.hadoop.conf.Configuration	hadoopConf()
dev.vortex.io.NativeReadable	openReadable(VortexFile file)	Opens a byte source for file, stating it only if the listing that produced it did not report a size.
int	readConcurrency()	Bound on concurrent read upcalls per file; zero keeps the native default.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

READ_CONCURRENCY_OPTION

public static final String READ_CONCURRENCY_OPTION

Option bounding how many concurrent read upcalls the native reader issues against one file. Each in-flight upcall leases one pooled Hadoop input stream. Zero keeps the native default.

See Also:

• Constant Field Values

Method Details

create

public static VortexIo create(VortexOptions options,
 org.apache.hadoop.conf.Configuration hadoopConf)

Captures hadoopConf and the read settings found in the format options.

broadcast

public static VortexIo broadcast(VortexOptions options,
 org.apache.hadoop.conf.Configuration hadoopConf,
 org.apache.spark.SparkContext context)

Like create(dev.vortex.spark.VortexOptions, org.apache.hadoop.conf.Configuration), but broadcasts hadoopConf through context. Each task that deserializes the result then shares the executor's copy, instead of reading the whole configuration back from its own bytes.

defaults

public static VortexIo defaults()

Hadoop I/O over a default configuration, for callers that have no Spark session to draw one from.

hadoopConf

public org.apache.hadoop.conf.Configuration hadoopConf()

readConcurrency

public int readConcurrency()

Bound on concurrent read upcalls per file; zero keeps the native default.

openReadable

public dev.vortex.io.NativeReadable openReadable(VortexFile file)

Opens a byte source for file, stating it only if the listing that produced it did not report a size. The caller owns the result and must close it once the scan built on it is done.

The readers skip listed files without VortexFile.EXTENSION before they get here. The check stops any other path from opening such a file as Vortex data.

Throws:

IllegalArgumentException - if the path does not carry VortexFile.EXTENSION