README

php-spark is a wrapper around arrays that mimics the MapReduce API of Apache Spark.

$data = new Dataset([1, 2, 3, 4]);
$result = $data
	->map(function ($v) {
		return 2 * $v;
	})
	->reduce(function ($v, $agg) {
		return $agg + $v;
	});

$result == 20;

php-spark is NOT a PHP driver for Apache Spark (and I wish this would exist).

Dataset

A dataset is an immutable array of data. It is the equivalent of Spark RDD (Resilient Distributed Dataset).

use Cylab\Spark\Dataset;

$d = new Dataset([1, 2, 3, 4]);
var_dump($d->collect());

Transformations

Transformations return another dataset.

Actions

Actions return other types of result.

Map

Map applies the provided function to all elements in the dataset and returns a new dataset containing the result of the map operation.

$d2 = $d->map(function ($v) { return 2 * $v; });

Reduce

The reduce function you provide must take two parameters: the current value and the aggregated value.

use Cylab\Spark\Dataset;

$d = new Dataset([1, 2, 3, 4]);
$result = $d->reduce(function ($v, $agg) {
	return $agg + $v;
});

$result == 10;

Tuple

Some methods expect the dataset to contain a list of <key, value> tuples.

use Cylab\Spark\Dataset;
use Cylab\Spark\Tuple;

$strings = ["foe", "bar", "foe"];
$d = new Dataset ($strings);
$d2 = $d->map(function($s) { return new Tuple($s, 1); });

ReduceByKey

For this method to work, the input dataset must be a list of <key, value> tuples. The reduce function is then applied to all elements with the same key.

$counts = $d2->reduceByKey(function ($count, $sum) {
	return $sum + $count;
});

First

Get the first element of a dataset.

// Tuple<"foe", 2>
var_dump($counts->first());

Future work

• flatMap
• sample
• union
• intersection
• aggregateByKey
• sortByKey
• join
• cartesian
• takeSample
• takeOrdered
• countByKey
• saveAsObjectFile
• saveAsJsonFile