ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
Apache Spark - A unified analytics engine for large-scale data processing
2863 commits 6 branches 181 tags 397 MiB
Scala 65.7%
Python 12.1%
Jupyter Notebook 7.3%
Java 7.3%
HiveQL 3.2%
Other 4%
Go to file
Gavin Li 4508089fc3 refine comments and add sc.clean
2013-06-17 05:23:46 +00:00
bagel Attempt to fix streaming test failures after yarn branch merge 2013-04-28 22:26:45 +05:30
bin Revert "Fix start-slave not passing instance number to spark-daemon." 2013-06-11 00:08:02 -04:00
conf added SPARK_WORKER_INSTANCES : allows spawning multiple worker instances/processes on every slave machine 2013-03-26 17:49:30 -07:00
core refine comments and add sc.clean 2013-06-17 05:23:46 +00:00
docs Fixed a couple typos and formating problems in the YARN documentation. 2013-05-17 18:05:46 -07:00
ec2 Fix SPARK-670: EC2 start command should require -i option. 2013-05-05 11:30:36 -07:00
examples Revert "Merge pull request #596 from esjewett/master" because the 2013-05-09 14:20:01 -07:00
project Removed the duplicated netty dependency in SBT build file. 2013-05-16 14:31:03 -07:00
python Fix Python saveAsTextFile doctest to not expect order to be preserved 2013-04-02 11:59:20 -07:00
repl Remove spark-repl's extraneous dependency on spark-streaming 2013-05-01 16:57:31 -07:00
repl-bin Attempt at fixing merge conflict 2013-04-24 09:24:17 +05:30
sbt Increase ReservedCodeCacheSize for sbt 2013-04-16 09:50:59 -07:00
streaming Exclude old versions of Netty from Maven-based build 2013-05-18 21:24:58 -07:00
.gitignore add test for JdbcRDD using embedded derby, per rxin suggestion 2013-05-14 23:44:04 -05:00
kmeans_data.txt Fixed bugs 2012-01-09 11:59:52 -08:00
LICENSE Added BSD license 2010-12-07 10:32:17 -08:00
lr_data.txt Test commit 2012-02-06 09:58:06 -08:00
pom.xml Fixed jvmArgs in maven build. 2013-06-12 17:24:22 -07:00
pyspark Adding IPYTHON environment variable support for launching pyspark using ipython shell 2013-02-07 16:54:31 +02:00
README.md Add comment to README that 2.10 not yet supported 2013-03-26 14:28:08 -07:00
run Only check for repl classes if the user is running the repl. Otherwise, 2013-05-16 14:31:38 -07:00
run.cmd Add spark-shell.cmd 2012-09-25 07:26:29 -07:00
run2.cmd 1) Add support for HADOOP_CONF_DIR (and/or YARN_CONF_DIR - use either) : which is used to specify the client side configuration directory : which needs to be part of the CLASSPATH. 2013-05-11 11:12:22 +05:30
spark-executor Further refactoring, and start of a standalone scheduler backend 2012-07-06 17:56:44 -07:00
spark-shell More work to allow Spark to run on the standalone deploy cluster. 2012-07-08 14:00:04 -07:00
spark-shell.cmd Add spark-shell.cmd 2012-09-25 07:26:29 -07:00

README.md

Spark

Lightning-Fast Cluster Computing - http://www.spark-project.org/

Online Documentation

You can find the latest Spark documentation, including a programming guide, on the project webpage at http://spark-project.org/documentation.html. This README file only contains basic setup instructions.

Building

Spark requires Scala 2.9.2 (Scala 2.10 is not yet supported). The project is built using Simple Build Tool (SBT), which is packaged with it. To build Spark and its example programs, run:

sbt/sbt package

Spark also supports building using Maven. If you would like to build using Maven, see the instructions for building Spark with Maven in the spark documentation..

To run Spark, you will need to have Scala's bin directory in your PATH, or you will need to set the SCALA_HOME environment variable to point to where you've installed Scala. Scala must be accessible through one of these methods on your cluster's worker nodes as well as its master.

To run one of the examples, use ./run <class> <params>. For example:

./run spark.examples.SparkLR local[2]

will run the Logistic Regression example locally on 2 CPUs.

Each of the example programs prints usage help if no params are given.

All of the Spark samples take a <host> parameter that is the cluster URL to connect to. This can be a mesos:// or spark:// URL, or "local" to run locally with one thread, or "local[N]" to run locally with N threads.

A Note About Hadoop Versions

Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported storage systems. Because the HDFS API has changed in different versions of Hadoop, you must build Spark against the same version that your cluster runs. You can change the version by setting the HADOOP_VERSION variable at the top of project/SparkBuild.scala, then rebuilding Spark.

Configuration

Please refer to the "Configuration" guide in the online documentation for a full overview on how to configure Spark. At the minimum, you will need to create a conf/spark-env.sh script (copy conf/spark-env.sh.template) and set the following two variables:

  • SCALA_HOME: Location where Scala is installed.

  • MESOS_NATIVE_LIBRARY: Your Mesos library (only needed if you want to run on Mesos). For example, this might be /usr/local/lib/libmesos.so on Linux.

Contributing to Spark

Contributions via GitHub pull requests are gladly accepted from their original author. Along with any pull requests, please state that the contribution is your original work and that you license the work to the project under the project's open source license. Whether or not you state this explicitly, by submitting any copyrighted material via pull request, email, or other means you agree to license the material under the project's open source license and warrant that you have the legal authority to do so.