41 lines
1.8 KiB
Plaintext
41 lines
1.8 KiB
Plaintext
BUILDING
|
|
|
|
Spark requires Scala 2.8. This version has been tested with 2.8.0.final.
|
|
|
|
To build and run Spark, you will need to have Scala's bin in your $PATH,
|
|
or you will need to set the SCALA_HOME environment variable to point
|
|
to where you've installed Scala. Scala must be accessible through one
|
|
of these methods on Mesos slave nodes as well as on the master.
|
|
|
|
To build Spark and the example programs, run make.
|
|
|
|
To run one of the examples, use ./run <class> <params>. For example,
|
|
./run SparkLR will run the Logistic Regression example. Each of the
|
|
example programs prints usage help if no params are given.
|
|
|
|
All of the Spark samples take a <host> parameter that is the Mesos master
|
|
to connect to. This can be a Mesos URL, or "local" to run locally with one
|
|
thread, or "local[N]" to run locally with N threads.
|
|
|
|
Tip: If you are building Spark and examples repeatedly, export USE_FSC=1
|
|
to have the Makefile use the fsc compiler daemon instead of scalac.
|
|
|
|
CONFIGURATION
|
|
|
|
Spark can be configured through two files: conf/java-opts and conf/spark-env.sh.
|
|
|
|
In java-opts, you can add flags to be passed to the JVM when running Spark.
|
|
|
|
In spark-env.sh, you can set any environment variables you wish to be available
|
|
when running Spark programs, such as PATH, SCALA_HOME, etc. There are also
|
|
several Spark-specific variables you can set:
|
|
- SPARK_CLASSPATH: Extra entries to be added to the classpath, separated by ":".
|
|
- SPARK_MEM: Memory for Spark to use, in the format used by java's -Xmx option
|
|
(for example, 200m meams 200 MB, 1g means 1 GB, etc).
|
|
- SPARK_LIBRARY_PATH: Extra entries to add to java.library.path for locating
|
|
shared libraries.
|
|
- SPARK_JAVA_OPTS: Extra options to pass to JVM.
|
|
|
|
Note that spark-env.sh must be a shell script (it must be executable and start
|
|
with a #! header to specify the shell to use).
|