2012-09-03 02:05:40 -04:00
---
layout: global
2012-09-12 22:47:31 -04:00
title: Spark Overview
2012-09-03 02:05:40 -04:00
---
2013-09-01 01:17:40 -04:00
Apache Spark is a fast and general-purpose cluster computing system.
It provides high-level APIs in [Scala ](scala-programming-guide.html ), [Java ](java-programming-guide.html ), and [Python ](python-programming-guide.html ) that make parallel jobs easy to write, and an optimized engine that supports general computation graphs.
2013-09-02 16:35:28 -04:00
It also supports a rich set of higher-level tools including [Shark ](http://shark.cs.berkeley.edu ) (Hive on Spark), [MLlib ](mllib-guide.html ) for machine learning, [Bagel ](bagel-programming-guide.html ) for graph processing, and [Spark Streaming ](streaming-programming-guide.html ).
2012-09-03 02:05:40 -04:00
# Downloading
2013-08-30 18:04:43 -04:00
Get Spark by visiting the [downloads page ](http://spark.incubator.apache.org/downloads.html ) of the Apache Spark site. This documentation is for Spark version {{site.SPARK_VERSION}}.
2012-09-03 02:05:40 -04:00
2013-09-02 01:12:03 -04:00
Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). All you need to run it is to have `java` to installed on your system `PATH` , or the `JAVA_HOME` environment variable pointing to a Java installation.
2012-09-03 02:05:40 -04:00
# Building
2013-04-07 20:31:19 -04:00
Spark uses [Simple Build Tool ](http://www.scala-sbt.org ), which is bundled with it. To compile the code, go into the top-level Spark directory and run
2012-09-03 02:05:40 -04:00
2013-08-27 22:39:54 -04:00
sbt/sbt assembly
2012-09-03 02:05:40 -04:00
2013-08-30 18:04:43 -04:00
For its Scala API, Spark {{site.SPARK_VERSION}} depends on Scala {{site.SCALA_VERSION}}. If you write applications in Scala, you will need to use this same version of Scala in your own program -- newer major versions may not work. You can get the right version of Scala from [scala-lang.org ](http://www.scala-lang.org/download/ ).
2013-03-17 17:47:44 -04:00
2013-09-06 00:29:37 -04:00
# Running the Examples and Shell
2012-09-03 02:05:40 -04:00
2013-08-30 18:04:43 -04:00
Spark comes with several sample programs in the `examples` directory.
2013-08-24 02:30:17 -04:00
To run one of the samples, use `./run-example <class> <params>` in the top-level Spark directory
2013-08-30 18:04:43 -04:00
(the `run-example` script sets up the appropriate paths and launches that program).
2013-09-01 01:17:40 -04:00
For example, try `./run-example org.apache.spark.examples.SparkPi local` .
Each example prints usage help when run with no parameters.
2012-09-03 02:05:40 -04:00
2012-09-25 22:31:07 -04:00
Note that all of the sample programs take a `<master>` parameter specifying the cluster URL
2012-10-08 13:13:26 -04:00
to connect to. This can be a [URL for a distributed cluster ](scala-programming-guide.html#master-urls ),
2012-09-25 22:31:07 -04:00
or `local` to run locally with one thread, or `local[N]` to run locally with N threads. You should start by using
`local` for testing.
2012-09-03 02:05:40 -04:00
2013-09-06 00:29:37 -04:00
Finally, you can run Spark interactively through modified versions of the Scala shell (`./spark-shell`) or
Python interpreter (`./pyspark`). These are a great way to learn the framework.
2012-09-03 02:05:40 -04:00
2013-09-06 00:29:37 -04:00
# Launching on a Cluster
2013-09-02 16:35:28 -04:00
2013-09-06 00:29:37 -04:00
The Spark [cluster mode overview ](cluster-overview.html ) explains the key concepts in running on a cluster.
Spark can run both by itself, or over several existing cluster managers. It currently provides several
options for deployment:
2013-09-02 16:35:28 -04:00
2013-09-06 00:29:37 -04:00
* [Amazon EC2 ](ec2-scripts.html ): our EC2 scripts let you launch a cluster in about 5 minutes
2013-09-02 16:35:28 -04:00
* [Standalone Deploy Mode ](spark-standalone.html ): simplest way to deploy Spark on a private cluster
* [Apache Mesos ](running-on-mesos.html )
* [Hadoop YARN ](running-on-yarn.html )
2012-09-25 18:46:18 -04:00
# A Note About Hadoop Versions
2012-09-03 02:05:40 -04:00
2013-08-30 15:38:23 -04:00
Spark uses the Hadoop-client library to talk to HDFS and other Hadoop-supported
2012-09-03 02:05:40 -04:00
storage systems. Because the HDFS protocol has changed in different versions of
2013-08-30 15:38:23 -04:00
Hadoop, you must build Spark against the same version that your cluster uses.
2013-08-31 20:40:33 -04:00
By default, Spark links to Hadoop 1.0.4. You can change this by setting the
`SPARK_HADOOP_VERSION` variable when compiling:
2013-08-30 15:38:23 -04:00
SPARK_HADOOP_VERSION=1.2.1 sbt/sbt assembly
2013-08-31 20:40:33 -04:00
In addition, if you wish to run Spark on [YARN ](running-on-yarn.md ), set
`SPARK_YARN` to `true` :
2013-08-30 15:38:23 -04:00
SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly
2012-09-03 02:05:40 -04:00
2013-09-02 01:12:03 -04:00
(Note that on Windows, you need to set the environment variables on separate lines, e.g., `set SPARK_HADOOP_VERSION=1.2.1` .)
2012-09-03 02:05:40 -04:00
# Where to Go from Here
2012-09-25 22:31:07 -04:00
**Programming guides:**
2012-09-26 02:59:04 -04:00
2012-10-09 17:30:23 -04:00
* [Quick Start ](quick-start.html ): a quick introduction to the Spark API; start here!
* [Spark Programming Guide ](scala-programming-guide.html ): an overview of Spark concepts, and details on the Scala API
2013-08-30 15:38:23 -04:00
* [Java Programming Guide ](java-programming-guide.html ): using Spark from Java
* [Python Programming Guide ](python-programming-guide.html ): using Spark from Python
* [Spark Streaming ](streaming-programming-guide.html ): using the alpha release of Spark Streaming
2013-08-31 17:21:10 -04:00
* [MLlib (Machine Learning) ](mllib-guide.html ): Spark's built-in machine learning library
2013-08-30 15:38:23 -04:00
* [Bagel (Pregel on Spark) ](bagel-programming-guide.html ): simple graph processing model
2012-09-25 18:46:18 -04:00
2013-01-01 16:52:14 -05:00
**API Docs:**
2013-08-30 15:38:23 -04:00
* [Spark for Java/Scala (Scaladoc) ](api/core/index.html )
* [Spark for Python (Epydoc) ](api/pyspark/index.html )
* [Spark Streaming for Java/Scala (Scaladoc) ](api/streaming/index.html )
* [MLlib (Machine Learning) for Java/Scala (Scaladoc) ](api/mllib/index.html )
* [Bagel (Pregel on Spark) for Scala (Scaladoc) ](api/bagel/index.html )
2012-09-25 18:46:18 -04:00
2012-09-25 22:31:07 -04:00
**Deployment guides:**
2012-09-26 02:59:04 -04:00
2013-09-06 00:29:37 -04:00
* [Cluster Overview ](cluster-overview.html ): overview of concepts and components when running on a cluster
* [Amazon EC2 ](ec2-scripts.html ): scripts that let you launch a cluster on EC2 in about 5 minutes
2012-10-09 17:30:23 -04:00
* [Standalone Deploy Mode ](spark-standalone.html ): launch a standalone cluster quickly without a third-party cluster manager
2013-09-06 00:29:37 -04:00
* [Mesos ](running-on-mesos.html ): deploy a private cluster using
2012-09-25 18:46:18 -04:00
[Apache Mesos ](http://incubator.apache.org/mesos )
2013-09-06 00:29:37 -04:00
* [YARN ](running-on-yarn.html ): deploy Spark on top of Hadoop NextGen (YARN)
2012-09-25 18:46:18 -04:00
2012-09-25 22:31:07 -04:00
**Other documents:**
2012-09-26 02:59:04 -04:00
2012-10-08 13:13:26 -04:00
* [Configuration ](configuration.html ): customize Spark via its configuration system
* [Tuning Guide ](tuning.html ): best practices to optimize performance and memory use
2013-08-30 15:38:23 -04:00
* [Hardware Provisioning ](hardware-provisioning.html ): recommendations for cluster hardware
2013-09-06 00:29:37 -04:00
* [Job Scheduling ](job-scheduling.html ): scheduling resources across and within Spark applications
* [Building Spark with Maven ](building-with-maven.html ): build Spark using the Maven system
* [Contributing to Spark ](https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark )
2012-09-03 02:05:40 -04:00
2012-09-25 22:31:07 -04:00
**External resources:**
2012-09-26 02:59:04 -04:00
2013-08-30 15:38:23 -04:00
* [Spark Homepage ](http://spark.incubator.apache.org )
2013-09-02 16:35:28 -04:00
* [Shark ](http://shark.cs.berkeley.edu ): Apache Hive over Spark
2013-08-30 15:38:23 -04:00
* [Mailing Lists ](http://spark.incubator.apache.org/mailing-lists.html ): ask questions about Spark here
* [AMP Camps ](http://ampcamp.berkeley.edu/ ): a series of training camps at UC Berkeley that featured talks and
exercises about Spark, Shark, Mesos, and more. [Videos ](http://ampcamp.berkeley.edu/agenda-2012 ),
2012-09-25 22:31:07 -04:00
[slides ](http://ampcamp.berkeley.edu/agenda-2012 ) and [exercises ](http://ampcamp.berkeley.edu/exercises-2012 ) are
available online for free.
2013-09-01 01:38:50 -04:00
* [Code Examples ](http://spark.incubator.apache.org/examples.html ): more are also available in the [examples subfolder ](https://github.com/apache/incubator-spark/tree/master/examples/src/main/scala/ ) of Spark
2013-02-27 02:20:49 -05:00
* [Paper Describing Spark ](http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf )
* [Paper Describing Spark Streaming ](http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-259.pdf )
2012-09-03 02:05:40 -04:00
# Community
2013-08-30 15:38:23 -04:00
To get help using Spark or keep up with Spark development, sign up for the [user mailing list ](http://spark.incubator.apache.org/mailing-lists.html ).
2012-09-03 02:05:40 -04:00
If you're in the San Francisco Bay Area, there's a regular [Spark meetup ](http://www.meetup.com/spark-users/ ) every few weeks. Come by to meet the developers and other users.
2012-10-08 13:13:26 -04:00
Finally, if you'd like to contribute code to Spark, read [how to contribute ](contributing-to-spark.html ).