2012-09-03 02:05:40 -04:00
---
layout: global
2012-09-12 22:47:31 -04:00
title: Spark Overview
2012-09-03 02:05:40 -04:00
---
2013-08-30 18:04:43 -04:00
Apache Spark is a cluster computing system that aims to make data analytics faster to run and faster to write.
It provides high-level APIs in [Scala ](scala-programming-guide.html ), [Java ](java-programming-guide.html ), and [Python ](python-programming-guide.html ), and a general execution engine that supports rich operator graphs.
2013-02-27 02:20:49 -05:00
Spark can run on the Apache Mesos cluster manager, Hadoop YARN, Amazon EC2, or without an independent resource manager ("standalone mode").
2012-09-03 02:05:40 -04:00
# Downloading
2013-08-30 18:04:43 -04:00
Get Spark by visiting the [downloads page ](http://spark.incubator.apache.org/downloads.html ) of the Apache Spark site. This documentation is for Spark version {{site.SPARK_VERSION}}.
2012-09-03 02:05:40 -04:00
# Building
2013-04-07 20:31:19 -04:00
Spark uses [Simple Build Tool ](http://www.scala-sbt.org ), which is bundled with it. To compile the code, go into the top-level Spark directory and run
2012-09-03 02:05:40 -04:00
2013-08-27 22:39:54 -04:00
sbt/sbt assembly
2012-09-03 02:05:40 -04:00
2013-08-30 18:04:43 -04:00
For its Scala API, Spark {{site.SPARK_VERSION}} depends on Scala {{site.SCALA_VERSION}}. If you write applications in Scala, you will need to use this same version of Scala in your own program -- newer major versions may not work. You can get the right version of Scala from [scala-lang.org ](http://www.scala-lang.org/download/ ).
2013-03-17 17:47:44 -04:00
2012-09-03 02:05:40 -04:00
# Testing the Build
2013-08-30 18:04:43 -04:00
Spark comes with several sample programs in the `examples` directory.
2013-08-24 02:30:17 -04:00
To run one of the samples, use `./run-example <class> <params>` in the top-level Spark directory
2013-08-30 18:04:43 -04:00
(the `run-example` script sets up the appropriate paths and launches that program).
For example, `./run-example spark.examples.SparkPi` will run a sample program that estimates Pi. Each
example prints usage help if no params are given.
2012-09-03 02:05:40 -04:00
2012-09-25 22:31:07 -04:00
Note that all of the sample programs take a `<master>` parameter specifying the cluster URL
2012-10-08 13:13:26 -04:00
to connect to. This can be a [URL for a distributed cluster ](scala-programming-guide.html#master-urls ),
2012-09-25 22:31:07 -04:00
or `local` to run locally with one thread, or `local[N]` to run locally with N threads. You should start by using
`local` for testing.
2012-09-03 02:05:40 -04:00
2013-08-30 18:04:43 -04:00
Finally, Spark can be used interactively through modified versions of the Scala shell (`./spark-shell`) or
Python interpreter (`./pyspark`). These are a great way to learn Spark.
2012-09-03 02:05:40 -04:00
2012-09-25 18:46:18 -04:00
# A Note About Hadoop Versions
2012-09-03 02:05:40 -04:00
2013-08-30 15:38:23 -04:00
Spark uses the Hadoop-client library to talk to HDFS and other Hadoop-supported
2012-09-03 02:05:40 -04:00
storage systems. Because the HDFS protocol has changed in different versions of
2013-08-30 15:38:23 -04:00
Hadoop, you must build Spark against the same version that your cluster uses.
You can do this by setting the `SPARK_HADOOP_VERSION` variable when compiling:
SPARK_HADOOP_VERSION=1.2.1 sbt/sbt assembly
In addition, if you wish to run Spark on [YARN ](running-on-yarn.md ), you should also
2013-08-30 18:04:43 -04:00
set `SPARK_YARN` :
2013-08-30 15:38:23 -04:00
SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly
2012-09-03 02:05:40 -04:00
# Where to Go from Here
2012-09-25 22:31:07 -04:00
**Programming guides:**
2012-09-26 02:59:04 -04:00
2012-10-09 17:30:23 -04:00
* [Quick Start ](quick-start.html ): a quick introduction to the Spark API; start here!
* [Spark Programming Guide ](scala-programming-guide.html ): an overview of Spark concepts, and details on the Scala API
2013-08-30 15:38:23 -04:00
* [Java Programming Guide ](java-programming-guide.html ): using Spark from Java
* [Python Programming Guide ](python-programming-guide.html ): using Spark from Python
* [Spark Streaming ](streaming-programming-guide.html ): using the alpha release of Spark Streaming
2013-08-31 17:21:10 -04:00
* [MLlib (Machine Learning) ](mllib-guide.html ): Spark's built-in machine learning library
2013-08-30 15:38:23 -04:00
* [Bagel (Pregel on Spark) ](bagel-programming-guide.html ): simple graph processing model
2012-09-25 18:46:18 -04:00
2013-01-01 16:52:14 -05:00
**API Docs:**
2013-08-30 15:38:23 -04:00
* [Spark for Java/Scala (Scaladoc) ](api/core/index.html )
* [Spark for Python (Epydoc) ](api/pyspark/index.html )
* [Spark Streaming for Java/Scala (Scaladoc) ](api/streaming/index.html )
* [MLlib (Machine Learning) for Java/Scala (Scaladoc) ](api/mllib/index.html )
* [Bagel (Pregel on Spark) for Scala (Scaladoc) ](api/bagel/index.html )
2012-09-25 18:46:18 -04:00
2012-09-25 22:31:07 -04:00
**Deployment guides:**
2012-09-26 02:59:04 -04:00
2012-10-08 13:13:26 -04:00
* [Running Spark on Amazon EC2 ](ec2-scripts.html ): scripts that let you launch a cluster on EC2 in about 5 minutes
2012-10-09 17:30:23 -04:00
* [Standalone Deploy Mode ](spark-standalone.html ): launch a standalone cluster quickly without a third-party cluster manager
2012-10-08 13:13:26 -04:00
* [Running Spark on Mesos ](running-on-mesos.html ): deploy a private cluster using
2012-09-25 18:46:18 -04:00
[Apache Mesos ](http://incubator.apache.org/mesos )
2012-10-08 13:13:26 -04:00
* [Running Spark on YARN ](running-on-yarn.html ): deploy Spark on top of Hadoop NextGen (YARN)
2012-09-25 18:46:18 -04:00
2012-09-25 22:31:07 -04:00
**Other documents:**
2012-09-26 02:59:04 -04:00
2012-10-08 13:13:26 -04:00
* [Configuration ](configuration.html ): customize Spark via its configuration system
* [Tuning Guide ](tuning.html ): best practices to optimize performance and memory use
2013-08-30 15:38:23 -04:00
* [Hardware Provisioning ](hardware-provisioning.html ): recommendations for cluster hardware
* [Building Spark with Maven ](building-with-maven.html ): Build Spark using the Maven build tool
2012-09-03 02:05:40 -04:00
* [Contributing to Spark ](contributing-to-spark.html )
2012-09-25 22:31:07 -04:00
**External resources:**
2012-09-26 02:59:04 -04:00
2013-08-30 15:38:23 -04:00
* [Spark Homepage ](http://spark.incubator.apache.org )
* [Mailing Lists ](http://spark.incubator.apache.org/mailing-lists.html ): ask questions about Spark here
* [AMP Camps ](http://ampcamp.berkeley.edu/ ): a series of training camps at UC Berkeley that featured talks and
exercises about Spark, Shark, Mesos, and more. [Videos ](http://ampcamp.berkeley.edu/agenda-2012 ),
2012-09-25 22:31:07 -04:00
[slides ](http://ampcamp.berkeley.edu/agenda-2012 ) and [exercises ](http://ampcamp.berkeley.edu/exercises-2012 ) are
available online for free.
2013-08-30 15:38:23 -04:00
* [Code Examples ](http://spark.incubator.apache.org/examples.html ): more are also available in the [examples subfolder ](https://github.com/mesos/spark/tree/master/examples/src/main/scala/spark/examples ) of Spark
2013-02-27 02:20:49 -05:00
* [Paper Describing Spark ](http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf )
* [Paper Describing Spark Streaming ](http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-259.pdf )
2012-09-03 02:05:40 -04:00
# Community
2013-08-30 15:38:23 -04:00
To get help using Spark or keep up with Spark development, sign up for the [user mailing list ](http://spark.incubator.apache.org/mailing-lists.html ).
2012-09-03 02:05:40 -04:00
If you're in the San Francisco Bay Area, there's a regular [Spark meetup ](http://www.meetup.com/spark-users/ ) every few weeks. Come by to meet the developers and other users.
2012-10-08 13:13:26 -04:00
Finally, if you'd like to contribute code to Spark, read [how to contribute ](contributing-to-spark.html ).