2012-09-03 02:05:40 -04:00
|
|
|
---
|
|
|
|
layout: global
|
|
|
|
title: Running Spark on Mesos
|
|
|
|
---
|
|
|
|
|
2014-05-14 12:45:33 -04:00
|
|
|
Spark can run on hardware clusters managed by [Apache Mesos](http://mesos.apache.org/).
|
|
|
|
|
|
|
|
The advantages of deploying Spark with Mesos include:
|
2014-05-16 20:35:05 -04:00
|
|
|
|
2014-05-14 12:45:33 -04:00
|
|
|
- dynamic partitioning between Spark and other
|
|
|
|
[frameworks](https://mesos.apache.org/documentation/latest/mesos-frameworks/)
|
|
|
|
- scalable partitioning between multiple instances of Spark
|
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
# How it Works
|
2014-05-14 12:45:33 -04:00
|
|
|
|
|
|
|
In a standalone cluster deployment, the cluster manager in the below diagram is a Spark master
|
|
|
|
instance. When using Mesos, the Mesos master replaces the Spark master as the cluster manager.
|
|
|
|
|
|
|
|
<p style="text-align: center;">
|
|
|
|
<img src="img/cluster-overview.png" title="Spark cluster components" alt="Spark cluster components" />
|
|
|
|
</p>
|
|
|
|
|
|
|
|
Now when a driver creates a job and starts issuing tasks for scheduling, Mesos determines what
|
|
|
|
machines handle what tasks. Because it takes into account other frameworks when scheduling these
|
|
|
|
many short-lived tasks, multiple frameworks can coexist on the same cluster without resorting to a
|
|
|
|
static partitioning of resources.
|
|
|
|
|
|
|
|
To get started, follow the steps below to install Mesos and deploy Spark jobs via Mesos.
|
|
|
|
|
|
|
|
|
|
|
|
# Installing Mesos
|
|
|
|
|
|
|
|
Spark {{site.SPARK_VERSION}} is designed for use with Mesos {{site.MESOS_VERSION}} and does not
|
|
|
|
require any special patches of Mesos.
|
|
|
|
|
|
|
|
If you already have a Mesos cluster running, you can skip this Mesos installation step.
|
|
|
|
|
|
|
|
Otherwise, installing Mesos for Spark is no different than installing Mesos for use by other
|
2014-05-16 20:35:05 -04:00
|
|
|
frameworks. You can install Mesos either from source or using prebuilt packages.
|
|
|
|
|
|
|
|
## From Source
|
|
|
|
|
|
|
|
To install Apache Mesos from source, follow these steps:
|
|
|
|
|
|
|
|
1. Download a Mesos release from a
|
|
|
|
[mirror](http://www.apache.org/dyn/closer.cgi/mesos/{{site.MESOS_VERSION}}/)
|
|
|
|
2. Follow the Mesos [Getting Started](http://mesos.apache.org/gettingstarted) page for compiling and
|
|
|
|
installing Mesos
|
|
|
|
|
|
|
|
**Note:** If you want to run Mesos without installing it into the default paths on your system
|
|
|
|
(e.g., if you lack administrative privileges to install it), pass the
|
|
|
|
`--prefix` option to `configure` to tell it where to install. For example, pass
|
|
|
|
`--prefix=/home/me/mesos`. By default the prefix is `/usr/local`.
|
2014-05-14 12:45:33 -04:00
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
## Third-Party Packages
|
2014-05-14 12:45:33 -04:00
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
The Apache Mesos project only publishes source releases, not binary packages. But other
|
2014-05-14 12:45:33 -04:00
|
|
|
third party projects publish binary releases that may be helpful in setting Mesos up.
|
|
|
|
|
|
|
|
One of those is Mesosphere. To install Mesos using the binary releases provided by Mesosphere:
|
|
|
|
|
|
|
|
1. Download Mesos installation package from [downloads page](http://mesosphere.io/downloads/)
|
|
|
|
2. Follow their instructions for installation and configuration
|
|
|
|
|
|
|
|
The Mesosphere installation documents suggest setting up ZooKeeper to handle Mesos master failover,
|
|
|
|
but Mesos can be run without ZooKeeper using a single master as well.
|
|
|
|
|
|
|
|
## Verification
|
|
|
|
|
|
|
|
To verify that the Mesos cluster is ready for Spark, navigate to the Mesos master webui at port
|
|
|
|
`:5050` Confirm that all expected machines are present in the slaves tab.
|
|
|
|
|
|
|
|
|
|
|
|
# Connecting Spark to Mesos
|
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
To use Mesos from Spark, you need a Spark binary package available in a place accessible by Mesos, and
|
2014-05-14 12:45:33 -04:00
|
|
|
a Spark driver program configured to connect to Mesos.
|
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
## Uploading Spark Package
|
2014-05-14 12:45:33 -04:00
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
When Mesos runs a task on a Mesos slave for the first time, that slave must have a Spark binary
|
|
|
|
package for running the Spark Mesos executor backend.
|
|
|
|
The Spark package can be hosted at any Hadoop-accessible URI, including HTTP via `http://`,
|
|
|
|
[Amazon Simple Storage Service](http://aws.amazon.com/s3) via `s3n://`, or HDFS via `hdfs://`.
|
2014-05-14 12:45:33 -04:00
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
To use a precompiled package:
|
2014-05-14 12:45:33 -04:00
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
1. Download a Spark binary package from the Spark [download page](https://spark.apache.org/downloads.html)
|
2014-05-14 12:45:33 -04:00
|
|
|
2. Upload to hdfs/http/s3
|
|
|
|
|
|
|
|
To host on HDFS, use the Hadoop fs put command: `hadoop fs -put spark-{{site.SPARK_VERSION}}.tar.gz
|
|
|
|
/path/to/spark-{{site.SPARK_VERSION}}.tar.gz`
|
|
|
|
|
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
Or if you are using a custom-compiled version of Spark, you will need to create a package using
|
2014-05-14 12:45:33 -04:00
|
|
|
the `make-distribution.sh` script included in a Spark source tarball/checkout.
|
|
|
|
|
|
|
|
1. Download and build Spark using the instructions [here](index.html)
|
2014-05-16 20:35:05 -04:00
|
|
|
2. Create a binary package using `make-distribution.sh --tgz`.
|
2014-05-14 12:45:33 -04:00
|
|
|
3. Upload archive to http/s3/hdfs
|
|
|
|
|
|
|
|
|
|
|
|
## Using a Mesos Master URL
|
|
|
|
|
|
|
|
The Master URLs for Mesos are in the form `mesos://host:5050` for a single-master Mesos
|
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
|
|
|
cluster, or `mesos://zk://host:2181` for a multi-master Mesos cluster using ZooKeeper.
|
2014-05-14 12:45:33 -04:00
|
|
|
|
|
|
|
The driver also needs some configuration in `spark-env.sh` to interact properly with Mesos:
|
|
|
|
|
|
|
|
1. In `spark.env.sh` set some environment variables:
|
|
|
|
* `export MESOS_NATIVE_LIBRARY=<path to libmesos.so>`. This path is typically
|
|
|
|
`<prefix>/lib/libmesos.so` where the prefix is `/usr/local` by default. See Mesos installation
|
|
|
|
instructions above. On Mac OS X, the library is called `libmesos.dylib` instead of
|
|
|
|
`libmesos.so`.
|
2014-05-16 20:35:05 -04:00
|
|
|
* `export SPARK_EXECUTOR_URI=<URL of spark-{{site.SPARK_VERSION}}.tar.gz uploaded above>`.
|
|
|
|
2. Also set `spark.executor.uri` to `<URL of spark-{{site.SPARK_VERSION}}.tar.gz>`.
|
2014-05-14 12:45:33 -04:00
|
|
|
|
|
|
|
Now when starting a Spark application against the cluster, pass a `mesos://`
|
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
|
|
|
URL as the master when creating a `SparkContext`. For example:
|
2012-09-13 02:05:47 -04:00
|
|
|
|
|
|
|
{% highlight scala %}
|
2013-12-30 22:17:28 -05:00
|
|
|
val conf = new SparkConf()
|
|
|
|
.setMaster("mesos://HOST:5050")
|
|
|
|
.setAppName("My app")
|
|
|
|
.set("spark.executor.uri", "<path to spark-{{site.SPARK_VERSION}}.tar.gz uploaded above>")
|
|
|
|
val sc = new SparkContext(conf)
|
2012-09-13 02:05:47 -04:00
|
|
|
{% endhighlight %}
|
2012-09-03 02:05:40 -04:00
|
|
|
|
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
|
|
|
(You can also use [`spark-submit`](submitting-applications.html) and configure `spark.executor.uri`
|
|
|
|
in the [conf/spark-defaults.conf](configuration.html#loading-default-configurations) file.)
|
|
|
|
|
2014-05-16 20:35:05 -04:00
|
|
|
When running a shell, the `spark.executor.uri` parameter is inherited from `SPARK_EXECUTOR_URI`, so
|
2014-05-14 12:45:33 -04:00
|
|
|
it does not need to be redundantly passed in as a system property.
|
|
|
|
|
|
|
|
{% highlight bash %}
|
|
|
|
./bin/spark-shell --master mesos://host:5050
|
|
|
|
{% endhighlight %}
|
|
|
|
|
|
|
|
|
2012-09-27 01:54:39 -04:00
|
|
|
# Mesos Run Modes
|
|
|
|
|
2014-05-14 12:45:33 -04:00
|
|
|
Spark can run over Mesos in two modes: "fine-grained" (default) and "coarse-grained".
|
|
|
|
|
|
|
|
In "fine-grained" mode (default), each Spark task runs as a separate Mesos task. This allows
|
|
|
|
multiple instances of Spark (and other frameworks) to share machines at a very fine granularity,
|
|
|
|
where each application gets more or fewer machines as it ramps up and down, but it comes with an
|
|
|
|
additional overhead in launching each task. This mode may be inappropriate for low-latency
|
|
|
|
requirements like interactive queries or serving web requests.
|
|
|
|
|
|
|
|
The "coarse-grained" mode will instead launch only *one* long-running Spark task on each Mesos
|
|
|
|
machine, and dynamically schedule its own "mini-tasks" within it. The benefit is much lower startup
|
|
|
|
overhead, but at the cost of reserving the Mesos resources for the complete duration of the
|
|
|
|
application.
|
2012-09-27 01:54:39 -04:00
|
|
|
|
2014-05-14 12:45:33 -04:00
|
|
|
To run in coarse-grained mode, set the `spark.mesos.coarse` property in your
|
|
|
|
[SparkConf](configuration.html#spark-properties):
|
2012-09-27 01:54:39 -04:00
|
|
|
|
|
|
|
{% highlight scala %}
|
2013-12-30 22:17:28 -05:00
|
|
|
conf.set("spark.mesos.coarse", "true")
|
2012-09-27 01:54:39 -04:00
|
|
|
{% endhighlight %}
|
|
|
|
|
2014-05-14 12:45:33 -04:00
|
|
|
In addition, for coarse-grained mode, you can control the maximum number of resources Spark will
|
|
|
|
acquire. By default, it will acquire *all* cores in the cluster (that get offered by Mesos), which
|
|
|
|
only makes sense if you run just one application at a time. You can cap the maximum number of cores
|
|
|
|
using `conf.set("spark.cores.max", "10")` (for example).
|
2012-09-27 01:54:39 -04:00
|
|
|
|
|
|
|
|
|
|
|
# Running Alongside Hadoop
|
2012-09-03 02:05:40 -04:00
|
|
|
|
2014-05-14 12:45:33 -04:00
|
|
|
You can run Spark and Mesos alongside your existing Hadoop cluster by just launching them as a
|
2014-05-16 20:35:05 -04:00
|
|
|
separate service on the machines. To access Hadoop data from Spark, a full `hdfs://` URL is required
|
2014-05-14 12:45:33 -04:00
|
|
|
(typically `hdfs://<namenode>:9000/path`, but you can find the right URL on your Hadoop Namenode web
|
|
|
|
UI).
|
|
|
|
|
|
|
|
In addition, it is possible to also run Hadoop MapReduce on Mesos for better resource isolation and
|
|
|
|
sharing between the two. In this case, Mesos will act as a unified scheduler that assigns cores to
|
|
|
|
either Hadoop or Spark, as opposed to having them share resources via the Linux scheduler on each
|
|
|
|
node. Please refer to [Hadoop on Mesos](https://github.com/mesos/hadoop).
|
|
|
|
|
|
|
|
In either case, HDFS runs separately from Hadoop MapReduce, without being scheduled through Mesos.
|
|
|
|
|
|
|
|
|
|
|
|
# Troubleshooting and Debugging
|
|
|
|
|
|
|
|
A few places to look during debugging:
|
|
|
|
|
|
|
|
- Mesos master on port `:5050`
|
|
|
|
- Slaves should appear in the slaves tab
|
|
|
|
- Spark applications should appear in the frameworks tab
|
|
|
|
- Tasks should appear in the details of a framework
|
|
|
|
- Check the stdout and stderr of the sandbox of failed tasks
|
|
|
|
- Mesos logs
|
|
|
|
- Master and slave logs are both in `/var/log/mesos` by default
|
2012-09-03 02:05:40 -04:00
|
|
|
|
2014-05-14 12:45:33 -04:00
|
|
|
And common pitfalls:
|
2012-09-03 02:05:40 -04:00
|
|
|
|
2014-05-14 12:45:33 -04:00
|
|
|
- Spark assembly not reachable/accessible
|
2014-05-16 20:35:05 -04:00
|
|
|
- Slaves must be able to download the Spark binary package from the `http://`, `hdfs://` or `s3n://` URL you gave
|
2014-05-14 12:45:33 -04:00
|
|
|
- Firewall blocking communications
|
|
|
|
- Check for messages about failed connections
|
|
|
|
- Temporarily disable firewalls for debugging and then poke appropriate holes
|