ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
8151 commits 6 branches 181 tags 397 MiB
Commit graph

69 commits

Author SHA1 Message Date
Sandy Ryza 5f419bf9f4 SPARK-1032. If Yarn app fails before registering, app master stays aroun... 
...d long after

This reopens https://github.com/apache/incubator-spark/pull/648 against the new repo.

Author: Sandy Ryza <sandy@cloudera.com>

Closes #28 from sryza/sandy-spark-1032 and squashes the following commits:

5953f50 [Sandy Ryza] SPARK-1032. If Yarn app fails before registering, app master stays around long after
 2014-02-28 09:40:47 -06:00
Mark Hamstra c2341c92bb Merge pull request #542 from markhamstra/versionBump. Closes #542. 
Version number to 1.0.0-SNAPSHOT

Since 0.9.0-incubating is done and out the door, we shouldn't be building 0.9.0-incubating-SNAPSHOT anymore.

@pwendell

Author: Mark Hamstra <markhamstra@gmail.com>

== Merge branch commits ==

commit 1b00a8a7c1a7f251b4bb3774b84b9e64758eaa71
Author: Mark Hamstra <markhamstra@gmail.com>
Date:   Wed Feb 5 09:30:32 2014 -0800

    Version number to 1.0.0-SNAPSHOT
 2014-02-08 16:00:43 -08:00
Thomas Graves 38020961d1 Merge pull request #526 from tgravescs/yarn_client_stop_am_fix. Closes #526. 
spark on yarn - yarn-client mode doesn't always exit immediately

https://spark-project.atlassian.net/browse/SPARK-1049

If you run in the yarn-client mode but you don't get all the workers you requested right away and then you exit your application, the application master stays around until it gets the number of workers you initially requested. This is a waste of resources.  The AM should exit immediately upon the client going away.

This fix simply checks to see if the driver closed while its waiting for the initial # of workers.

Author: Thomas Graves <tgraves@apache.org>

== Merge branch commits ==

commit 03f40a62584b6bdd094ba91670cd4aa6afe7cd81
Author: Thomas Graves <tgraves@apache.org>
Date:   Fri Jan 31 11:23:10 2014 -0600

    spark on yarn - yarn-client mode doesn't always exit immediately
 2014-02-05 23:37:07 -08:00
Sandy Ryza adf42611f1 Incorporate Tom's comments - update doc and code to reflect that core requests may not always be honored 2014-01-21 00:38:02 -08:00
Sandy Ryza 3e85b87d90 SPARK-1033. Ask for cores in Yarn container requests 2014-01-20 14:42:32 -08:00
Raymond Liu 4c22c55ad6 Address comments to fix code formats 2014-01-14 10:41:42 +08:00
Raymond Liu 161ab93989 Yarn workerRunnable refactor 2014-01-14 10:36:00 +08:00
Raymond Liu 79a5ba3497 Yarn Client refactor 2014-01-14 10:33:48 +08:00
Henry Saputra 5a8abfb70e Address code review concerns and comments. 2014-01-12 19:15:09 -08:00
Henry Saputra 91a563608e Merge branch 'master' into remove_simpleredundantreturn_scala 2014-01-12 10:34:13 -08:00
Henry Saputra 93a65e5fde Remove simple redundant return statement for Scala methods/functions: 
-) Only change simple return statements at the end of method
-) Ignore the complex if-else check
-) Ignore the ones inside synchronized
 2014-01-12 10:30:04 -08:00
Thomas Graves 9bdfbc0492 Update based on review comments 2014-01-09 15:45:33 -06:00
Thomas Graves 5fccd7051d add application type 2014-01-09 10:32:35 -06:00
Thomas Graves c617083e47 yarn-client addJar fix and misc other 2014-01-09 10:24:35 -06:00
Raymond Liu da4694a0d8 Minor typo fix for yarn client 2014-01-07 10:24:10 +08:00
liguoqiang 8ddbd531a4 merge upstream/master 2014-01-03 16:06:34 +08:00
Raymond Liu 18b3633e54 minor fix for loginfo 2014-01-03 12:14:38 +08:00
Raymond Liu c59029402d move duplicate pom config into parent pom 2014-01-03 12:14:38 +08:00
Raymond Liu ebdfa6bb97 Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2 2014-01-03 12:14:38 +08:00