ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
18253 commits 6 branches 181 tags 397 MiB
Commit graph

546 commits

Author SHA1 Message Date
Henry Saputra 9c934b640f Remove the semicolons at the end of Scala code to make it more pure Scala code. 
Also remove unused imports as I found them along the way.
Remove return statements when returning value in the Scala code.

Passing compile and tests.
 2013-11-19 10:19:03 -08:00
Harvey Feng a98f5a0ebb Misc style changes in the 'yarn' package. 2013-11-17 22:35:56 -08:00
Henry Saputra c33f802044 Simple cleanup on Spark's Scala code while testing core and yarn modules: 
-) Remove some of unused imports as I found them
-) Remove ";" in the imports statements
-) Remove () at the end of method call like size that does not have size effect.
 2013-11-15 10:32:20 -08:00
Raymond Liu a60620b76a Merge branch 'master' into scala-2.10 2013-11-14 12:44:19 +08:00
Kay Ousterhout 46f9c6b858 Fixed naming issues and added back ability to specify max task failures. 2013-11-13 17:12:14 -08:00
Kay Ousterhout 150615a31e Merge remote-tracking branch 'upstream/master' into consolidate_schedulers 
Conflicts:
	core/src/main/scala/org/apache/spark/scheduler/ClusterScheduler.scala
 2013-11-13 14:38:44 -08:00
Raymond Liu 0f2e3c6e31 Merge branch 'master' into scala-2.10 2013-11-13 16:55:11 +08:00
tgravescs a35472e1dd Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs. 2013-11-04 16:16:28 -06:00
Kay Ousterhout 5e91495f5c Deduplicate Local and Cluster schedulers. 
The code in LocalScheduler/LocalTaskSetManager was nearly identical
to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy
made making updating the schedulers unnecessarily painful and error-
prone. This commit combines the two into a single TaskScheduler/
TaskSetManager.
 2013-10-30 18:48:34 -07:00
tgravescs b6571541a6 Fix the Worker to use CoarseGrainedExecutorBackend and modify classpath to be explicit 
about inclusion of spark.jar and app.jar
 2013-10-21 14:05:15 -05:00
tgravescs cc7df2b3cc Fix yarn build 2013-10-16 10:09:16 -05:00
Matei Zaharia 8f11c36fe1 Merge remote-tracking branch 'tgravescs/sparkYarnDistCache' 
Closes #11

Conflicts:
	docs/running-on-yarn.md
	yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
 2013-10-10 19:34:33 -07:00
Prashant Sharma 7be75682b9 Merge branch 'master' into wip-merge-master 
Conflicts:
	bagel/pom.xml
	core/pom.xml
	core/src/test/scala/org/apache/spark/ui/UISuite.scala
	examples/pom.xml
	mllib/pom.xml
	pom.xml
	project/SparkBuild.scala
	repl/pom.xml
	streaming/pom.xml
	tools/pom.xml

In scala 2.10, a shorter representation is used for naming artifacts
 so changed to shorter scala version for artifacts and made it a property in pom.
 2013-10-08 11:29:40 +05:30
Patrick Wendell aa9fb84994 Merging build changes in from 0.8 2013-10-05 22:07:00 -07:00
Prashant Sharma c810ee0690 Merge branch 'master' into scala-2.10 
Conflicts:
	core/src/test/scala/org/apache/spark/DistributedSuite.scala
	project/SparkBuild.scala
 2013-10-05 15:52:57 +05:30
tgravescs 0fff4ee852 Adding in the --addJars option to make SparkContext.addJar work on yarn and cleanup 
the classpaths
 2013-10-03 11:52:16 -05:00
tgravescs c021b8c202 Add default value to usage statement 2013-10-03 08:07:19 -05:00
tgravescs bc3b20abdc Allow users to set the application name for Spark on Yarn 2013-10-02 12:54:17 -05:00
Prashant Sharma 5829692885 Merge branch 'master' into scala-2.10 
Conflicts:
	core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala
	docs/_config.yml
	project/SparkBuild.scala
	repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
 2013-10-01 11:57:24 +05:30
Prashant Sharma 7ff4c2d399 fixed maven build for scala 2.10 2013-09-26 10:48:24 +05:30
Patrick Wendell 6079721fa1 Update build version in master 2013-09-24 11:41:51 -07:00
Y.CORP.YAHOO.COM\tgraves a314b30733 Fix spacing so that the java.io.tmpdir doesn't run on with SPARK_JAVA_OPTS 2013-09-23 14:48:17 -05:00
Y.CORP.YAHOO.COM\tgraves 9d4246863a Support distributed cache files and archives on spark on yarn and attempt to cleanup the staging directory on exit 2013-09-23 09:09:59 -05:00
Prashant Sharma 276c37a51c Akka 2.2 migration 2013-09-22 08:20:12 +05:30
Patrick Wendell e9eba8c3ce Use different Hadoop version for YARN artifacts. 
This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.
 2013-09-13 15:34:57 -07:00
Jey Kottalam 70661246fd Fix YARN assembly generation under Maven 2013-09-06 11:31:16 -07:00
Y.CORP.YAHOO.COM\tgraves c8cc276110 Review comment changes and update to org.apache packaging 2013-09-03 10:50:21 -05:00
Y.CORP.YAHOO.COM\tgraves 547fc4a412 Merge remote-tracking branch 'mesos/master' into yarnUILink 
Conflicts:
	core/src/main/scala/org/apache/spark/ui/UIUtils.scala
	core/src/main/scala/org/apache/spark/ui/jobs/PoolTable.scala
	core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
	docs/running-on-yarn.md
 2013-09-03 08:36:59 -05:00
Matei Zaharia 12b2f1f9c9 Add missing license headers found with RAT 2013-09-02 12:23:03 -07:00
Matei Zaharia 5701eb92c7 Fix some URLs 2013-09-01 14:13:16 -07:00
Matei Zaharia 46eecd110a Initial work to rename package to org.apache.spark 2013-09-01 14:13:13 -07:00
Y.CORP.YAHOO.COM\tgraves bac46266a9 Link the Spark UI to the Yarn UI 2013-08-30 15:55:32 -05:00
Matei Zaharia baa84e7e4c Merge pull request #865 from tgravescs/fixtmpdir 
Spark on Yarn should use yarn approved directories for spark.local.dir and tmp
 2013-08-28 12:44:46 -07:00
Y.CORP.YAHOO.COM\tgraves 3f206bf0b5 Updated based on review comments. 2013-08-27 14:34:27 -05:00
Y.CORP.YAHOO.COM\tgraves 63dc635de6 fix typos 2013-08-26 17:06:20 -05:00
Y.CORP.YAHOO.COM\tgraves c9464c74a1 Add ability for user to specify environment variables 2013-08-26 16:44:27 -05:00
Y.CORP.YAHOO.COM\tgraves dfb4c697bc Throw exception if the yarn local dirs isn't set 2013-08-26 13:57:01 -05:00
Y.CORP.YAHOO.COM\tgraves c0b4095ee8 Change to use Yarn appropriate directories rather then /tmp or the user specified spark.local.dir 2013-08-26 12:48:37 -05:00
Matei Zaharia aa2b89d98d Merge remote-tracking branch 'jey/hadoop-agnostic' 
Conflicts:
	core/src/main/scala/spark/PairRDDFunctions.scala
 2013-08-20 10:14:15 -07:00
Jey Kottalam 44000b10ff Make YARN POM file valid 2013-08-18 16:23:22 -07:00
Jey Kottalam ad580b94d5 Maven build now also works with YARN 2013-08-16 13:50:12 -07:00
Jey Kottalam 8bb0bd11ce YARN ApplicationMaster shouldn't wait forever 2013-08-15 16:50:37 -07:00
Jey Kottalam e2d7656ca3 re-enable YARN support 2013-08-15 16:50:37 -07:00
Jey Kottalam bd0bab47c9 SparkEnv isn't available this early, and not needed anyway 2013-08-15 16:50:37 -07:00
Jey Kottalam 5d0785b4e5 remove hadoop-yarn's org/apache/... 2013-08-15 16:50:37 -07:00
Jey Kottalam b877e20a33 move yarn to its own directory 2013-08-15 16:50:36 -07:00