ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
6168 commits 6 branches 181 tags 397 MiB
Commit graph

105 commits

Author SHA1 Message Date
Raymond Liu 4c22c55ad6 Address comments to fix code formats 2014-01-14 10:41:42 +08:00
Raymond Liu 161ab93989 Yarn workerRunnable refactor 2014-01-14 10:36:00 +08:00
Raymond Liu 79a5ba3497 Yarn Client refactor 2014-01-14 10:33:48 +08:00
Henry Saputra 5a8abfb70e Address code review concerns and comments. 2014-01-12 19:15:09 -08:00
Henry Saputra 91a563608e Merge branch 'master' into remove_simpleredundantreturn_scala 2014-01-12 10:34:13 -08:00
Henry Saputra 93a65e5fde Remove simple redundant return statement for Scala methods/functions: 
-) Only change simple return statements at the end of method
-) Ignore the complex if-else check
-) Ignore the ones inside synchronized
 2014-01-12 10:30:04 -08:00
Thomas Graves 9bdfbc0492 Update based on review comments 2014-01-09 15:45:33 -06:00
Thomas Graves 5fccd7051d add application type 2014-01-09 10:32:35 -06:00
Thomas Graves c617083e47 yarn-client addJar fix and misc other 2014-01-09 10:24:35 -06:00
Raymond Liu 67af803136 Export --file for YarnClient mode to support sending extra files to worker on yarn cluster 2014-01-07 10:24:11 +08:00
Raymond Liu da4694a0d8 Minor typo fix for yarn client 2014-01-07 10:24:10 +08:00
liguoqiang 8ddbd531a4 merge upstream/master 2014-01-03 16:06:34 +08:00
liguoqiang b27b75f1c5 Modify spark on yarn to create SparkConf process 2014-01-03 15:34:24 +08:00
liguoqiang 010e72c079 Modify spark on yarn to create SparkConf process 2014-01-03 15:01:38 +08:00
Raymond Liu f442afc22e fix docs for yarn 2014-01-03 14:14:35 +08:00
Raymond Liu 18b3633e54 minor fix for loginfo 2014-01-03 12:14:38 +08:00
Raymond Liu c59029402d move duplicate pom config into parent pom 2014-01-03 12:14:38 +08:00
Raymond Liu ebdfa6bb97 Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2 2014-01-03 12:14:38 +08:00
Raymond Liu ddc5054b35 Fix yarn/README.md 2014-01-03 12:14:38 +08:00
Raymond Liu 79b6b4ddc2 Clean up unused files for yarn 2014-01-03 12:14:38 +08:00
Raymond Liu 7c96faee74 Fix pom for build yarn/2.x with yarn/common into one jar 2014-01-03 12:14:38 +08:00
Raymond Liu c5422e02b8 merge yarn/scheduler yarn/common code into one directory 2014-01-03 12:14:37 +08:00
Raymond Liu ad60710010 Need to send dummy hello message to actually estabilish akka connection. 2014-01-03 12:14:37 +08:00
Raymond Liu dd6d347f4f A few clean up for yarn 2.0 code 2014-01-03 12:14:37 +08:00
Raymond Liu be343d2a56 Fix yarn/README.md and update docs/running-on-yarn.md 2014-01-03 12:12:38 +08:00
Raymond Liu 67cd752e74 Add README for yarn modules 2014-01-03 12:12:38 +08:00
Raymond Liu e867e31145 some code clean up for Yarn 2.2 2014-01-03 12:12:37 +08:00
Raymond Liu 8818661721 Fix pom file for scala binary version 2014-01-03 12:12:37 +08:00
Raymond Liu 96e25e567c Fix yarn/assemble pom file 2014-01-03 12:12:37 +08:00
Raymond Liu aec96dd108 Change profile name new-yarn to hadoop2.2-yarn 2014-01-03 12:12:37 +08:00
Raymond Liu d1528c7c8c Fix pom for yarn code reorgnaize commit 2014-01-03 12:12:37 +08:00
Raymond Liu 3dc379ce5a Reorganize yarn related codes into sub projects to remove duplicate files. 2014-01-03 12:12:37 +08:00
Thomas Graves fced7885cb fix yarn-client 2014-01-02 17:11:16 -06:00
Thomas Graves c6de982be6 Fix yarn build after sparkConf changes 2014-01-02 16:50:35 -06:00
Matei Zaharia b4ceed40d6 Merge remote-tracking branch 'origin/master' into conf2 
Conflicts:
	core/src/main/scala/org/apache/spark/SparkContext.scala
	core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
	core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
	core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala
	core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala
	core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala
	core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
	new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
	streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
	streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
	streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
	streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala
	streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala
	streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
	streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
	streaming/src/test/scala/org/apache/spark/streaming/WindowOperationsSuite.scala
 2013-12-29 15:08:08 -05:00
Matei Zaharia 642029e7f4 Various fixes to configuration code 
- Got rid of global SparkContext.globalConf
- Pass SparkConf to serializers and compression codecs
- Made SparkConf public instead of private[spark]
- Improved API of SparkContext and SparkConf
- Switched executor environment vars to be passed through SparkConf
- Fixed some places that were still using system properties
- Fixed some tests, though others are still failing

This still fails several tests in core, repl and streaming, likely due
to properties not being set or cleared correctly (some of the tests run
fine in isolation).
 2013-12-28 17:13:15 -05:00
liguoqiang 2bd76f693d Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn 2013-12-26 11:10:35 +08:00
liguoqiang 14fcef72db Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn 2013-12-26 11:05:07 +08:00
Prashant Sharma 2573add94c spark-544, introducing SparkConf and related configuration overhaul. 2013-12-25 00:09:36 +05:30
Kay Ousterhout c06945cfe0 Merge remote branch 'upstream/master' into consolidate_schedulers 
Conflicts:
	core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
	core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
	core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
 2013-12-20 14:39:30 -08:00
Patrick Wendell a51f3404ad Merge pull request #265 from markhamstra/scala.binary.version 
DRY out the POMs with scala.binary.version

...instead of hard-coding 2.10 repeatedly.

As long as it's not a `<project>`-level `<artifactId>`, I think that we are okay parameterizing these.
 2013-12-15 22:02:30 -08:00
Reynold Xin c55e698559 Merge pull request #257 from tgravescs/sparkYarnFixName 
Fix the --name option for Spark on Yarn

Looks like the --name option accidentally got broken in one of the merges.  The Client hangs if the --name option is used right now.
 2013-12-15 12:49:02 -08:00
Mark Hamstra 09ed7ddfa0 Use scala.binary.version in POMs 2013-12-15 12:39:58 -08:00
Thomas Graves 842eb55fb5 Fix the --name option for Spark on Yarn 2013-12-12 11:11:09 -06:00
Prashant Sharma 603af51bb5 Merge branch 'master' into akka-bug-fix 
Conflicts:
	core/pom.xml
	core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
	pom.xml
	project/SparkBuild.scala
	streaming/pom.xml
	yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
 2013-12-11 10:21:53 +05:30
Prashant Sharma 17db6a9041 Style fixes and addressed review comments at #221 2013-12-10 11:47:16 +05:30
Prashant Sharma c1201f47e0 fixed yarn build 2013-12-09 13:00:50 +05:30
Aaron Davidson 40f63eb034 Merge master into 127 2013-12-08 11:16:52 -08:00
Prashant Sharma 7ad6921ae0 Incorporated Patrick's feedback comment on #211 and made maven build/dep-resolution atleast a bit faster. 2013-12-07 12:45:57 +05:30
Prashant Sharma 4e70480038 A left over akka -> akka.tcp changes 2013-12-06 12:29:53 +05:30