ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3012 commits 6 branches 181 tags 397 MiB
Commit graph

3012 commits

This branch
This branch
All branches
Author SHA1 Message Date
Reynold Xin 0eab7a78b9 Fixed a couple typos and formating problems in the YARN documentation. 2013-05-17 18:05:46 -07:00
Reynold Xin 7760d78b3a Merge branch 'master' of https://github.com/mridulm/spark 2013-05-17 17:58:36 -07:00
Reynold Xin 6729c2ead8 Merge branch 'master' of github.com:mesos/spark 2013-05-17 17:58:06 -07:00
Andrew xia 3d4672eaa9 Merge branch 'master' into xiajunluan 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/scheduler/cluster/ClusterScheduler.scala
	core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
 2013-05-18 07:28:03 +08:00
Andrew xia d19753b9c7 expose TaskSetManager type to resourceOffer function in ClusterScheduler 2013-05-18 06:45:19 +08:00
Mridul Muralidharan da2642bead Fix example jar name 2013-05-17 06:58:46 +05:30
Reynold Xin 6c27c38451 Merge pull request #615 from rxin/build-fix 
Maven build fix & two other small changes
 2013-05-16 17:33:56 -07:00
Reynold Xin dc146406ae Updated Scala version in docs generation ruby script. 2013-05-16 17:07:14 -07:00
Reynold Xin 3b3300383a Updated Scala version in docs generation ruby script. 2013-05-16 16:51:28 -07:00
Reynold Xin 43644a293f Only check for repl classes if the user is running the repl. Otherwise, 
check for core classes in run. This fixed the problem that core tests
depend on whether repl module is compiled or not.
 2013-05-16 14:31:38 -07:00
Reynold Xin 61cf176238 Added dependency on netty-all in Maven. 2013-05-16 14:31:26 -07:00
Reynold Xin f742435f18 Removed the duplicated netty dependency in SBT build file. 2013-05-16 14:31:03 -07:00
Andrew xia c6e2770bfe Fix ClusterScheduler bug to avoid allocating tasks to same slave 2013-05-17 05:10:38 +08:00
Mridul Muralidharan f0881f8d48 Hope this does not turn into a bike shed change 2013-05-17 01:58:50 +05:30
Mridul Muralidharan f16c781709 Fix documentation to use yarn-standalone as master 2013-05-16 17:50:22 +05:30
Mridul Muralidharan feddd2530d Filter out nulls - prevent NPE 2013-05-16 17:49:14 +05:30
Mridul Muralidharan 87540a7b38 Fix running on yarn documentation 2013-05-16 15:27:58 +05:30
Josh Rosen b8e46b6074 Abort job if result exceeds Akka frame size; add test. 2013-05-16 01:57:57 -07:00
Matei Zaharia 2f576aba8f Merge pull request #602 from rxin/shufflemerge 
Manual merge & cleanup of Shane's Shuffle Performance Optimization
 2013-05-15 18:06:24 -07:00
Reynold Xin 48c6f46a6f Merge pull request #612 from ash211/patch-4 
Docs: Mention spark shell's default for MASTER
 2013-05-15 10:47:19 -07:00
Andrew Ash afcad7b3aa Docs: Mention spark shell's default for MASTER 2013-05-15 14:45:14 -03:00
Reynold Xin 203d7b7c14 Merge pull request #593 from squito/driver_ui_link 
Master UI has link to Application UI
 2013-05-15 00:47:20 -07:00
Reynold Xin f3491cb89b Merge branch 'master' of github.com:mesos/spark into shufflemerge 
Conflicts:
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/test/scala/spark/DistributedSuite.scala
	project/SparkBuild.scala
 2013-05-15 00:31:52 -07:00
Reynold Xin f9d40a5848 Added a comment in JdbcRDD for example usage. 2013-05-14 23:29:57 -07:00
Reynold Xin 404f9ff617 Added derby dependency to Maven pom files for the JDBC Java test. 2013-05-14 23:28:34 -07:00
Reynold Xin 81ad2fa331 Merge branch 'jdbc' of github.com:koeninger/spark 
Conflicts:
	project/SparkBuild.scala
 2013-05-14 23:12:00 -07:00
Imran Rashid 38d4b97c6d use threads classloader when deserializing task results; classnotfoundexception includes classloader 2013-05-14 22:32:14 -07:00
Imran Rashid d7d1da79d3 when akka starts, use akkas default classloader (current thread) 2013-05-14 22:32:09 -07:00
Cody Koeninger b16c4896f6 add test for JdbcRDD using embedded derby, per rxin suggestion 2013-05-14 23:44:04 -05:00
Matei Zaharia 016ac86830 Merge pull request #601 from rxin/emptyrdd-master 
EmptyRDD (master branch 0.8)
 2013-05-13 21:45:36 -07:00
Matei Zaharia b9aef263df Merge branch 'master' of github.com:mesos/spark 2013-05-13 17:40:21 -07:00
Matei Zaharia 4b354e0a08 Merge pull request #589 from mridulm/master 
Add support for instance local scheduling
 2013-05-13 17:39:19 -07:00
Matei Zaharia 5dbc9b2006 Merge pull request #608 from pwendell/SPARK-738 
SPARK-738: Spark should detect and wrap nonserializable exceptions
 2013-05-12 11:03:10 -07:00
Patrick Wendell 7f0833647b Capturing class name 2013-05-12 07:54:03 -07:00
Patrick Wendell 72b9c4cb6e Small fix 2013-05-11 23:53:50 -07:00
Patrick Wendell 1c15b85051 Removing import 2013-05-11 23:52:53 -07:00
Patrick Wendell 059ab88754 Changing technique to use same code path in all cases 2013-05-11 23:50:54 -07:00
Cody Koeninger 3da2305ed0 code cleanup per rxin comments 2013-05-11 23:59:07 -05:00
Josh Rosen 440719109e Throw exception if task result exceeds Akka frame size. 
This partially addresses SPARK-747.
 2013-05-11 19:17:13 -07:00
Patrick Wendell a5c28bb888 Removing unnecessary map 2013-05-11 14:20:39 -07:00
Patrick Wendell 0345954530 SPARK-738: Spark should detect and squash nonserializable exceptions 2013-05-11 14:17:09 -07:00
Mridul Muralidharan ee37612bc9 1) Add support for HADOOP_CONF_DIR (and/or YARN_CONF_DIR - use either) : which is used to specify the client side configuration directory : which needs to be part of the CLASSPATH. 
2) Move from var+=".." to var="$var.." : the former does not work on older bash shells unfortunately.
 2013-05-11 11:12:22 +05:30
seanm f25282def5 fixing kafkaStream Java API and adding test 2013-05-10 17:34:28 -06:00
seanm 3632980b1b fixing indentation 2013-05-10 15:54:26 -06:00
Reynold Xin 63e1999f60 Merge pull request #606 from markhamstra/foreachPartition_fix 
Actually use the cleaned closure in foreachPartition
 2013-05-10 13:54:03 -07:00
Mark Hamstra 6e6b3e0d7e Actually use the cleaned closure in foreachPartition 2013-05-10 13:02:34 -07:00
seanm b95c1bdbba count() now uses a transform instead of ConstantInputDStream 2013-05-10 12:47:24 -06:00
seanm d761e7359d adding kafkaStream API tests 2013-05-10 12:05:10 -06:00
Ethan Jewett ee6f6aa6cd Add hBase example 2013-05-09 18:33:38 -05:00
Reynold Xin 012c9e5ab0 Revert "Merge pull request #596 from esjewett/master" because the 
dependency on hbase introduces netty-3.2.2 which conflicts with
netty-3.5.3 already in Spark. This caused multiple test failures.

This reverts commit 0f1b7a06e1, reversing
changes made to aacca1b8a8.
 2013-05-09 14:20:01 -07:00