ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5440 commits 6 branches 181 tags 397 MiB
Commit graph

5440 commits

This branch
This branch
All branches
Author SHA1 Message Date
Holden Karau b4a1ffc6c2 Switch from sbt to ./sbt in the README file 2014-01-04 20:17:30 -08:00
Holden Karau 97123be1d7 Pass commands down to system sbt as well 2014-01-04 20:16:56 -08:00
Holden Karau 9e9a913c2f Add a script to download sbt if not present on the system 2014-01-04 20:08:35 -08:00
Reynold Xin d43ad3ef2c Merge pull request #292 from soulmachine/naive-bayes 
standard Naive Bayes classifier

Has implemented the standard Naive Bayes classifier. This is an updated version of #288, which is closed because of misoperations.
 2014-01-04 16:29:30 -08:00
Hossein Falaki 8d0c2f7399 Added python binding for bulk recommendation 2014-01-04 16:23:17 -08:00
Andrew Or 4de9c9554c Use AtomicInteger for numRunningTasks 2014-01-04 11:16:30 -08:00
Thomas Graves ad35c1a5f2 Fix handling of empty SPARK_EXAMPLES_JAR 2014-01-04 11:42:17 -06:00
Tathagata Das 3d4474330d Removed the exponential backoff for testing. 2014-01-04 08:39:00 -08:00
Andrew Or 2db7884f6f Address Mark's comments 2014-01-04 01:20:09 -08:00
Andrew Or 4296d96c82 Assign spill threshold as a fraction of maximum memory 
Further, divide this threshold by the number of tasks running concurrently.

Note that this does not guard against the following scenario: a new task
quickly fills up its share of the memory before old tasks finish spilling
their contents, in which case the total memory used by such maps may exceed
what was specified. Currently, spark.shuffle.safetyFraction mitigates the
effect of this.
 2014-01-04 00:00:57 -08:00
Patrick Wendell 10fe23bc34 Merge pull request #329 from pwendell/remove-binaries 
SPARK-1002: Remove Binaries from Spark Source

This adds a few changes on top of the work by @scrapcodes.
 2014-01-03 23:50:14 -08:00
Patrick Wendell 604fad9c39 Merge remote-tracking branch 'apache-github/master' into remove-binaries 
Conflicts:
	core/src/test/scala/org/apache/spark/DriverSuite.scala
	docs/python-programming-guide.md
 2014-01-03 21:29:33 -08:00
Patrick Wendell 9e6f3bdcda Changes on top of Prashant's patch. 
Closes #316
 2014-01-03 18:30:17 -08:00
Andrew Or 333d58df86 Remove unnecessary ClassTag's 2014-01-03 17:55:26 -08:00
Hossein Falaki 8b5be06752 Added table of contents and minor fixes 2014-01-03 16:38:33 -08:00
Patrick Wendell c4d6145f7f Merge pull request #325 from witgo/master 
Modify spark on yarn to create SparkConf process
 2014-01-03 16:30:53 -08:00
Andrew Or 838b0e7d15 Refactor using SparkConf 2014-01-03 16:13:40 -08:00
Hossein Falaki dfe57fa84c Removed unnecessary blank line 2014-01-03 15:40:53 -08:00
Hossein Falaki 2c1cba851c Added unit tests for bulk prediction in MatrixFactorizationModel 2014-01-03 15:35:20 -08:00
Hossein Falaki 67f937ec22 Added a method to enable bulk prediction 2014-01-03 15:34:16 -08:00
Patrick Wendell 4ae101ff38 Merge pull request #317 from ScrapCodes/spark-915-segregate-scripts 
Spark-915 segregate scripts
 2014-01-03 11:24:35 -08:00
Luca Rosellini 0b6db8c186 Added ‘-i’ command line option to spark REPL. 
We had to create a new implementation of both scala.tools.nsc.CompilerCommand and scala.tools.nsc.Settings, because using scala.tools.nsc.GenericRunnerSettings would bring in other options (-howtorun, -save and -execute) which don’t make sense in Spark.
Any new Spark specific command line option could now be added to org.apache.spark.repl.SparkRunnerSettings class.

Since the behavior of loading a script from the command line should be the same as loading it using the “:load” command inside the shell, the script should be loaded when the SparkContext is available, that’s why we had to move the call to ‘loadfiles(settings)’ _after_ the call to postInitialization(). This still doesn’t work if ‘isAsync = true’.
 2014-01-03 12:57:06 +01:00
Prashant Sharma 9ae382c363 sbin/compute-classpath* bin/compute-classpath* 2014-01-03 15:12:29 +05:30
Prashant Sharma 74ba97fcf7 sbin/spark-class* -> bin/spark-class* 2014-01-03 15:08:01 +05:30
Prashant Sharma bc311bb826 Restored the previously removed test 2014-01-03 14:52:37 +05:30
Prashant Sharma 94f2fffa23 fixed review comments 2014-01-03 14:43:37 +05:30
Luca Rosellini 87248bddac Merge pull request #1 from apache/master 
Merge latest Spark changes
 2014-01-03 00:45:31 -08:00
liguoqiang 8ddbd531a4 merge upstream/master 2014-01-03 16:06:34 +08:00
liguoqiang b27b75f1c5 Modify spark on yarn to create SparkConf process 2014-01-03 15:34:24 +08:00
Patrick Wendell 30b9db0abe Merge pull request #285 from colorant/yarn-refactor 
Yarn refactor
 2014-01-02 23:15:55 -08:00
liguoqiang 010e72c079 Modify spark on yarn to create SparkConf process 2014-01-03 15:01:38 +08:00
Prashant Sharma b4bb80002b Merge branch 'master' into spark-1002-remove-jars 2014-01-03 12:12:04 +05:30
Raymond Liu f442afc22e fix docs for yarn 2014-01-03 14:14:35 +08:00
Andrew Or df413e996f Merge remote-tracking branch 'spark/master' 
Conflicts:
	core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala
 2014-01-02 20:51:23 -08:00
Raymond Liu 18b3633e54 minor fix for loginfo 2014-01-03 12:14:38 +08:00
Raymond Liu c59029402d move duplicate pom config into parent pom 2014-01-03 12:14:38 +08:00
Raymond Liu ebdfa6bb97 Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2 2014-01-03 12:14:38 +08:00
Raymond Liu a47ebf7228 Add yarn/common/src/test dir in building script 2014-01-03 12:14:38 +08:00
Raymond Liu ddc5054b35 Fix yarn/README.md 2014-01-03 12:14:38 +08:00
Raymond Liu 79b6b4ddc2 Clean up unused files for yarn 2014-01-03 12:14:38 +08:00
Raymond Liu 7c96faee74 Fix pom for build yarn/2.x with yarn/common into one jar 2014-01-03 12:14:38 +08:00
Raymond Liu d1a6f7aabc Use unmanaged source dir to include common yarn code 2014-01-03 12:14:37 +08:00
Raymond Liu c5422e02b8 merge yarn/scheduler yarn/common code into one directory 2014-01-03 12:14:37 +08:00
Raymond Liu ad60710010 Need to send dummy hello message to actually estabilish akka connection. 2014-01-03 12:14:37 +08:00
Raymond Liu dd6d347f4f A few clean up for yarn 2.0 code 2014-01-03 12:14:37 +08:00
Raymond Liu 7815a3ace9 Update maven build documentation 2014-01-03 12:12:38 +08:00
Raymond Liu be343d2a56 Fix yarn/README.md and update docs/running-on-yarn.md 2014-01-03 12:12:38 +08:00
Raymond Liu 67cd752e74 Add README for yarn modules 2014-01-03 12:12:38 +08:00
Raymond Liu e867e31145 some code clean up for Yarn 2.2 2014-01-03 12:12:37 +08:00
Raymond Liu 8818661721 Fix pom file for scala binary version 2014-01-03 12:12:37 +08:00