Thomas Graves
ad35c1a5f2
Fix handling of empty SPARK_EXAMPLES_JAR
2014-01-04 11:42:17 -06:00
Tathagata Das
3d4474330d
Removed the exponential backoff for testing.
2014-01-04 08:39:00 -08:00
Andrew Or
2db7884f6f
Address Mark's comments
2014-01-04 01:20:09 -08:00
Andrew Or
4296d96c82
Assign spill threshold as a fraction of maximum memory
...
Further, divide this threshold by the number of tasks running concurrently.
Note that this does not guard against the following scenario: a new task
quickly fills up its share of the memory before old tasks finish spilling
their contents, in which case the total memory used by such maps may exceed
what was specified. Currently, spark.shuffle.safetyFraction mitigates the
effect of this.
2014-01-04 00:00:57 -08:00
Patrick Wendell
10fe23bc34
Merge pull request #329 from pwendell/remove-binaries
...
SPARK-1002: Remove Binaries from Spark Source
This adds a few changes on top of the work by @scrapcodes.
2014-01-03 23:50:14 -08:00
Patrick Wendell
604fad9c39
Merge remote-tracking branch 'apache-github/master' into remove-binaries
...
Conflicts:
core/src/test/scala/org/apache/spark/DriverSuite.scala
docs/python-programming-guide.md
2014-01-03 21:29:33 -08:00
Patrick Wendell
9e6f3bdcda
Changes on top of Prashant's patch.
...
Closes #316
2014-01-03 18:30:17 -08:00
Andrew Or
333d58df86
Remove unnecessary ClassTag's
2014-01-03 17:55:26 -08:00
Hossein Falaki
8b5be06752
Added table of contents and minor fixes
2014-01-03 16:38:33 -08:00
Patrick Wendell
c4d6145f7f
Merge pull request #325 from witgo/master
...
Modify spark on yarn to create SparkConf process
2014-01-03 16:30:53 -08:00
Andrew Or
838b0e7d15
Refactor using SparkConf
2014-01-03 16:13:40 -08:00
Hossein Falaki
dfe57fa84c
Removed unnecessary blank line
2014-01-03 15:40:53 -08:00
Hossein Falaki
2c1cba851c
Added unit tests for bulk prediction in MatrixFactorizationModel
2014-01-03 15:35:20 -08:00
Hossein Falaki
67f937ec22
Added a method to enable bulk prediction
2014-01-03 15:34:16 -08:00
Patrick Wendell
4ae101ff38
Merge pull request #317 from ScrapCodes/spark-915-segregate-scripts
...
Spark-915 segregate scripts
2014-01-03 11:24:35 -08:00
Luca Rosellini
0b6db8c186
Added ‘-i’ command line option to spark REPL.
...
We had to create a new implementation of both scala.tools.nsc.CompilerCommand and scala.tools.nsc.Settings, because using scala.tools.nsc.GenericRunnerSettings would bring in other options (-howtorun, -save and -execute) which don’t make sense in Spark.
Any new Spark specific command line option could now be added to org.apache.spark.repl.SparkRunnerSettings class.
Since the behavior of loading a script from the command line should be the same as loading it using the “:load” command inside the shell, the script should be loaded when the SparkContext is available, that’s why we had to move the call to ‘loadfiles(settings)’ _after_ the call to postInitialization(). This still doesn’t work if ‘isAsync = true’.
2014-01-03 12:57:06 +01:00
Prashant Sharma
9ae382c363
sbin/compute-classpath* bin/compute-classpath*
2014-01-03 15:12:29 +05:30
Prashant Sharma
74ba97fcf7
sbin/spark-class* -> bin/spark-class*
2014-01-03 15:08:01 +05:30
Prashant Sharma
bc311bb826
Restored the previously removed test
2014-01-03 14:52:37 +05:30
Prashant Sharma
94f2fffa23
fixed review comments
2014-01-03 14:43:37 +05:30
Luca Rosellini
87248bddac
Merge pull request #1 from apache/master
...
Merge latest Spark changes
2014-01-03 00:45:31 -08:00
liguoqiang
8ddbd531a4
merge upstream/master
2014-01-03 16:06:34 +08:00
liguoqiang
b27b75f1c5
Modify spark on yarn to create SparkConf process
2014-01-03 15:34:24 +08:00
Patrick Wendell
30b9db0abe
Merge pull request #285 from colorant/yarn-refactor
...
Yarn refactor
2014-01-02 23:15:55 -08:00
liguoqiang
010e72c079
Modify spark on yarn to create SparkConf process
2014-01-03 15:01:38 +08:00
Prashant Sharma
b4bb80002b
Merge branch 'master' into spark-1002-remove-jars
2014-01-03 12:12:04 +05:30
Raymond Liu
f442afc22e
fix docs for yarn
2014-01-03 14:14:35 +08:00
Andrew Or
df413e996f
Merge remote-tracking branch 'spark/master'
...
Conflicts:
core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala
2014-01-02 20:51:23 -08:00
Raymond Liu
18b3633e54
minor fix for loginfo
2014-01-03 12:14:38 +08:00
Raymond Liu
c59029402d
move duplicate pom config into parent pom
2014-01-03 12:14:38 +08:00
Raymond Liu
ebdfa6bb97
Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2
2014-01-03 12:14:38 +08:00
Raymond Liu
a47ebf7228
Add yarn/common/src/test dir in building script
2014-01-03 12:14:38 +08:00
Raymond Liu
ddc5054b35
Fix yarn/README.md
2014-01-03 12:14:38 +08:00
Raymond Liu
79b6b4ddc2
Clean up unused files for yarn
2014-01-03 12:14:38 +08:00
Raymond Liu
7c96faee74
Fix pom for build yarn/2.x with yarn/common into one jar
2014-01-03 12:14:38 +08:00
Raymond Liu
d1a6f7aabc
Use unmanaged source dir to include common yarn code
2014-01-03 12:14:37 +08:00
Raymond Liu
c5422e02b8
merge yarn/scheduler yarn/common code into one directory
2014-01-03 12:14:37 +08:00
Raymond Liu
ad60710010
Need to send dummy hello message to actually estabilish akka connection.
2014-01-03 12:14:37 +08:00
Raymond Liu
dd6d347f4f
A few clean up for yarn 2.0 code
2014-01-03 12:14:37 +08:00
Raymond Liu
7815a3ace9
Update maven build documentation
2014-01-03 12:12:38 +08:00
Raymond Liu
be343d2a56
Fix yarn/README.md and update docs/running-on-yarn.md
2014-01-03 12:12:38 +08:00
Raymond Liu
67cd752e74
Add README for yarn modules
2014-01-03 12:12:38 +08:00
Raymond Liu
e867e31145
some code clean up for Yarn 2.2
2014-01-03 12:12:37 +08:00
Raymond Liu
8818661721
Fix pom file for scala binary version
2014-01-03 12:12:37 +08:00
Raymond Liu
96e25e567c
Fix yarn/assemble pom file
2014-01-03 12:12:37 +08:00
Raymond Liu
aec96dd108
Change profile name new-yarn to hadoop2.2-yarn
2014-01-03 12:12:37 +08:00
Raymond Liu
d1528c7c8c
Fix pom for yarn code reorgnaize commit
2014-01-03 12:12:37 +08:00
Raymond Liu
3dc379ce5a
Reorganize yarn related codes into sub projects to remove duplicate files.
2014-01-03 12:12:37 +08:00
Tathagata Das
a1b8dd53e3
Added StreamingContext.getOrCreate to for automatic recovery, and added RecoverableNetworkWordCount example to use it.
2014-01-02 19:07:22 -08:00
Patrick Wendell
498a5f0a1c
Merge pull request #323 from tgravescs/sparkconf_yarn_fix
...
fix spark on yarn after the sparkConf changes
This fixes it so that spark on yarn now compiles and works after the sparkConf changes.
There are also other issues I discovered along the way that are broken:
- mvn builds for yarn don't assemble correctly
- unset SPARK_EXAMPLES_JAR isn't handled properly anymore
- I'm pretty sure spark.conf doesn't actually work as its not distributed with yarn
those things can be fixed in separate pr unless others disagree.
2014-01-02 19:06:40 -08:00