ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4062 commits 6 branches 181 tags 397 MiB
Commit graph

4062 commits

This branch
This branch
All branches
Author SHA1 Message Date
jerryshao 5850f599dd Refactor FairSchedulableBuilder: 
1. Configuration can be read from classpath if not set explicitly.
2. Add missing close handler.
 2013-09-22 14:20:55 +08:00
Reynold Xin a2ea069a5f Merge pull request #937 from jerryshao/localProperties-fix 
Fix PR926 local properties issues in Spark Streaming like scenarios
 2013-09-21 23:04:42 -07:00
Reynold Xin f06f2da2cb Merge pull request #941 from ilikerps/master 
Add "org.apache." prefix to packages in spark-class
 2013-09-21 22:43:34 -07:00
Reynold Xin 7bb12a2af3 Merge pull request #940 from ankurdave/clear-port-properties-after-tests 
After unit tests, clear port properties unconditionally
 2013-09-21 22:42:46 -07:00
jerryshao aa0c29f747 Add barrier for local properties unit test and fix some styles 2013-09-22 09:53:11 +08:00
Aaron Davidson 8933f9e98e Add "org.apache." prefix to packages in spark-class 
Lacking this, the if/case statements never trigger on Spark 0.8.0+.
 2013-09-20 19:27:08 -07:00
Ankur Dave 026dba6aba After unit tests, clear port properties unconditionally 
In MapOutputTrackerSuite, the "remote fetch" test sets spark.driver.port
and spark.hostPort, assuming that they will be cleared by
LocalSparkContext. However, the test never sets sc, so it remains null,
causing LocalSparkContext to skip clearing these properties. Subsequent
tests therefore fail with java.net.BindException: "Address already in
use".

This commit makes LocalSparkContext clear the properties even if sc is
null.
 2013-09-19 22:05:23 -07:00
Patrick Wendell cd7222c3dd Merge pull request #938 from ilikerps/master 
Fix issue with spark_ec2 seeing empty security groups
 2013-09-19 14:21:24 -07:00
Aaron Davidson f589ce771a Fix issue with spark_ec2 seeing empty security groups 
Under unknown, but occasional, circumstances, reservation.groups is empty
despite reservation.instances each having groups. This means that the
spark_ec2 get_existing_clusters() method would fail to find any instances.
To fix it, we simply use the instances' groups as the source of truth.

Note that this is actually just a revival of PR #827, now that the issue
has been reproduced.
 2013-09-19 14:09:26 -07:00
jerryshao ffa5f8e11d Fix issue when local properties pass from parent to child thread 2013-09-18 17:33:24 +08:00
Patrick Wendell 2aff7989ab Merge pull request #933 from jey/yarn-typo-fix 
Fix typo in Maven build docs
 2013-09-15 14:05:04 -07:00
Jey Kottalam ac0dd99394 Fix typo in Maven build docs 2013-09-15 13:29:22 -07:00
Patrick Wendell dbd2c4fd94 Merge pull request #932 from pwendell/mesos-version 
Bumping Mesos version to 0.13.0
 2013-09-15 13:20:41 -07:00
Patrick Wendell 9fb0b9d77f Merge pull request #931 from pwendell/yarn-docs 
Explain yarn.version in Maven build docs
 2013-09-15 13:02:53 -07:00
Patrick Wendell c856860c5b Bumping Mesos version to 0.13.0 2013-09-15 12:46:26 -07:00
Patrick Wendell 362ea0c051 Explain yarn.version in Maven build docs 2013-09-15 12:40:49 -07:00
Patrick Wendell c4c1db2dd5 Merge pull request #929 from pwendell/master 
Use different Hadoop version for YARN artifacts.
 2013-09-13 19:52:12 -07:00
Patrick Wendell e9eba8c3ce Use different Hadoop version for YARN artifacts. 
This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.
 2013-09-13 15:34:57 -07:00
Patrick Wendell 45ec1cc90e Add git scm url for publishing 2013-09-12 13:47:31 -07:00
Patrick Wendell a310de69e0 Merge pull request #926 from kayousterhout/dynamic 
Changed localProperties to use ThreadLocal (not DynamicVariable).
 2013-09-11 19:36:11 -07:00
Matei Zaharia 58c7d8b138 Merge pull request #927 from benh/mesos-docs 
Updated Spark on Mesos documentation.
 2013-09-11 17:33:42 -07:00
Benjamin Hindman 8e2602dd70 More updates to Spark on Mesos documentation. 2013-09-11 16:08:54 -07:00
Benjamin Hindman a0f0c1bed2 Updated Spark on Mesos documentation. 2013-09-11 16:05:25 -07:00
Kay Ousterhout 93c4253275 Changed localProperties to use ThreadLocal (not DynamicVariable). 
The fact that DynamicVariable uses an InheritableThreadLocal
can cause problems where the properties end up being shared
across threads in certain circumstances.
 2013-09-11 13:01:39 -07:00
Patrick Wendell 91a59e6b10 Merge pull request #919 from mateiz/jets3t 
Add explicit jets3t dependency, which is excluded in hadoop-client
 2013-09-11 10:21:48 -07:00
Patrick Wendell b9128d34bf Merge pull request #922 from pwendell/port-change 
Change default port number from 3030 to 4030.
 2013-09-11 10:03:06 -07:00
Patrick Wendell bddf135670 Change port from 3030 to 4040 2013-09-11 10:01:38 -07:00
Reynold Xin e07eef83af Merge pull request #925 from davidmccauley/master 
SPARK-894 - Not all WebUI fields delivered VIA JSON
 2013-09-11 07:35:39 -07:00
David McCauley 5dd875c5b5 SPARK-894 - Not all WebUI fields delivered VIA JSON 2013-09-11 10:46:37 +01:00
Matei Zaharia 8432f27fbe Merge pull request #923 from haoyuan/master 
fix run-example script
 2013-09-10 23:19:53 -07:00
Haoyuan Li 45d964b606 better expression 2013-09-10 23:18:22 -07:00
Patrick Wendell d40f1403f3 Merge pull request #921 from pwendell/master 
Fix HDFS access bug with assembly build.
 2013-09-10 23:05:29 -07:00
Haoyuan Li 56b9407848 fix run-example script 2013-09-10 23:03:09 -07:00
Patrick Wendell 0c1985b153 Fix HDFS access bug with assembly build. 
Due to this change in HDFS:
https://issues.apache.org/jira/browse/HADOOP-7549

there is a bug when using the new assembly builds. The symptom is that any HDFS access
results in an exception saying "No filesystem for scheme 'hdfs'". This adds a merge
strategy in the assembly build which fixes the problem.
 2013-09-10 22:05:13 -07:00
Matei Zaharia 2425eb85ca Update Python API features 2013-09-10 11:12:59 -07:00
Matei Zaharia f117dc6d0d Add explicit jets3t dependency, which is excluded in hadoop-client 2013-09-10 06:39:25 +00:00
shivaram 8c14f4b722 Merge pull request #917 from pwendell/master 
Document libgfortran dependency for MLBase
 2013-09-09 22:07:58 -07:00
Patrick Wendell cefee1ed1a Document fortran dependency for MLBase 2013-09-09 21:45:04 -07:00
Matei Zaharia c81377b9ed Merge pull request #915 from ooyala/master 
Get rid of / improve ugly NPE when Utils.deleteRecursively() fails
 2013-09-09 20:16:19 -07:00
Matei Zaharia 61d2a010e1 Merge pull request #916 from mateiz/mkdist-fix 
Fix copy issue in https://github.com/mesos/spark/pull/899
 2013-09-09 18:21:01 -07:00
Evan Chan fdb8b0eec3 Style fix: put body of if within curly braces 2013-09-09 14:29:32 -07:00
Matei Zaharia f5a8afa6c3 Fix copy issue in https://github.com/mesos/spark/pull/899 2013-09-09 13:47:56 -07:00
Matei Zaharia a85758c200 Merge pull request #907 from stephenh/document_coalesce_shuffle 
Add better docs for coalesce.
 2013-09-09 13:45:40 -07:00
Evan Chan 27726079e4 Print out more friendly error if listFiles() fails 
listFiles() could return null if the I/O fails, and this currently results in an ugly NPE which is hard to diagnose.
 2013-09-09 12:58:12 -07:00
Matei Zaharia 084fc36961 Merge pull request #912 from tgravescs/ganglia-pom 
Add metrics-ganglia to core pom file
 2013-09-09 12:01:35 -07:00
Y.CORP.YAHOO.COM\tgraves 2186d93285 Add metrics-ganglia to core pom file 2013-09-09 12:37:33 -05:00
Matei Zaharia 0456384939 Merge pull request #911 from pwendell/ganglia-sink 
Adding Manen dependency for Ganglia
 2013-09-09 09:57:54 -07:00
Stephen Haberman 59003d387d Use a set since shuffle could change order. 2013-09-09 11:45:03 -05:00
Stephen Haberman 6471bfec73 Reword 'evenly distributed' to 'distributed with a hash partitioner. 2013-09-09 11:44:15 -05:00
Patrick Wendell 528fdbae97 Adding Manen dependency 2013-09-09 09:32:18 -07:00