Reynold Xin
ff540a015b
Merge branch 'master' of github.com:markhamstra/incubator-spark
2013-09-23 11:55:02 -07:00
Reynold Xin
f4dc9d37f8
Merge branch 'master' of github.com:mesos/spark
2013-09-23 11:52:52 -07:00
Y.CORP.YAHOO.COM\tgraves
9d4246863a
Support distributed cache files and archives on spark on yarn and attempt to cleanup the staging directory on exit
2013-09-23 09:09:59 -05:00
Nick Pentreath
d952f04c8e
Merge remote-tracking branch 'upstream/master' into implicit-als
2013-09-23 13:07:40 +02:00
Kay Ousterhout
c75eb14fe5
Send Task results through the block manager when larger than Akka frame size.
...
This change requires adding an extra failure mode: tasks can complete
successfully, but the result gets lost or flushed from the block manager
before it's been fetched.
2013-09-22 21:20:48 -07:00
Holden Karau
7fe0b0ff56
Switch indent from 2 to 4 spaces
2013-09-22 19:44:51 -07:00
Reynold Xin
834686b108
Merge pull request #928 from jerryshao/fairscheduler-refactor
...
Refactor FairSchedulableBuilder
2013-09-22 15:06:48 -07:00
Harvey
ef34cfb26c
Move Configuration broadcasts to SparkContext.
2013-09-22 14:43:58 -07:00
Harvey
a6eeb5ffd5
Add a cache for HadoopRDD metadata needed during computation.
...
Currently, the cache is in SparkHadoopUtils, since it's conveniently a member of the SparkEnv.
2013-09-22 03:09:17 -07:00
jerryshao
77e9da1f34
Change Exception to NoSuchElementException and minor style fix
2013-09-22 16:50:08 +08:00
jerryshao
85024acd2e
Remove infix style and others
2013-09-22 14:20:55 +08:00
jerryshao
5850f599dd
Refactor FairSchedulableBuilder:
...
1. Configuration can be read from classpath if not set explicitly.
2. Add missing close handler.
2013-09-22 14:20:55 +08:00
Reynold Xin
a2ea069a5f
Merge pull request #937 from jerryshao/localProperties-fix
...
Fix PR926 local properties issues in Spark Streaming like scenarios
2013-09-21 23:04:42 -07:00
Reynold Xin
f06f2da2cb
Merge pull request #941 from ilikerps/master
...
Add "org.apache." prefix to packages in spark-class
2013-09-21 22:43:34 -07:00
Reynold Xin
7bb12a2af3
Merge pull request #940 from ankurdave/clear-port-properties-after-tests
...
After unit tests, clear port properties unconditionally
2013-09-21 22:42:46 -07:00
Harvey
be0fc7246f
Split HadoopRDD into one for general Hadoop datasets and one tailored to Hadoop files, which is a common case.
...
This is the first step to avoiding unnecessary Configuration broadcasts per HadoopRDD instantiation.
2013-09-21 21:14:14 -07:00
jerryshao
aa0c29f747
Add barrier for local properties unit test and fix some styles
2013-09-22 09:53:11 +08:00
Aaron Davidson
8933f9e98e
Add "org.apache." prefix to packages in spark-class
...
Lacking this, the if/case statements never trigger on Spark 0.8.0+.
2013-09-20 19:27:08 -07:00
Reynold Xin
42571d30d0
Smarter take/limit implementation.
2013-09-20 17:09:53 -07:00
Reynold Xin
119de80294
Merge branch 'master' of github.com:mesos/spark
2013-09-20 15:03:55 -07:00
Joey
a00317bb9b
Merge pull request #1 from ankurdave/aggregateNeighbors-returns-graph
...
Return Graph from Graph.aggregateNeighbors
2013-09-20 11:29:31 -07:00
Ankur Dave
bf05dc7e78
Add a unit test for aggregateNeighbors
2013-09-19 23:45:15 -07:00
Mike
9524b943a4
Synchronize on "entries" the remaining update to "currentMemory".
...
Make "currentMemory" @volatile, so that it's reads in ensureFreeSpace() are atomic and up-to-date--i.e., currentMemory can't increase while putLock is held (though it could decrease, which would only help ensureFreeSpace()).
2013-09-19 23:31:35 -07:00
Ankur Dave
7cadeffdf4
Merge branch 'master' into aggregateNeighbors-returns-graph
2013-09-19 23:14:26 -07:00
Ankur Dave
f08e520f4c
Initialize sc in GraphSuite to avoid NullPointerException
2013-09-19 23:12:24 -07:00
Ankur Dave
f02d5c8c53
Fix typo in aggregateNeighbors docs
2013-09-19 23:06:37 -07:00
Ankur Dave
6a5e665cdc
Merge pull request #3 from ankurdave/clear-port-properties-after-tests
...
After unit tests, clear port properties unconditionally
Also submitted to Spark upstream (mesos/spark#940 )
2013-09-19 22:41:44 -07:00
Ankur Dave
3ebbcaf21c
After unit tests, clear port properties unconditionally
2013-09-19 22:14:38 -07:00
Ankur Dave
026dba6aba
After unit tests, clear port properties unconditionally
...
In MapOutputTrackerSuite, the "remote fetch" test sets spark.driver.port
and spark.hostPort, assuming that they will be cleared by
LocalSparkContext. However, the test never sets sc, so it remains null,
causing LocalSparkContext to skip clearing these properties. Subsequent
tests therefore fail with java.net.BindException: "Address already in
use".
This commit makes LocalSparkContext clear the properties even if sc is
null.
2013-09-19 22:05:23 -07:00
Ankur Dave
68ad33a127
Merge pull request #2 from ankurdave/package-fixes
...
Package fixes (spark.graph -> org.apache.spark.graph)
2013-09-19 21:30:27 -07:00
Ankur Dave
d3cbde0085
Import appropriate Spark core classes
2013-09-19 19:29:58 -07:00
Ankur Dave
c278907bf0
Move BytecodeUtils to the right package
2013-09-19 19:28:22 -07:00
Ankur Dave
9632ad3b21
Move IndexedRDDSuite to org.apache.spark
2013-09-19 19:25:52 -07:00
Ankur Dave
4c694bd705
Move IndexedRDD and GraphSuite to org.apache.spark
2013-09-19 19:13:07 -07:00
Patrick Wendell
cd7222c3dd
Merge pull request #938 from ilikerps/master
...
Fix issue with spark_ec2 seeing empty security groups
2013-09-19 14:21:24 -07:00
Aaron Davidson
f589ce771a
Fix issue with spark_ec2 seeing empty security groups
...
Under unknown, but occasional, circumstances, reservation.groups is empty
despite reservation.instances each having groups. This means that the
spark_ec2 get_existing_clusters() method would fail to find any instances.
To fix it, we simply use the instances' groups as the source of truth.
Note that this is actually just a revival of PR #827 , now that the issue
has been reproduced.
2013-09-19 14:09:26 -07:00
Ankur Dave
4e967af6af
Return Graph from default aggregateNeighbors also
2013-09-18 16:18:33 -07:00
Ankur Dave
b04f1a4019
Implement aggregateNeighbors returning Graph
2013-09-18 16:18:33 -07:00
Ankur Dave
9ff783599b
Return Graph from aggregateNeighbors; update callers
...
This commit only affects the Graph API, not GraphImpl.
2013-09-18 16:18:33 -07:00
Joseph E. Gonzalez
9335aff946
Merge branch 'indexed_rdd' of /Users/jegonzal/Documents/amplab/spark
2013-09-18 11:58:33 -07:00
jerryshao
ffa5f8e11d
Fix issue when local properties pass from parent to child thread
2013-09-18 17:33:24 +08:00
Joseph E. Gonzalez
55696e2584
GraphX now builds with all merged changes.
2013-09-17 22:42:12 -07:00
Joseph E. Gonzalez
c0b392a160
Mering changes from master
2013-09-17 22:03:33 -07:00
Joseph E. Gonzalez
8b59fb72c4
Merging latest changes from spark main branch
2013-09-17 20:56:12 -07:00
Joseph E. Gonzalez
a3fb29938c
Merging changes between Reynold's branch and Joey's modifications.
2013-09-17 17:34:24 -07:00
Reynold Xin
3443d3fd43
Merge branch 'master' of github.com:mesos/spark
2013-09-16 13:10:35 -07:00
Patrick Wendell
2aff7989ab
Merge pull request #933 from jey/yarn-typo-fix
...
Fix typo in Maven build docs
2013-09-15 14:05:04 -07:00
Jey Kottalam
ac0dd99394
Fix typo in Maven build docs
2013-09-15 13:29:22 -07:00
Patrick Wendell
dbd2c4fd94
Merge pull request #932 from pwendell/mesos-version
...
Bumping Mesos version to 0.13.0
2013-09-15 13:20:41 -07:00
Patrick Wendell
9fb0b9d77f
Merge pull request #931 from pwendell/yarn-docs
...
Explain yarn.version in Maven build docs
2013-09-15 13:02:53 -07:00