ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4450 commits 6 branches 181 tags 397 MiB
Commit graph

4450 commits

This branch
This branch
All branches
Author SHA1 Message Date
Aaron Davidson 8933f9e98e Add "org.apache." prefix to packages in spark-class 
Lacking this, the if/case statements never trigger on Spark 0.8.0+.
 2013-09-20 19:27:08 -07:00
Reynold Xin 42571d30d0 Smarter take/limit implementation. 2013-09-20 17:09:53 -07:00
Reynold Xin 119de80294 Merge branch 'master' of github.com:mesos/spark 2013-09-20 15:03:55 -07:00
Vadim Chekan fbe40c5806 Serialize and restore spark.cleaner.ttl to savepoint 2013-09-20 12:13:48 -07:00
Joey a00317bb9b Merge pull request #1 from ankurdave/aggregateNeighbors-returns-graph 
Return Graph from Graph.aggregateNeighbors
 2013-09-20 11:29:31 -07:00
Ankur Dave bf05dc7e78 Add a unit test for aggregateNeighbors 2013-09-19 23:45:15 -07:00
Reynold Xin 1d87616b61 Made output of CoGroup and aggregations interruptible. 2013-09-19 23:31:36 -07:00
Mike 9524b943a4 Synchronize on "entries" the remaining update to "currentMemory". 
Make "currentMemory" @volatile, so that it's reads in ensureFreeSpace() are atomic and up-to-date--i.e., currentMemory can't increase while putLock is held (though it could decrease, which would only help ensureFreeSpace()).
 2013-09-19 23:31:35 -07:00
Ankur Dave 7cadeffdf4 Merge branch 'master' into aggregateNeighbors-returns-graph 2013-09-19 23:14:26 -07:00
Ankur Dave f08e520f4c Initialize sc in GraphSuite to avoid NullPointerException 2013-09-19 23:12:24 -07:00
Ankur Dave f02d5c8c53 Fix typo in aggregateNeighbors docs 2013-09-19 23:06:37 -07:00
Ankur Dave 6a5e665cdc Merge pull request #3 from ankurdave/clear-port-properties-after-tests 
After unit tests, clear port properties unconditionally

Also submitted to Spark upstream (mesos/spark#940)
 2013-09-19 22:41:44 -07:00
Ankur Dave 3ebbcaf21c After unit tests, clear port properties unconditionally 2013-09-19 22:14:38 -07:00
Ankur Dave 026dba6aba After unit tests, clear port properties unconditionally 
In MapOutputTrackerSuite, the "remote fetch" test sets spark.driver.port
and spark.hostPort, assuming that they will be cleared by
LocalSparkContext. However, the test never sets sc, so it remains null,
causing LocalSparkContext to skip clearing these properties. Subsequent
tests therefore fail with java.net.BindException: "Address already in
use".

This commit makes LocalSparkContext clear the properties even if sc is
null.
 2013-09-19 22:05:23 -07:00
Ankur Dave 68ad33a127 Merge pull request #2 from ankurdave/package-fixes 
Package fixes (spark.graph -> org.apache.spark.graph)
 2013-09-19 21:30:27 -07:00
Ankur Dave d3cbde0085 Import appropriate Spark core classes 2013-09-19 19:29:58 -07:00
Ankur Dave c278907bf0 Move BytecodeUtils to the right package 2013-09-19 19:28:22 -07:00
Ankur Dave 9632ad3b21 Move IndexedRDDSuite to org.apache.spark 2013-09-19 19:25:52 -07:00
Ankur Dave 4c694bd705 Move IndexedRDD and GraphSuite to org.apache.spark 2013-09-19 19:13:07 -07:00
Reynold Xin c5e40954eb Wrap around cached data to InterruptibleIterator. 2013-09-19 18:44:38 -07:00
Reynold Xin c68e72be59 Added comment to InterruptibleIterator. 2013-09-19 18:40:06 -07:00
Reynold Xin 70953810b4 Added task killing iterator to RDDs that take inputs. 2013-09-19 18:33:16 -07:00
Reynold Xin f19984dafe More logging changes (task killing for local cluster doesn't work yet). 2013-09-19 18:14:51 -07:00
Reynold Xin 85a0dffe0f Made task killing work for standalone cluster schedulers. 2013-09-19 16:41:29 -07:00
Patrick Wendell cd7222c3dd Merge pull request #938 from ilikerps/master 
Fix issue with spark_ec2 seeing empty security groups
 2013-09-19 14:21:24 -07:00
Aaron Davidson f589ce771a Fix issue with spark_ec2 seeing empty security groups 
Under unknown, but occasional, circumstances, reservation.groups is empty
despite reservation.instances each having groups. This means that the
spark_ec2 get_existing_clusters() method would fail to find any instances.
To fix it, we simply use the instances' groups as the source of truth.

Note that this is actually just a revival of PR #827, now that the issue
has been reproduced.
 2013-09-19 14:09:26 -07:00
Reynold Xin 9f8190c17d Fixed a bug for zero partition in JobWaiter. 2013-09-18 22:42:35 -07:00
Ankur Dave 4e967af6af Return Graph from default aggregateNeighbors also 2013-09-18 16:18:33 -07:00
Ankur Dave b04f1a4019 Implement aggregateNeighbors returning Graph 2013-09-18 16:18:33 -07:00
Ankur Dave 9ff783599b Return Graph from aggregateNeighbors; update callers 
This commit only affects the Graph API, not GraphImpl.
 2013-09-18 16:18:33 -07:00
Joseph E. Gonzalez 9335aff946 Merge branch 'indexed_rdd' of /Users/jegonzal/Documents/amplab/spark 2013-09-18 11:58:33 -07:00
Reynold Xin 9332246bd0 Added a hack to kill all active jobs in SparkContext. 2013-09-18 04:38:24 -07:00
Reynold Xin bf515688e7 Allow SparkContext.submitJob to submit a job for only a subset of the partitions. 2013-09-18 04:16:18 -07:00
jerryshao ffa5f8e11d Fix issue when local properties pass from parent to child thread 2013-09-18 17:33:24 +08:00
Joseph E. Gonzalez 55696e2584 GraphX now builds with all merged changes. 2013-09-17 22:42:12 -07:00
Joseph E. Gonzalez c0b392a160 Mering changes from master 2013-09-17 22:03:33 -07:00
Reynold Xin 37d8f37a8e Added a submitJob interface that returns a Future of the result. 2013-09-17 21:13:59 -07:00
Joseph E. Gonzalez 8b59fb72c4 Merging latest changes from spark main branch 2013-09-17 20:56:12 -07:00
Joseph E. Gonzalez a3fb29938c Merging changes between Reynold's branch and Joey's modifications. 2013-09-17 17:34:24 -07:00
Reynold Xin 1cb42e6b2d Properly handle job failure when the job gets killed. 2013-09-16 22:10:45 -07:00
Reynold Xin cbc48be13b Initial commit for job killing. 2013-09-16 18:54:06 -07:00
Reynold Xin 3443d3fd43 Merge branch 'master' of github.com:mesos/spark 2013-09-16 13:10:35 -07:00
Patrick Wendell 2aff7989ab Merge pull request #933 from jey/yarn-typo-fix 
Fix typo in Maven build docs
 2013-09-15 14:05:04 -07:00
Jey Kottalam ac0dd99394 Fix typo in Maven build docs 2013-09-15 13:29:22 -07:00
Patrick Wendell dbd2c4fd94 Merge pull request #932 from pwendell/mesos-version 
Bumping Mesos version to 0.13.0
 2013-09-15 13:20:41 -07:00
Patrick Wendell 9fb0b9d77f Merge pull request #931 from pwendell/yarn-docs 
Explain yarn.version in Maven build docs
 2013-09-15 13:02:53 -07:00
Patrick Wendell c856860c5b Bumping Mesos version to 0.13.0 2013-09-15 12:46:26 -07:00
Patrick Wendell 362ea0c051 Explain yarn.version in Maven build docs 2013-09-15 12:40:49 -07:00
Holden Karau 68068977b8 Fix build on ubuntu 2013-09-14 20:51:11 -07:00
Holden Karau 452db1083c Merge branch 'master' of https://github.com/mesos/spark 2013-09-14 15:54:04 -07:00