ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2641 commits 6 branches 181 tags 397 MiB
Commit graph

2641 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mridul Muralidharan 609a817f52 Integrate review comments on pull request 2013-05-02 06:44:33 +05:30
Mridul Muralidharan 27764a00f4 Fix some npe introduced accidentally 2013-05-01 20:56:05 +05:30
Mridul Muralidharan d960e7e0f8 a) Add support for hyper local scheduling - specific to a host + port - before trying host local scheduling. 
b) Add some fixes to test code to ensure it passes (and fixes some other issues).

c) Fix bug in task scheduling which incorrectly used availableCores instead of all cores on the node.
 2013-05-01 20:24:00 +05:30
Mridul Muralidharan 60cabb35cb Add addition catch block for exception too 2013-05-01 01:17:14 +05:30
Mridul Muralidharan c446ac31d7 Spurious commit, reverting gitignore change 2013-05-01 00:32:30 +05:30
Mridul Muralidharan 3b748ced22 Be more aggressive and defensive in all uses of SelectionKey in select loop 2013-05-01 00:30:30 +05:30
Mridul Muralidharan 0f45477be1 Change indentation 2013-05-01 00:10:02 +05:30
Mridul Muralidharan 538614acfe Be more aggressive and defensive in select also 2013-05-01 00:05:32 +05:30
Mridul Muralidharan 48854e1dbf If key is not valid, close connection 2013-04-30 23:59:33 +05:30
Mridul Muralidharan e46d547ccd Fix issues reported by Reynold 2013-04-30 16:15:56 +05:30
Mridul Muralidharan 430c531464 Remove debug statements 2013-04-29 00:24:30 +05:30
Mridul Muralidharan 3a89a76b87 Make log message more descriptive to aid in debugging 2013-04-29 00:04:12 +05:30
Mridul Muralidharan 9bd439502e Remove spurious commit 2013-04-28 23:09:08 +05:30
Mridul Muralidharan 7fa6978a1e Allow CheckpointWriter pending tasks to finish 2013-04-28 23:08:10 +05:30
Mridul Muralidharan 00c7a37604 Merge branch 'master' of github.com:mridulm/spark 2013-04-28 22:44:34 +05:30
Mridul Muralidharan afee902443 Attempt to fix streaming test failures after yarn branch merge 2013-04-28 22:26:45 +05:30
Matei Zaharia 1f20ef2567 Merge branch 'master' of github.com:mesos/spark 2013-04-25 20:03:13 -07:00
Matei Zaharia 1b169f190c Exclude old versions of Netty, which had a different Maven organization 2013-04-25 19:52:12 -07:00
Matei Zaharia cf54b824ff Merge pull request #580 from pwendell/quickstart 
SPARK-739 Have quickstart standlone job use README
 2013-04-25 11:45:58 -07:00
Patrick Wendell a72134a6ac SPARK-739 Have quickstart standlone job use README 2013-04-25 10:39:28 -07:00
Matei Zaharia 6e6b5204ea Create an empty directory when checkpointing a 0-partition RDD (fixes a 
test failure on Hadoop 2.0)
 2013-04-25 00:42:37 -07:00
Matei Zaharia eef9ea1993 Update unit test memory to 2 GB 2013-04-25 00:42:29 -07:00
Matei Zaharia 01d9ba5038 Add back line removed during YARN merge 2013-04-25 00:11:27 -07:00
Matei Zaharia 118a6c76f5 Merge pull request #575 from mridulm/master 
Manual merge of yarn branch to trunk
 2013-04-24 08:42:30 -07:00
Mridul Muralidharan 3b594a4e3b Do not add signature files - results in validation errors when using assembled file 2013-04-24 10:18:25 +05:30
Mridul Muralidharan dd515ca3ee Attempt at fixing merge conflict 2013-04-24 09:24:17 +05:30
Mridul Muralidharan d09db1c051 concurrentRestrictions fails for this PR - but works for master, probably some version change 2013-04-24 09:15:29 +05:30
Mridul Muralidharan adcda84f96 Pull latest SparkBuild.scala from master and merge conflicts 2013-04-24 08:57:25 +05:30
Mridul Muralidharan 5b85c715c8 Revert back to 2.0.2-alpha : 0.23.7 has protocol changes which break against cloudera 2013-04-24 02:57:51 +05:30
Mridul Muralidharan 8faf5c51c3 Patch from Thomas Graves to improve the YARN Client, and move to more production ready hadoop yarn branch 2013-04-24 02:31:57 +05:30
Mridul Muralidharan b11058f42c Ensure that maven package adds yarn jars as part of shaded jar for hadoop2-yarn profile 2013-04-23 22:48:32 +05:30
Mridul Muralidharan 7acab3ab45 Fix review comments, add a new api to SparkHadoopUtil to create appropriate Configuration. Modify an example to show how to use SplitInfo 2013-04-22 08:01:13 +05:30
Matei Zaharia 17e076de80 Turn on forking in test JVMs to reduce the pressure on perm gen and code 
cache sizes due to having 2 instances of the Scala compiler and a bunch
of classloaders.
 2013-04-18 22:25:57 -07:00
Mridul Muralidharan ac2e8e8720 Add some basic documentation 2013-04-19 00:13:19 +05:30
Mridul Muralidharan 5ee2f5c483 Cache pattern, add (commented out) alternatives for check* apis 2013-04-17 23:13:34 +05:30
Mridul Muralidharan f07961060d Add a small note on spark.tasks.schedule.aggression 2013-04-17 23:13:02 +05:30
Matei Zaharia 5d8a71c484 Merge pull request #570 from jey/increase-codecache-size 
Increase ReservedCodeCacheSize for sbt
 2013-04-16 19:48:02 -07:00
Mridul Muralidharan 5d891534fd Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet. Also, add netty dependency explicitly to prevent resolving to older 2.3x version. Additionally, comment out retrievePattern to ensure correct netty is picked up 2013-04-17 05:54:43 +05:30
Mridul Muralidharan 46779b4745 Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet 2013-04-17 05:53:28 +05:30
Mridul Muralidharan 02dffd2eb0 Ensure all ask/await block for spark.akka.askTimeout - so that it is controllable : instead of arbitrary timeouts spread across codebase. In our tests, we use 30 seconds, though default of 10 is maintained 2013-04-17 05:52:57 +05:30
Mridul Muralidharan a402b23bcd Fudge order of classpath - so that our jars take precedence over what is in CLASSPATH variable. Sounds logical, hope there is no issue cos of it 2013-04-17 05:52:00 +05:30
Mridul Muralidharan bcdde331c3 Move from master to driver 2013-04-17 04:12:18 +05:30
Jey Kottalam 6bfe4bf3eb Increase ReservedCodeCacheSize for sbt 2013-04-16 09:50:59 -07:00
Mridul Muralidharan ad80f68eb5 remove spurious debug statements 2013-04-16 22:15:34 +05:30
Mridul Muralidharan f7969f72ee Fix exception when checkpoint path does not exist (no data in rdd which is being checkpointed for example) 2013-04-16 21:51:38 +05:30
Mridul Muralidharan 323ab8ff3b Scala does not prevent variable shadowing ! Sick error due to it ... 2013-04-16 17:05:10 +05:30
Mridul Muralidharan 59c380d69a Fix npe 2013-04-16 03:29:38 +05:30
Mridul Muralidharan dd2b64ec97 Fix bug with atomic update 2013-04-16 03:19:24 +05:30
Mridul Muralidharan 5540ab8243 Use hostname instead of hostport for executor, fix creation of workdir 2013-04-16 02:57:43 +05:30
Mridul Muralidharan eb7e95e833 Commit job to persist files 2013-04-16 02:56:36 +05:30