ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2662 commits 6 branches 181 tags 397 MiB
Commit graph

2662 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mridul Muralidharan 8faf5c51c3 Patch from Thomas Graves to improve the YARN Client, and move to more production ready hadoop yarn branch 2013-04-24 02:31:57 +05:30
Mridul Muralidharan b11058f42c Ensure that maven package adds yarn jars as part of shaded jar for hadoop2-yarn profile 2013-04-23 22:48:32 +05:30
Mridul Muralidharan 7acab3ab45 Fix review comments, add a new api to SparkHadoopUtil to create appropriate Configuration. Modify an example to show how to use SplitInfo 2013-04-22 08:01:13 +05:30
Matei Zaharia 17e076de80 Turn on forking in test JVMs to reduce the pressure on perm gen and code 
cache sizes due to having 2 instances of the Scala compiler and a bunch
of classloaders.
 2013-04-18 22:25:57 -07:00
Mridul Muralidharan ac2e8e8720 Add some basic documentation 2013-04-19 00:13:19 +05:30
Mridul Muralidharan 5ee2f5c483 Cache pattern, add (commented out) alternatives for check* apis 2013-04-17 23:13:34 +05:30
Mridul Muralidharan f07961060d Add a small note on spark.tasks.schedule.aggression 2013-04-17 23:13:02 +05:30
Matei Zaharia 5d8a71c484 Merge pull request #570 from jey/increase-codecache-size 
Increase ReservedCodeCacheSize for sbt
 2013-04-16 19:48:02 -07:00
Mridul Muralidharan 5d891534fd Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet. Also, add netty dependency explicitly to prevent resolving to older 2.3x version. Additionally, comment out retrievePattern to ensure correct netty is picked up 2013-04-17 05:54:43 +05:30
Mridul Muralidharan 46779b4745 Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet 2013-04-17 05:53:28 +05:30
Mridul Muralidharan 02dffd2eb0 Ensure all ask/await block for spark.akka.askTimeout - so that it is controllable : instead of arbitrary timeouts spread across codebase. In our tests, we use 30 seconds, though default of 10 is maintained 2013-04-17 05:52:57 +05:30
Mridul Muralidharan a402b23bcd Fudge order of classpath - so that our jars take precedence over what is in CLASSPATH variable. Sounds logical, hope there is no issue cos of it 2013-04-17 05:52:00 +05:30
Mridul Muralidharan bcdde331c3 Move from master to driver 2013-04-17 04:12:18 +05:30
Jey Kottalam 6bfe4bf3eb Increase ReservedCodeCacheSize for sbt 2013-04-16 09:50:59 -07:00
Mridul Muralidharan ad80f68eb5 remove spurious debug statements 2013-04-16 22:15:34 +05:30
Mridul Muralidharan f7969f72ee Fix exception when checkpoint path does not exist (no data in rdd which is being checkpointed for example) 2013-04-16 21:51:38 +05:30
Mridul Muralidharan 323ab8ff3b Scala does not prevent variable shadowing ! Sick error due to it ... 2013-04-16 17:05:10 +05:30
Mridul Muralidharan 59c380d69a Fix npe 2013-04-16 03:29:38 +05:30
Mridul Muralidharan dd2b64ec97 Fix bug with atomic update 2013-04-16 03:19:24 +05:30
Mridul Muralidharan 5540ab8243 Use hostname instead of hostport for executor, fix creation of workdir 2013-04-16 02:57:43 +05:30
Mridul Muralidharan eb7e95e833 Commit job to persist files 2013-04-16 02:56:36 +05:30
Matei Zaharia a64c107449 Make ShuffledRDD.prev transient 2013-04-15 16:41:51 -04:00
Mridul Muralidharan 19652a44be Fix issue with FileSuite failing 2013-04-15 19:16:36 +05:30
Mridul Muralidharan 54b3d45b81 Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues 2013-04-15 18:26:50 +05:30
Mridul Muralidharan d90d2af103 Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues 2013-04-15 18:12:11 +05:30
Matei Zaharia ec5e553b41 Merge pull request #558 from ash211/patch-jackson-conflict 
Don't pull in old versions of Jackson via hadoop-core
 2013-04-14 08:20:13 -07:00
Matei Zaharia c1c219e263 Merge pull request #564 from maspotts/master 
Allow latest scala in PATH, with SCALA_HOME as override (instead of vice-versa)
 2013-04-14 08:11:23 -07:00
Matei Zaharia c35d530bcf Fix compile error 2013-04-13 12:43:12 -04:00
Matei Zaharia 7c10b3e3cd Merge pull request #565 from andyk/master 
Update wording of section on RDD operations in quick start guide in docs
 2013-04-12 20:55:22 -07:00
Andy Konwinski 60a91b3b59 Update quick-start.md heading on Operations (not just Transformations). 2013-04-12 12:34:51 -07:00
Mike 6f68860891 Reversed the order of tests to find a scala executable (in the case when SPARK_LAUNCH_WITH_SCALA is defined): instead of checking in the PATH first, and only then (if not found) for SCALA_HOME, now we check for SCALA_HOME first, and only then (if not defined) do we look in the PATH. The advantage is that now if the user has a more recent (non-compatible) version of scala in her PATH, she can use SCALA_HOME to point to the older (compatible) version for use with spark. 
Suggested by Josh Rosen in this thread:

  https://groups.google.com/forum/?fromgroups=#!topic/spark-users/NC9JKvP8808
 2013-04-11 20:52:06 -07:00
Matei Zaharia 077ae0a197 Merge pull request #561 from ash211/patch-4 
Add details when BlockManager heartbeats time out
 2013-04-11 19:34:14 -07:00
Matei Zaharia e2aa87558d Merge branch 'master' of github.com:mesos/spark 2013-04-11 22:30:05 -04:00
Matei Zaharia ed336e0d44 Fix tests from different projects running in parallel in SBT 0.12 2013-04-11 22:29:37 -04:00
Andrew Ash 29d3440efb Add details when BlockManager heartbeats time out 
Makes it more clear what the threshold was for tuning spark.storage.blockManagerSlaveTimeoutMs

Before:
WARN  "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats

After:
WARN  "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats: 19216ms exceeds 15000ms
 2013-04-11 01:54:02 -03:00
Matei Zaharia c91ff8d493 Merge pull request #560 from ash211/patch-3 
Typos: cluser -> cluster
 2013-04-10 15:08:23 -07:00
Andrew Ash 6efc8cae8f Typos: cluser -> cluster 2013-04-10 13:44:10 -03:00
Matei Zaharia 7cd83bf0f8 Merge pull request #559 from ash211/patch-example-whitespace 
Uniform whitespace across scala examples
 2013-04-09 22:07:35 -07:00
Matei Zaharia 271a4f3bb6 Merge pull request #555 from holdenk/master 
Retry failed ssh commands in the ec2 python script.
 2013-04-09 22:04:52 -07:00
Holden Karau 9bf24e1d61 Just use a loop for retries 2013-04-09 21:37:02 -07:00
Andrew Ash f1d8871ca1 Uniform whitespace across scala examples 2013-04-09 23:35:13 -04:00
Andrew Ash 18bd41d1a3 Don't pull in old versions of Jackson via hadoop-core 2013-04-09 14:44:47 -04:00
Matei Zaharia adba773fab Fix passing of superstep in Bagel to avoid seeing new values of the 
superstep value upon recomputation, and set the default storage level in
Bagel to MEMORY_AND_DISK
 2013-04-08 17:34:38 -04:00
Matei Zaharia eed54a25d8 Merge pull request #553 from pwendell/akka-standalone 
SPARK-724 - Have Akka logging enabled by default for standalone daemons
 2013-04-08 09:44:30 -07:00
Matei Zaharia 65caa8f711 Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0' 
Conflicts:
	docs/_config.yml
	project/SparkBuild.scala
 2013-04-08 12:43:17 -04:00
Matei Zaharia 054feb6448 Fixed a bug with zip 2013-04-07 21:15:21 -04:00
Matei Zaharia b5900d47b1 Fix compile warning 2013-04-07 20:55:42 -04:00
Matei Zaharia 1cb3eb9762 Merge remote-tracking branch 'kalpit/master' 
Conflicts:
	project/SparkBuild.scala
 2013-04-07 20:54:18 -04:00
Patrick Wendell b496decf0a Updating based on code review 2013-04-07 17:44:48 -07:00
Matei Zaharia a1586412d6 Updated link to SBT 2013-04-07 20:31:19 -04:00