ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3119 commits 6 branches 181 tags 397 MiB
Commit graph

3119 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mridul Muralidharan dd2b64ec97 Fix bug with atomic update 2013-04-16 03:19:24 +05:30
Mridul Muralidharan 5540ab8243 Use hostname instead of hostport for executor, fix creation of workdir 2013-04-16 02:57:43 +05:30
Mridul Muralidharan eb7e95e833 Commit job to persist files 2013-04-16 02:56:36 +05:30
Matei Zaharia a64c107449 Make ShuffledRDD.prev transient 2013-04-15 16:41:51 -04:00
seanm b42d68c8ce fixing Spark Streaming count() so that 0 will be emitted when there is nothing to count 2013-04-15 12:54:55 -06:00
Mridul Muralidharan 19652a44be Fix issue with FileSuite failing 2013-04-15 19:16:36 +05:30
Mridul Muralidharan 54b3d45b81 Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues 2013-04-15 18:26:50 +05:30
Mridul Muralidharan d90d2af103 Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues 2013-04-15 18:12:11 +05:30
Matei Zaharia ec5e553b41 Merge pull request #558 from ash211/patch-jackson-conflict 
Don't pull in old versions of Jackson via hadoop-core
 2013-04-14 08:20:13 -07:00
Matei Zaharia c1c219e263 Merge pull request #564 from maspotts/master 
Allow latest scala in PATH, with SCALA_HOME as override (instead of vice-versa)
 2013-04-14 08:11:23 -07:00
Matei Zaharia c35d530bcf Fix compile error 2013-04-13 12:43:12 -04:00
Matei Zaharia 7c10b3e3cd Merge pull request #565 from andyk/master 
Update wording of section on RDD operations in quick start guide in docs
 2013-04-12 20:55:22 -07:00
Andy Konwinski 60a91b3b59 Update quick-start.md heading on Operations (not just Transformations). 2013-04-12 12:34:51 -07:00
Mike 6f68860891 Reversed the order of tests to find a scala executable (in the case when SPARK_LAUNCH_WITH_SCALA is defined): instead of checking in the PATH first, and only then (if not found) for SCALA_HOME, now we check for SCALA_HOME first, and only then (if not defined) do we look in the PATH. The advantage is that now if the user has a more recent (non-compatible) version of scala in her PATH, she can use SCALA_HOME to point to the older (compatible) version for use with spark. 
Suggested by Josh Rosen in this thread:

  https://groups.google.com/forum/?fromgroups=#!topic/spark-users/NC9JKvP8808
 2013-04-11 20:52:06 -07:00
Matei Zaharia 077ae0a197 Merge pull request #561 from ash211/patch-4 
Add details when BlockManager heartbeats time out
 2013-04-11 19:34:14 -07:00
Matei Zaharia e2aa87558d Merge branch 'master' of github.com:mesos/spark 2013-04-11 22:30:05 -04:00
Matei Zaharia ed336e0d44 Fix tests from different projects running in parallel in SBT 0.12 2013-04-11 22:29:37 -04:00
Andrew Ash 29d3440efb Add details when BlockManager heartbeats time out 
Makes it more clear what the threshold was for tuning spark.storage.blockManagerSlaveTimeoutMs

Before:
WARN  "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats

After:
WARN  "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats: 19216ms exceeds 15000ms
 2013-04-11 01:54:02 -03:00
Matei Zaharia c91ff8d493 Merge pull request #560 from ash211/patch-3 
Typos: cluser -> cluster
 2013-04-10 15:08:23 -07:00
Andrew Ash 6efc8cae8f Typos: cluser -> cluster 2013-04-10 13:44:10 -03:00
Matei Zaharia 7cd83bf0f8 Merge pull request #559 from ash211/patch-example-whitespace 
Uniform whitespace across scala examples
 2013-04-09 22:07:35 -07:00
Matei Zaharia 271a4f3bb6 Merge pull request #555 from holdenk/master 
Retry failed ssh commands in the ec2 python script.
 2013-04-09 22:04:52 -07:00
Holden Karau 9bf24e1d61 Just use a loop for retries 2013-04-09 21:37:02 -07:00
Andrew Ash f1d8871ca1 Uniform whitespace across scala examples 2013-04-09 23:35:13 -04:00
Matei Zaharia 8ac9efba5a Merge pull request #527 from Reinvigorate/sm-kafka-cleanup 
KafkaInputDStream fixes and improvements
 2013-04-09 13:50:50 -07:00
Andrew Ash 18bd41d1a3 Don't pull in old versions of Jackson via hadoop-core 2013-04-09 14:44:47 -04:00
Andrew xia 2f883c515f Contiue to update codes for scala code style 
1.refactor braces for "class" "if" "while" "for" "match"
2.make code lines less than 100
3.refactor class parameter and extends defination
 2013-04-09 13:02:50 +08:00
Matei Zaharia adba773fab Fix passing of superstep in Bagel to avoid seeing new values of the 
superstep value upon recomputation, and set the default storage level in
Bagel to MEMORY_AND_DISK
 2013-04-08 17:34:38 -04:00
Matei Zaharia eed54a25d8 Merge pull request #553 from pwendell/akka-standalone 
SPARK-724 - Have Akka logging enabled by default for standalone daemons
 2013-04-08 09:44:30 -07:00
Matei Zaharia 65caa8f711 Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0' 
Conflicts:
	docs/_config.yml
	project/SparkBuild.scala
 2013-04-08 12:43:17 -04:00
Matei Zaharia 054feb6448 Fixed a bug with zip 2013-04-07 21:15:21 -04:00
Matei Zaharia b5900d47b1 Fix compile warning 2013-04-07 20:55:42 -04:00
Matei Zaharia 1cb3eb9762 Merge remote-tracking branch 'kalpit/master' 
Conflicts:
	project/SparkBuild.scala
 2013-04-07 20:54:18 -04:00
Patrick Wendell b496decf0a Updating based on code review 2013-04-07 17:44:48 -07:00
Matei Zaharia a1586412d6 Updated link to SBT 2013-04-07 20:31:19 -04:00
Matei Zaharia 6962d40b44 Fix deprecated warning 2013-04-07 20:27:33 -04:00
Matei Zaharia 34a47b8bc9 Update Scala version in docs 2013-04-07 20:27:03 -04:00
Matei Zaharia b362df39ea Merge pull request #552 from MLnick/master 
Bumping version for Twitter Algebird to latest
 2013-04-07 17:17:52 -07:00
Matei Zaharia 4b301902f5 Merge pull request #554 from andyk/scala2.9.3 
Fixes SPARK-723 - Update build to Scala 2.9.3
 2013-04-07 17:15:10 -07:00
Mridul Muralidharan 6798a09df8 Add support for building against hadoop2-yarn : adding new maven profile for it 2013-04-07 17:47:38 +05:30
shane-huang df47b40b76 Shuffle Performance fix: Use netty embeded OIO file server instead of ConnectionManager 
Shuffle Performance Optimization: do not send 0-byte block requests to reduce network messages
change reference from io.Source to scala.io.Source to avoid looking into io.netty package

Signed-off-by: shane-huang <shengsheng.huang@intel.com>
 2013-04-07 14:37:12 +08:00
Holden Karau ff2130a0ad Retry failed ssh commands. This is especially useful during system startup when the hosts may not have yet come on-line but can be useful at other points for people with flakey connections 2013-04-06 00:35:50 -07:00
Andy Konwinski 5555811bd5 Update build to Scala 2.9.3 2013-04-04 13:26:45 -07:00
Patrick Wendell 9b68ceaa26 SPARK-724 - Have Akka logging enabled by default for standalone daemons 
See the JIRA for more details.

I was only able to test the bash version (don't have Windows)
so maybe check over that the syntax is correct there.
 2013-04-03 14:29:46 -07:00
Nick Pentreath 0f54344fd8 Bumping Algebird version in examples now that it supports JDK 1.6 2013-04-03 13:15:34 +02:00
Matei Zaharia dfe98ca798 Merge pull request #550 from erikvanoosten/master 
corrected Algebird example
 2013-04-02 19:24:12 -07:00
Matei Zaharia b5d78307ca Merge pull request #551 from jey/python-bugfixes 
Python bugfixes
 2013-04-02 19:23:45 -07:00
Jey Kottalam 9a731f5a6d Fix Python saveAsTextFile doctest to not expect order to be preserved 2013-04-02 11:59:20 -07:00
Jey Kottalam 20604001e2 Fix argv handling in Python transitive closure example 2013-04-02 11:59:07 -07:00
Erik van oosten b5e60c3253 Corrected order of CountMinSketchMonoid arguments 2013-04-02 15:25:22 +03:00