ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2888 commits 6 branches 181 tags 397 MiB
Commit graph

2888 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia c1c219e263 Merge pull request #564 from maspotts/master 
Allow latest scala in PATH, with SCALA_HOME as override (instead of vice-versa)
 2013-04-14 08:11:23 -07:00
Matei Zaharia c35d530bcf Fix compile error 2013-04-13 12:43:12 -04:00
Matei Zaharia 7c10b3e3cd Merge pull request #565 from andyk/master 
Update wording of section on RDD operations in quick start guide in docs
 2013-04-12 20:55:22 -07:00
Andy Konwinski 60a91b3b59 Update quick-start.md heading on Operations (not just Transformations). 2013-04-12 12:34:51 -07:00
Mike 6f68860891 Reversed the order of tests to find a scala executable (in the case when SPARK_LAUNCH_WITH_SCALA is defined): instead of checking in the PATH first, and only then (if not found) for SCALA_HOME, now we check for SCALA_HOME first, and only then (if not defined) do we look in the PATH. The advantage is that now if the user has a more recent (non-compatible) version of scala in her PATH, she can use SCALA_HOME to point to the older (compatible) version for use with spark. 
Suggested by Josh Rosen in this thread:

  https://groups.google.com/forum/?fromgroups=#!topic/spark-users/NC9JKvP8808
 2013-04-11 20:52:06 -07:00
Matei Zaharia 077ae0a197 Merge pull request #561 from ash211/patch-4 
Add details when BlockManager heartbeats time out
 2013-04-11 19:34:14 -07:00
Matei Zaharia e2aa87558d Merge branch 'master' of github.com:mesos/spark 2013-04-11 22:30:05 -04:00
Matei Zaharia ed336e0d44 Fix tests from different projects running in parallel in SBT 0.12 2013-04-11 22:29:37 -04:00
Andrew Ash 29d3440efb Add details when BlockManager heartbeats time out 
Makes it more clear what the threshold was for tuning spark.storage.blockManagerSlaveTimeoutMs

Before:
WARN  "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats

After:
WARN  "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats: 19216ms exceeds 15000ms
 2013-04-11 01:54:02 -03:00
Matei Zaharia c91ff8d493 Merge pull request #560 from ash211/patch-3 
Typos: cluser -> cluster
 2013-04-10 15:08:23 -07:00
Andrew Ash 6efc8cae8f Typos: cluser -> cluster 2013-04-10 13:44:10 -03:00
Matei Zaharia 7cd83bf0f8 Merge pull request #559 from ash211/patch-example-whitespace 
Uniform whitespace across scala examples
 2013-04-09 22:07:35 -07:00
Matei Zaharia 271a4f3bb6 Merge pull request #555 from holdenk/master 
Retry failed ssh commands in the ec2 python script.
 2013-04-09 22:04:52 -07:00
Holden Karau 9bf24e1d61 Just use a loop for retries 2013-04-09 21:37:02 -07:00
Andrew Ash f1d8871ca1 Uniform whitespace across scala examples 2013-04-09 23:35:13 -04:00
Andrew Ash 18bd41d1a3 Don't pull in old versions of Jackson via hadoop-core 2013-04-09 14:44:47 -04:00
Andrew xia 2f883c515f Contiue to update codes for scala code style 
1.refactor braces for "class" "if" "while" "for" "match"
2.make code lines less than 100
3.refactor class parameter and extends defination
 2013-04-09 13:02:50 +08:00
Matei Zaharia adba773fab Fix passing of superstep in Bagel to avoid seeing new values of the 
superstep value upon recomputation, and set the default storage level in
Bagel to MEMORY_AND_DISK
 2013-04-08 17:34:38 -04:00
Matei Zaharia eed54a25d8 Merge pull request #553 from pwendell/akka-standalone 
SPARK-724 - Have Akka logging enabled by default for standalone daemons
 2013-04-08 09:44:30 -07:00
Matei Zaharia 65caa8f711 Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0' 
Conflicts:
	docs/_config.yml
	project/SparkBuild.scala
 2013-04-08 12:43:17 -04:00
Matei Zaharia 054feb6448 Fixed a bug with zip 2013-04-07 21:15:21 -04:00
Matei Zaharia b5900d47b1 Fix compile warning 2013-04-07 20:55:42 -04:00
Matei Zaharia 1cb3eb9762 Merge remote-tracking branch 'kalpit/master' 
Conflicts:
	project/SparkBuild.scala
 2013-04-07 20:54:18 -04:00
Patrick Wendell b496decf0a Updating based on code review 2013-04-07 17:44:48 -07:00
Matei Zaharia a1586412d6 Updated link to SBT 2013-04-07 20:31:19 -04:00
Matei Zaharia 6962d40b44 Fix deprecated warning 2013-04-07 20:27:33 -04:00
Matei Zaharia 34a47b8bc9 Update Scala version in docs 2013-04-07 20:27:03 -04:00
Matei Zaharia b362df39ea Merge pull request #552 from MLnick/master 
Bumping version for Twitter Algebird to latest
 2013-04-07 17:17:52 -07:00
Matei Zaharia 4b301902f5 Merge pull request #554 from andyk/scala2.9.3 
Fixes SPARK-723 - Update build to Scala 2.9.3
 2013-04-07 17:15:10 -07:00
Mridul Muralidharan 6798a09df8 Add support for building against hadoop2-yarn : adding new maven profile for it 2013-04-07 17:47:38 +05:30
Reynold Xin 583a5858e7 Added more logging to Analytics. 2013-04-07 15:45:28 +08:00
Reynold Xin 3728e1bc40 Code to run bagel vs graph experiments. 2013-04-07 15:05:46 +08:00
shane-huang df47b40b76 Shuffle Performance fix: Use netty embeded OIO file server instead of ConnectionManager 
Shuffle Performance Optimization: do not send 0-byte block requests to reduce network messages
change reference from io.Source to scala.io.Source to avoid looking into io.netty package

Signed-off-by: shane-huang <shengsheng.huang@intel.com>
 2013-04-07 14:37:12 +08:00
Holden Karau ff2130a0ad Retry failed ssh commands. This is especially useful during system startup when the hosts may not have yet come on-line but can be useful at other points for people with flakey connections 2013-04-06 00:35:50 -07:00
Reynold Xin 9eec317835 Minor cleanup. 2013-04-05 23:22:55 +08:00
Reynold Xin 822d9c5b70 Rename rawGraph to graph in Pregel. 2013-04-05 16:16:04 +08:00
Reynold Xin d40c1d5122 Added unit test and fix a partitioner problem. 2013-04-05 15:33:50 +08:00
Joseph E. Gonzalez 14205548ce Merged with master 
Merge branch 'graph' of https://github.com/rxin/spark into graph
 2013-04-04 23:51:06 -07:00
Joseph E. Gonzalez 092708e57e better parsing of graph text files 2013-04-04 23:50:53 -07:00
Reynold Xin e5dd61e720 Rename rawGraph to graph. 2013-04-05 14:47:39 +08:00
Reynold Xin c973e564b9 Changed EdgeDirection from Enumeration to case classes. 2013-04-05 14:46:37 +08:00
Reynold Xin 34ef3e52dc Merge branch 'graph' of github.com:rxin/spark into graph 2013-04-05 14:21:44 +08:00
Reynold Xin 9764e579b8 Minor cleanup. 2013-04-05 14:21:24 +08:00
Joseph E. Gonzalez 3d7a4b1fef More realistic version of Pregel. 2013-04-04 23:17:27 -07:00
Reynold Xin 7134856351 numVertexPartitions and numEdgePartitions are now part of the 
constructor and are immutable.

Also done some cleanups.
 2013-04-05 12:54:59 +08:00
Joseph E. Gonzalez d5b0f4dfa6 added a function to collect the neighborhood of a vertex as well as the skeleton of the program for classic pregel 2013-04-04 19:54:14 -07:00
Joseph E. Gonzalez ad908f7545 added pregel pagerank 2013-04-04 19:31:14 -07:00
Joseph E. Gonzalez d510045d8c fixing the the silly bug in the dynamic pagerank gather function 2013-04-04 19:24:06 -07:00
Joseph E. Gonzalez b537613570 added dynamic graphlab 2013-04-04 19:18:15 -07:00
Joseph E. Gonzalez 4a2b8aa557 added dynamic graphlab but I have not yet tested it. 2013-04-04 18:59:56 -07:00