ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
548 commits 6 branches 181 tags 397 MiB
Commit graph

548 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ismael Juma 635f501492 Fix copy & paste error in version. 2011-07-18 00:13:37 +01:00
Ismael Juma f686e3dacb Initial work on converting build to SBT 0.10.1 2011-07-15 03:38:30 +01:00
Matei Zaharia cf8f5de61b Merge branch 'master' into scala-2.9 
Conflicts:
	project/build.properties
	repl/src/main/scala/spark/repl/SparkInterpreterLoop.scala
 2011-07-14 17:48:56 -04:00
Matei Zaharia 02678724a4 Update version number to 0.4-SNAPSHOT 2011-07-14 17:47:39 -04:00
Matei Zaharia 7c77b2fa6a Merge branch 'master' into scala-2.9 
Conflicts:
	project/build.properties
 2011-07-14 17:39:34 -04:00
Matei Zaharia c86af80022 Change version to 0.3 2011-07-14 17:38:43 -04:00
Matei Zaharia 1450fd74d9 Merge branch 'master' into scala-2.9 2011-07-14 17:37:24 -04:00
Matei Zaharia ccf48388cd Lowered default number of splits for files 2011-07-14 17:37:04 -04:00
Matei Zaharia 146a18c2a4 Merge branch 'master' into scala-2.9 2011-07-14 17:29:17 -04:00
Matei Zaharia c8eb8b2b90 Set class loader for remote actors to fix a bug that happens in 2.9 2011-07-14 17:29:11 -04:00
Matei Zaharia 8ea67307b9 Merge branch 'master' into scala-2.9 2011-07-14 14:47:12 -04:00
Matei Zaharia e4c3402d2d Renamed ParallelArray to ParallelCollection 2011-07-14 14:47:01 -04:00
Matei Zaharia 9ac461d85d Remove RDD.toString because it looked confusing 2011-07-14 14:39:32 -04:00
Matei Zaharia 797b4547c3 Fix tracking of updates in accumulators to solve an issue that would manifest in the 2.9 interpreter 2011-07-14 14:08:34 -04:00
Matei Zaharia 3efd9e94d8 Merge branch 'master' into scala-2.9 2011-07-14 12:42:57 -04:00
Matei Zaharia 0ccfe20755 Forgot to add a file 2011-07-14 12:42:50 -04:00
Matei Zaharia 38f38dda5b Merge branch 'master' into scala-2.9 2011-07-14 12:42:02 -04:00
Matei Zaharia 969644df8e Cleaned up a few issues to do with default parallelism levels. Also 
renamed HadoopFileWriter to HadoopWriter (since it's not only for files)
and fixed a bug for lookup().
 2011-07-14 12:40:56 -04:00
Matei Zaharia 2fb906e8e5 Merge branch 'master' into scala-2.9 2011-07-14 00:20:14 -04:00
Matei Zaharia 2604939f64 Simplified and documented code a little and added test 2011-07-14 00:19:00 -04:00
Matei Zaharia 2439e51a03 Merge branch 'master' into implicit-sequencefile 2011-07-13 23:20:22 -04:00
Matei Zaharia d0c7958364 Merge branch 'master' into scala-2.9 
Conflicts:
	core/src/main/scala/spark/HadoopFileWriter.scala
 2011-07-13 23:09:33 -04:00
Matei Zaharia 9c0069188b Updated save code to allow non-file-based OutputFormats and added a test 
for file-related stuff
 2011-07-13 23:04:06 -04:00
Matei Zaharia da8a3b8926 Increase default value of spark.locality.wait a little 2011-07-13 20:07:24 -04:00
Matei Zaharia 080869c6ef Merge branch 'master' into scala-2.9 2011-07-13 00:20:08 -04:00
Matei Zaharia 842e14d567 Added mapPartitions operation and a bunch of tests for RDD ops 2011-07-13 00:19:52 -04:00
Matei Zaharia 9b568d37f7 Merge branch 'master' into scala-2.9 
Conflicts:
	core/src/main/scala/spark/RDD.scala
 2011-07-11 22:25:53 -04:00
Matei Zaharia d05fea24f3 Simplified parallel shuffle fetcher to use URLConnection 2011-07-11 22:12:36 -04:00
Matei Zaharia 25c3a7781c Moved PairRDD and SequenceFileRDD functions to separate source files 2011-07-10 00:06:15 -04:00
Matei Zaharia b7f1f62ff5 bug fix 2011-07-09 18:53:02 -04:00
Matei Zaharia 003480f374 Register byte[] with Kryo serializer 2011-07-09 18:08:07 -04:00
Matei Zaharia aea5cb4413 Added parallel shuffle fetcher 2011-07-09 17:25:56 -04:00
Matei Zaharia 4b1646a25f Support for non-filesystem-based Hadoop data sources 2011-07-06 20:37:55 -04:00
Matei Zaharia 07a97d47c2 Support for non-filesystem-based Hadoop data sources 2011-07-06 20:37:34 -04:00
Matei Zaharia 3488c386a9 Initial work to make stuff like sequenceFile[Int, Int] work without 
requiring the user to provide a Writable type. The approach here might
not be the best but it seems to work correctly.
 2011-06-28 17:07:04 -07:00
Matei Zaharia 5633299ec6 Merge remote-tracking branch 'origin/master' into scala-2.9 2011-06-27 22:50:59 -07:00
Matei Zaharia b0ecf1ee41 Don't pass a null context when running tasks locally 2011-06-27 22:50:43 -07:00
Matei Zaharia 85cad5d9dd Fixed HadoopFileWriter to compile for Scala 2.9 2011-06-27 22:44:14 -07:00
Matei Zaharia 393607d5ef Merge branch 'master' into scala-2.9 2011-06-27 18:08:25 -07:00
Matei Zaharia 2f652f1656 Fix a compile error 2011-06-27 18:07:16 -07:00
tdas ae63972a89 Merge pull request #64 from mesos/td-rdd-save 
Functionality to save RDDs to Hadoop files
 2011-06-27 13:44:55 -07:00
Tathagata Das 3f08e1129f Merge branch 'master' into td-rdd-save 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
 2011-06-27 13:43:44 -07:00
Tathagata Das ad842ac823 Merge branch 'master' into td-rdd-save 
Conflicts:
	core/src/main/scala/spark/RDD.scala
 2011-06-27 13:39:11 -07:00
Matei Zaharia bae8a97968 Merge branch 'master' into scala-2.9 
Conflicts:
	repl/src/main/scala/spark/repl/SparkInterpreterLoop.scala
 2011-06-26 19:22:27 -07:00
Matei Zaharia b187675b68 Print version number 0.3 in REPL 2011-06-26 18:27:01 -07:00
Matei Zaharia c4dd68ae21 Merge branch 'mos-bt' 
This merge keeps only the broadcast work in mos-bt because the structure
of shuffle has changed with the new RDD design. We still need some kind
of parallel shuffle but that will be added later.

Conflicts:
	core/src/main/scala/spark/BitTorrentBroadcast.scala
	core/src/main/scala/spark/ChainedBroadcast.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/Utils.scala
	core/src/main/scala/spark/shuffle/BasicLocalFileShuffle.scala
	core/src/main/scala/spark/shuffle/DfsShuffle.scala
 2011-06-26 18:22:12 -07:00
Tathagata Das 38f2ba99cc Further changes to HadoopFileWriter. Implemented ability to save RDDs as SequenceFiles and ObjectFiles. 
1> HadoopFileWriter changed to take class types as constructor parameters (no more generic type)
2> Multiple types of RDD.saveAsHadoopFile() implemented to provide more saving options
3> RDD.saveAsSequenceFile() automatically converts basic types to Writable types before saving as SequenceFile
4> RDD.saveAsObjectFile() serializes objects and saves them to a ObjectFile
5> SparkContext.objectFile() opens the saved ObjectFiles
 2011-06-24 19:51:21 -07:00
Matei Zaharia b626562d54 Merge pull request #63 from ogrisel/outer-join 
Implemented RDD.leftOuterJoin and RDD.rightOuterJoin
 2011-06-24 12:22:15 -07:00
Olivier Grisel 2e3531d8bf Implemented RDD.leftOuterJoin and RDD.rightOuterJoin 2011-06-24 11:00:51 +02:00
Matei Zaharia 095dd9c444 Merge pull request #62 from ogrisel/cogroup-test 
Add missing test for RDD.groupWith
 2011-06-23 10:12:24 -07:00