ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1512 commits 6 branches 181 tags 397 MiB
Commit graph

1512 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 70f02fa912 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-05 22:00:22 -07:00
Matei Zaharia 95ef307ef5 Merge pull request #249 from andyk/move-rdds-to-their-own-package 
Move RDD classes/files to their own package/directory
 2012-10-05 21:59:57 -07:00
Andy Konwinski a242cdd0a6 Factor subclasses of RDD out of RDD.scala into their own classes 
in the rdd package.
 2012-10-05 19:53:54 -07:00
Andy Konwinski d7363a6b8a Moves all files in core/src/main/scala/ that have RDD in their name 
from that directory to a new core/src/main/scala/rdd directory.
 2012-10-05 19:23:45 -07:00
Andy Konwinski e0067da082 Moves all files in core/src/main/scala/ that have RDD in them from 
package spark to package spark.rdd and updates all references to them.
 2012-10-05 19:23:45 -07:00
Matei Zaharia 69588baf65 Cleaning up code slightly 2012-10-05 19:16:09 -07:00
root f52bc09a34 Reduce some overly aggressive logging in connection manager 2012-10-06 01:54:39 +00:00
Matei Zaharia e6e27a05d8 Links quick start from nav bar 2012-10-05 17:06:55 -07:00
Shivaram Venkataraman b6e4f46a96 Fix SizeEstimator tests to work with String classes in JDK 6 and 7 
Conflicts:

	core/src/test/scala/spark/BoundedMemoryCacheSuite.scala
 2012-10-05 16:58:57 -07:00
Matei Zaharia 7eae2d1229 Merge branch 'master' into reduce-logging 2012-10-05 16:36:16 -07:00
Matei Zaharia d9bcc2a6e3 Merge branch 'master' of github.com:mesos/spark 2012-10-05 16:35:48 -07:00
Matei Zaharia 1620d59ea4 Merge pull request #246 from shivaram/size-estimator-fix-master 
Fix SizeEstimator tests to work with String classes in JDK 6 and 7
 2012-10-05 10:28:30 -07:00
Matei Zaharia e3ae98b54e Merge pull request #247 from squito/dev 
Dev
 2012-10-05 10:27:18 -07:00
Imran Rashid e0698f8f26 change tests to show utility of localValue 2012-10-04 23:05:42 -07:00
Imran Rashid 82a3327862 make accumulator.localValue public, add tests 
Conflicts:
	core/src/test/scala/spark/AccumulatorSuite.scala
 2012-10-04 23:05:01 -07:00
Matei Zaharia 8c82f43db3 Scaladoc documentation for some core Spark functionality 2012-10-04 22:59:36 -07:00
Matei Zaharia 08edb04f47 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-04 22:11:32 -07:00
Shivaram Venkataraman 5975d2ee3b Fix SizeEstimator tests to work with String classes in JDK 6 and 7 2012-10-04 19:42:57 -07:00
Matei Zaharia 5a7b370225 Only group elements ten at a time into SequenceFile records in 
saveAsObjectFile
 2012-10-04 16:49:30 -07:00
Matei Zaharia 66d7066d4f Let the reducer retry if a fetch fails before reading all records 2012-10-04 16:41:17 -07:00
Matei Zaharia c535762deb Don't check for JARs in core/lib anymore 2012-10-04 15:11:43 -07:00
Matei Zaharia 588120cd71 Add more logging for number of records fetched by each reduce 2012-10-04 11:54:47 -07:00
Matei Zaharia d6d071f19a Merge pull request #245 from pwendell/mem_tuning 
Some additions to the Tuning Guide.
 2012-10-03 22:00:48 -07:00
Patrick Wendell e84c068fab Some additions to the Tuning Guide. 
1. Slight change in organization
2. Added pre-requisites
3. Made a new section about determining memory footprint
   of an RDD
4. Other small changes
 2012-10-03 14:06:34 -07:00
Matei Zaharia c04aeaf365 Merge pull request #243 from andyk/mesos-from-maven-central 
Removes the included mesos-0.9.0.jar and pulls it from Maven Central instead
 2012-10-03 11:10:00 -07:00
Reynold Xin 66d848175a Merge pull request #244 from rxin/dev 
Made Serializer and JavaSerializer non private.
 2012-10-03 11:01:48 -07:00
Reynold Xin 45f4b7cc7e Made Serializer and JavaSerializer non private. 2012-10-03 10:20:59 -07:00
Andy Konwinski 5897567679 Removes the included mesos-0.9.0.jar and adds a libraryDependency to 
the build file so that mesos-0.9.0-incubating.jar (which contains the
same class files, but has a silightly different name) will be pulled
down from Maven Central instead.
 2012-10-03 08:58:05 -07:00
Matei Zaharia 42cd148507 Merge pull request #236 from pwendell/quickstart 
A Spark "Quick Start" example
 2012-10-03 08:31:43 -07:00
Matei Zaharia 833f1d0c86 Made StorageLevel public 2012-10-03 08:27:25 -07:00
Patrick Wendell 35b767f478 Responding to Matei's comments 2012-10-02 23:54:03 -07:00
Matei Zaharia 6cf5dffc72 Make more stuff private[spark] 2012-10-02 22:28:55 -07:00
Mosharaf Chowdhury 119e50c7b9 Conflict fixed 2012-10-02 22:25:39 -07:00
Mosharaf Chowdhury ff813e4380 Merge remote-tracking branch 'upstream/dev' into dev 2012-10-02 22:17:17 -07:00
Matei Zaharia 87f4451f20 Simplify README even further 2012-10-02 22:14:40 -07:00
Matei Zaharia a17e66a689 Merge pull request #242 from pwendell/readme-update 
Changing version of Scala in README
 2012-10-02 22:12:08 -07:00
Patrick Wendell 89be3c3e76 Changing version of Scala in README 2012-10-02 22:10:23 -07:00
Matei Zaharia a3e54de6ca Merge pull request #241 from shivaram/tuning-doc 
First cut at adding documentation for GC tuning
 2012-10-02 21:26:54 -07:00
Matei Zaharia 626f701931 Merge pull request #240 from dennybritz/private_classes 
Package-Private Classes
 2012-10-02 21:24:32 -07:00
Shivaram Venkataraman 3d2b900b08 First cut at adding documentation for GC tuning 2012-10-02 20:07:18 -07:00
Denny 0361353a70 Make Java API abstract wrapped functions private 2012-10-02 20:02:53 -07:00
Denny b9badcd5bd accidentially removed trait 2012-10-02 19:35:07 -07:00
Denny 18a1faedf6 Stylistic changes and Public Accumulable and Broadcast 2012-10-02 19:28:37 -07:00
Denny b7a913e1fa Make dependency classes public - used by spark 2012-10-02 19:04:23 -07:00
Denny 4d9f4b01af Make classes package private 2012-10-02 19:00:19 -07:00
Matei Zaharia 97cbd699d7 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-02 17:31:01 -07:00
Matei Zaharia 5fda59ab99 Added a test for overly large blocks in memory store 2012-10-02 17:30:40 -07:00
Matei Zaharia 6098f7e87a Fixed cache replacement behavior of BlockManager: 
- Partitions that get dropped to disk will now be loaded back into RAM
  after they're accessed again
- Same-RDD rule for cache replacement is now implemented (don't drop
  partitions from an RDD to make room for other partitions from itself)
- Items stored as MEMORY_AND_DISK go into memory only first, instead of
  being eagerly written out to disk
- MemoryStore.ensureFreeSpace is called within a lock on the writer
  thread to prevent race conditions (this can still be optimized to
  allow multiple concurrent calls to it but it's a start)
- MemoryStore does not accept blocks larger than its limit
 2012-10-02 17:25:38 -07:00
Matei Zaharia 6112b1a83c Don't build an assembly for the REPL 2012-10-02 17:08:16 -07:00
Matei Zaharia c8ca6bc59b Merge pull request #238 from rxin/dev 
Allow whitespaces in cluster URL configuration for local cluster.
 2012-10-02 16:30:40 -07:00