ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
20241 commits 6 branches 181 tags 397 MiB
Commit graph

20241 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia a3bf0ce57f Don't crash on ask timeout exceptions in deploy.Client.stop() (fixes a crash in tests) 2012-10-07 07:25:41 +00:00
Matei Zaharia eca570f66a Removed the need to sleep in tests due to waiting for Akka to shut down 2012-10-07 00:17:59 -07:00
Josh Rosen 4f72066a9a Document the Dependency classes. 2012-10-07 00:05:37 -07:00
Josh Rosen 3f2571fe98 Remove unused isShuffle field from Dependency. 2012-10-07 00:03:55 -07:00
Matei Zaharia b2fc3dd902 Log message 2012-10-07 06:43:52 +00:00
Matei Zaharia ea096f7cd5 More logging 2012-10-07 06:35:48 +00:00
root 554b42cb24 Log more info in MapOutputTracker 2012-10-07 05:02:18 +00:00
root a73b25826b Made Akka thread pool and message batch sizes configurable 2012-10-07 04:19:54 +00:00
root ce915cadee Made run script add test-classes onto the classpath only if SPARK_TESTING is set; fixes #216 2012-10-07 04:19:16 +00:00
root 975009d688 Avoid acquiring locks in BlockManager when fetching shuffle outputs 2012-10-07 04:02:10 +00:00
root 0bc63f7ef1 Log initial number of fetches in reducer 2012-10-07 03:51:04 +00:00
Matei Zaharia dc28a3ac0a Modified shuffle to limit the maximum outstanding data size in bytes, 
instead of the maximum number of outstanding fetches. This should make
it faster when there are many small map output files, as well as more
robust to overallocating memory on large map outputs.
 2012-10-06 20:07:10 -07:00
Matei Zaharia 9a3b3f32a3 Pass sizes of map outputs back to MapOutputTracker 2012-10-06 18:46:04 -07:00
Matei Zaharia 0e42832e6a Made block store return the size of each block put in 2012-10-06 18:00:53 -07:00
Matei Zaharia b0110de5b6 Warn about user programs that try to set spark.cache.class 2012-10-06 17:27:14 -07:00
Matei Zaharia 65113b7e1b Only group elements ten at a time into SequenceFile records in 
saveAsObjectFile
 2012-10-06 17:14:41 -07:00
Matei Zaharia dbf1f3dd5b Make reduce logging less verbose 2012-10-06 17:10:09 -07:00
Matei Zaharia 716e10ca32 Minor formatting fixes 2012-10-05 22:03:06 -07:00
Matei Zaharia 70f02fa912 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-05 22:00:22 -07:00
Matei Zaharia 95ef307ef5 Merge pull request #249 from andyk/move-rdds-to-their-own-package 
Move RDD classes/files to their own package/directory
 2012-10-05 21:59:57 -07:00
Andy Konwinski a242cdd0a6 Factor subclasses of RDD out of RDD.scala into their own classes 
in the rdd package.
 2012-10-05 19:53:54 -07:00
Andy Konwinski d7363a6b8a Moves all files in core/src/main/scala/ that have RDD in their name 
from that directory to a new core/src/main/scala/rdd directory.
 2012-10-05 19:23:45 -07:00
Andy Konwinski e0067da082 Moves all files in core/src/main/scala/ that have RDD in them from 
package spark to package spark.rdd and updates all references to them.
 2012-10-05 19:23:45 -07:00
Matei Zaharia 69588baf65 Cleaning up code slightly 2012-10-05 19:16:09 -07:00
root f52bc09a34 Reduce some overly aggressive logging in connection manager 2012-10-06 01:54:39 +00:00
Matei Zaharia e6e27a05d8 Links quick start from nav bar 2012-10-05 17:06:55 -07:00
Shivaram Venkataraman b6e4f46a96 Fix SizeEstimator tests to work with String classes in JDK 6 and 7 
Conflicts:

	core/src/test/scala/spark/BoundedMemoryCacheSuite.scala
 2012-10-05 16:58:57 -07:00
Matei Zaharia 7eae2d1229 Merge branch 'master' into reduce-logging 2012-10-05 16:36:16 -07:00
Matei Zaharia d9bcc2a6e3 Merge branch 'master' of github.com:mesos/spark 2012-10-05 16:35:48 -07:00
Matei Zaharia 1620d59ea4 Merge pull request #246 from shivaram/size-estimator-fix-master 
Fix SizeEstimator tests to work with String classes in JDK 6 and 7
 2012-10-05 10:28:30 -07:00
Matei Zaharia e3ae98b54e Merge pull request #247 from squito/dev 
Dev
 2012-10-05 10:27:18 -07:00
Imran Rashid e0698f8f26 change tests to show utility of localValue 2012-10-04 23:05:42 -07:00
Imran Rashid 82a3327862 make accumulator.localValue public, add tests 
Conflicts:
	core/src/test/scala/spark/AccumulatorSuite.scala
 2012-10-04 23:05:01 -07:00
Matei Zaharia 8c82f43db3 Scaladoc documentation for some core Spark functionality 2012-10-04 22:59:36 -07:00
Matei Zaharia 08edb04f47 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-04 22:11:32 -07:00
Shivaram Venkataraman 5975d2ee3b Fix SizeEstimator tests to work with String classes in JDK 6 and 7 2012-10-04 19:42:57 -07:00
Matei Zaharia 5a7b370225 Only group elements ten at a time into SequenceFile records in 
saveAsObjectFile
 2012-10-04 16:49:30 -07:00
Matei Zaharia 66d7066d4f Let the reducer retry if a fetch fails before reading all records 2012-10-04 16:41:17 -07:00
Matei Zaharia c535762deb Don't check for JARs in core/lib anymore 2012-10-04 15:11:43 -07:00
Matei Zaharia 588120cd71 Add more logging for number of records fetched by each reduce 2012-10-04 11:54:47 -07:00
Matei Zaharia d6d071f19a Merge pull request #245 from pwendell/mem_tuning 
Some additions to the Tuning Guide.
 2012-10-03 22:00:48 -07:00
Patrick Wendell e84c068fab Some additions to the Tuning Guide. 
1. Slight change in organization
2. Added pre-requisites
3. Made a new section about determining memory footprint
   of an RDD
4. Other small changes
 2012-10-03 14:06:34 -07:00
Matei Zaharia c04aeaf365 Merge pull request #243 from andyk/mesos-from-maven-central 
Removes the included mesos-0.9.0.jar and pulls it from Maven Central instead
 2012-10-03 11:10:00 -07:00
Reynold Xin 66d848175a Merge pull request #244 from rxin/dev 
Made Serializer and JavaSerializer non private.
 2012-10-03 11:01:48 -07:00
Reynold Xin 45f4b7cc7e Made Serializer and JavaSerializer non private. 2012-10-03 10:20:59 -07:00
Andy Konwinski 5897567679 Removes the included mesos-0.9.0.jar and adds a libraryDependency to 
the build file so that mesos-0.9.0-incubating.jar (which contains the
same class files, but has a silightly different name) will be pulled
down from Maven Central instead.
 2012-10-03 08:58:05 -07:00
Matei Zaharia 42cd148507 Merge pull request #236 from pwendell/quickstart 
A Spark "Quick Start" example
 2012-10-03 08:31:43 -07:00
Matei Zaharia 833f1d0c86 Made StorageLevel public 2012-10-03 08:27:25 -07:00
Patrick Wendell 35b767f478 Responding to Matei's comments 2012-10-02 23:54:03 -07:00
Matei Zaharia 6cf5dffc72 Make more stuff private[spark] 2012-10-02 22:28:55 -07:00