ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
24062 commits 6 branches 181 tags 397 MiB
Commit graph

24062 commits

This branch
This branch
All branches
Author SHA1 Message Date
Josh Rosen fd94e5443c Use only cPickle for serialization in Python API. 
Objects serialized with JSON can be compared for equality, but JSON can be slow
to serialize and only supports a limited range of data types.
 2012-08-21 14:01:27 -07:00
Imran Rashid 4d2efe9555 change tests to show utility of localValue 2012-08-20 15:17:31 -07:00
Matei Zaharia 25a6a39e6d Added other SparkContext constructors to JavaSparkContext 2012-08-19 18:59:16 -07:00
Josh Rosen 13b9514966 Bundle cloudpickle with pyspark. 2012-08-19 17:17:42 -07:00
Josh Rosen 886b39de55 Add Python API. 2012-08-18 22:33:51 -07:00
Imran Rashid 823878c77f add accumulators for mutable collections, with correct typing! 2012-08-17 15:52:42 -07:00
Imran Rashid 206a3833ce make accumulator.localValue public, add tests 2012-08-14 14:08:22 -07:00
Matei Zaharia 9a0c128fec Merge pull request #172 from dennybritz/dev 
Rsync root directory in EC2 script
 2012-08-14 13:05:22 -07:00
Denny 8dc7242544 Use root login in standalone AMI 2012-08-14 10:18:24 -07:00
Denny 7152c7c12d rsync root directory in EC2 script 2012-08-14 09:26:47 -07:00
Matei Zaharia 942e604c62 Merge pull request #171 from shivaram/for-size-estimator-pull 
Size estimator changes for dev
 2012-08-13 15:29:40 -07:00
Shivaram Venkataraman 0f4fbb057b Change BlockManagerSuite test cases to use a deterministic size estimator and 
update the results to match the new estimates
 2012-08-13 13:32:23 -07:00
Shivaram Venkataraman 22ba3a3f77 Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-13 13:32:10 -07:00
Shivaram Venkataraman 1f68c4b03b Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-13 13:31:54 -07:00
Shivaram Venkataraman 1ea269110c Move object size and pointer size initialization into a function to enable unit-testing 2012-08-13 13:31:45 -07:00
Shivaram Venkataraman 44661df9cc If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-13 13:31:39 -07:00
Shivaram Venkataraman 0dd8fe73ba Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-13 13:31:29 -07:00
Shivaram Venkataraman 80104ce1da Add link to Java wiki which specifies what changes with compressed oops 2012-08-13 13:31:21 -07:00
Shivaram Venkataraman 00ab5490b3 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-13 13:31:11 -07:00
Matei Zaharia 680df96c43 Merge pull request #169 from shivaram/master 
Changes to SizeEstimator more accurate
 2012-08-12 22:29:11 -07:00
Shivaram Venkataraman 2ee731211a Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-12 17:18:01 -07:00
Shivaram Venkataraman 54502238a2 Move object size and pointer size initialization into a function to enable unit-testing 2012-08-12 17:16:27 -07:00
Matei Zaharia 6ae3c375a9 Renamed apply() to call() in Java API and allowed it to throw Exceptions 2012-08-12 23:10:19 +02:00
Matei Zaharia 0141879c40 Use Promises instead of having a Future wait on a thread in 
ConnectionManager.
 2012-08-12 22:16:32 +02:00
Matei Zaharia 845a870242 Return remotely fetched blocks in a pipelined fashion from BlockManager 2012-08-12 20:01:38 +02:00
Matei Zaharia e17ed9a21d Switch to Akka futures in connection manager. 
It's still not good because each Future ends up waiting on a lock, but
it seems to work better than Scala Actors, and more importantly it
allows us to use onComplete and other listeners on futures.
 2012-08-12 19:40:37 +02:00
Matei Zaharia ad8a7612a4 Changed multi-get method in BlockManager to return an iterator 2012-08-12 19:18:01 +02:00
Shivaram Venkataraman 73452cc649 Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-11 16:42:35 -07:00
Shivaram Venkataraman 64b8fd62f0 If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-11 16:40:33 -07:00
Shivaram Venkataraman c0e773aa01 Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-11 14:38:05 -07:00
Shivaram Venkataraman f2475ca95a Add link to Java wiki which specifies what changes with compressed oops 2012-08-11 02:34:20 -07:00
Shivaram Venkataraman 980585b220 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-11 02:18:39 -07:00
Matei Zaharia 3c94e5c188 Merge pull request #168 from shivaram/dev 
Use JavaConversion to get a scala iterator
 2012-08-10 00:57:33 -07:00
Matei Zaharia e463e7a333 Merge pull request #167 from JoshRosen/piped-rdd-fixes 
Detect non-zero exit status from PipedRDD process
 2012-08-10 00:56:42 -07:00
Josh Rosen 59c22fb444 Print exit status in PipedRDD failure exception. 2012-08-10 00:33:56 -07:00
Matei Zaharia 8069bd5b41 Removed separate launcher for EC2 standalone cluster 2012-08-09 22:45:24 +02:00
Shivaram Venkataraman 1803cce692 Use an implicit conversion to get the scala iterator 2012-08-08 14:31:04 -07:00
Shivaram Venkataraman 674fcf56bf Use JavaConversion to get a scala iterator 2012-08-08 14:10:23 -07:00
Matei Zaharia bec4d362c8 Merge pull request #166 from shivaram/dev 
Avoid a copy in ShuffleMapTask
 2012-08-08 09:11:19 -07:00
Shivaram Venkataraman f4aaec7a48 Avoid a copy in ShuffleMapTask by creating an iterator that will be used by the 
block manager.
 2012-08-08 00:47:02 -07:00
Tathagata Das cae894ee7a Added new Clock interface that is used by RecurringTimer to scheduler events on system time or manually-configured time. 2012-08-06 14:52:46 -07:00
Mosharaf Chowdhury d821dd3ccc BroadcastManager is a class now (replaced Braodcast object) 2012-08-05 01:10:51 -07:00
Mosharaf Chowdhury b4804119f9 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-04 20:42:12 -07:00
Matei Zaharia 88b016db2a Merge pull request #160 from dennybritz/clusterscripts 
Standalone cluster scripts
 2012-08-04 17:45:20 -07:00
Matei Zaharia 1c5ae3edf2 Merge pull request #151 from dennybritz/fix/examples_jar 
Examples ship to to cluster
 2012-08-04 17:39:41 -07:00
Denny 8fb955fd40 Add Apache license to non-trivial scripts taken from Hadoop. 2012-08-04 17:04:33 -07:00
Denny 38d86d2616 updated readme 2012-08-04 16:58:47 -07:00
Denny 48cac4171c Renamed EXAMPLES_JAR to SPARK_EXAMPLES_JAR 2012-08-04 16:56:32 -07:00
Denny 63c2020f93 Merge branch 'master' into fix/examples_jar 2012-08-04 16:55:18 -07:00
Mosharaf Chowdhury 1b0534af8f Merge branch 'dev' into bc-bm 2012-08-04 00:30:08 -07:00