Imran Rashid
4d2efe9555
change tests to show utility of localValue
2012-08-20 15:17:31 -07:00
Matei Zaharia
25a6a39e6d
Added other SparkContext constructors to JavaSparkContext
2012-08-19 18:59:16 -07:00
Josh Rosen
13b9514966
Bundle cloudpickle with pyspark.
2012-08-19 17:17:42 -07:00
Josh Rosen
886b39de55
Add Python API.
2012-08-18 22:33:51 -07:00
Imran Rashid
823878c77f
add accumulators for mutable collections, with correct typing!
2012-08-17 15:52:42 -07:00
Imran Rashid
206a3833ce
make accumulator.localValue public, add tests
2012-08-14 14:08:22 -07:00
Matei Zaharia
9a0c128fec
Merge pull request #172 from dennybritz/dev
...
Rsync root directory in EC2 script
2012-08-14 13:05:22 -07:00
Denny
8dc7242544
Use root login in standalone AMI
2012-08-14 10:18:24 -07:00
Denny
7152c7c12d
rsync root directory in EC2 script
2012-08-14 09:26:47 -07:00
Matei Zaharia
942e604c62
Merge pull request #171 from shivaram/for-size-estimator-pull
...
Size estimator changes for dev
2012-08-13 15:29:40 -07:00
Shivaram Venkataraman
0f4fbb057b
Change BlockManagerSuite test cases to use a deterministic size estimator and
...
update the results to match the new estimates
2012-08-13 13:32:23 -07:00
Shivaram Venkataraman
22ba3a3f77
Add test-cases for 32-bit and no-compressed oops scenarios.
2012-08-13 13:32:10 -07:00
Shivaram Venkataraman
1f68c4b03b
Update test cases to match the new size estimates. Uses 64-bit and compressed
...
oops setting to get deterministic results
2012-08-13 13:31:54 -07:00
Shivaram Venkataraman
1ea269110c
Move object size and pointer size initialization into a function to enable unit-testing
2012-08-13 13:31:45 -07:00
Shivaram Venkataraman
44661df9cc
If spark.test.useCompressedOops is set, use that to infer compressed oops
...
setting. This is useful to get a deterministic test case
2012-08-13 13:31:39 -07:00
Shivaram Venkataraman
0dd8fe73ba
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
2012-08-13 13:31:29 -07:00
Shivaram Venkataraman
80104ce1da
Add link to Java wiki which specifies what changes with compressed oops
2012-08-13 13:31:21 -07:00
Shivaram Venkataraman
00ab5490b3
Changes to make size estimator more accurate. Fixes object size, pointer size
...
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
2012-08-13 13:31:11 -07:00
Matei Zaharia
680df96c43
Merge pull request #169 from shivaram/master
...
Changes to SizeEstimator more accurate
2012-08-12 22:29:11 -07:00
Shivaram Venkataraman
2ee731211a
Add test-cases for 32-bit and no-compressed oops scenarios.
2012-08-12 17:18:01 -07:00
Shivaram Venkataraman
54502238a2
Move object size and pointer size initialization into a function to enable unit-testing
2012-08-12 17:16:27 -07:00
Matei Zaharia
6ae3c375a9
Renamed apply() to call() in Java API and allowed it to throw Exceptions
2012-08-12 23:10:19 +02:00
Matei Zaharia
0141879c40
Use Promises instead of having a Future wait on a thread in
...
ConnectionManager.
2012-08-12 22:16:32 +02:00
Matei Zaharia
845a870242
Return remotely fetched blocks in a pipelined fashion from BlockManager
2012-08-12 20:01:38 +02:00
Matei Zaharia
e17ed9a21d
Switch to Akka futures in connection manager.
...
It's still not good because each Future ends up waiting on a lock, but
it seems to work better than Scala Actors, and more importantly it
allows us to use onComplete and other listeners on futures.
2012-08-12 19:40:37 +02:00
Matei Zaharia
ad8a7612a4
Changed multi-get method in BlockManager to return an iterator
2012-08-12 19:18:01 +02:00
Shivaram Venkataraman
73452cc649
Update test cases to match the new size estimates. Uses 64-bit and compressed
...
oops setting to get deterministic results
2012-08-11 16:42:35 -07:00
Shivaram Venkataraman
64b8fd62f0
If spark.test.useCompressedOops is set, use that to infer compressed oops
...
setting. This is useful to get a deterministic test case
2012-08-11 16:40:33 -07:00
Shivaram Venkataraman
c0e773aa01
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
2012-08-11 14:38:05 -07:00
Shivaram Venkataraman
f2475ca95a
Add link to Java wiki which specifies what changes with compressed oops
2012-08-11 02:34:20 -07:00
Shivaram Venkataraman
980585b220
Changes to make size estimator more accurate. Fixes object size, pointer size
...
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
2012-08-11 02:18:39 -07:00
Matei Zaharia
3c94e5c188
Merge pull request #168 from shivaram/dev
...
Use JavaConversion to get a scala iterator
2012-08-10 00:57:33 -07:00
Matei Zaharia
e463e7a333
Merge pull request #167 from JoshRosen/piped-rdd-fixes
...
Detect non-zero exit status from PipedRDD process
2012-08-10 00:56:42 -07:00
Josh Rosen
59c22fb444
Print exit status in PipedRDD failure exception.
2012-08-10 00:33:56 -07:00
Matei Zaharia
8069bd5b41
Removed separate launcher for EC2 standalone cluster
2012-08-09 22:45:24 +02:00
Shivaram Venkataraman
1803cce692
Use an implicit conversion to get the scala iterator
2012-08-08 14:31:04 -07:00
Shivaram Venkataraman
674fcf56bf
Use JavaConversion to get a scala iterator
2012-08-08 14:10:23 -07:00
Matei Zaharia
bec4d362c8
Merge pull request #166 from shivaram/dev
...
Avoid a copy in ShuffleMapTask
2012-08-08 09:11:19 -07:00
Shivaram Venkataraman
f4aaec7a48
Avoid a copy in ShuffleMapTask by creating an iterator that will be used by the
...
block manager.
2012-08-08 00:47:02 -07:00
Tathagata Das
cae894ee7a
Added new Clock interface that is used by RecurringTimer to scheduler events on system time or manually-configured time.
2012-08-06 14:52:46 -07:00
Mosharaf Chowdhury
d821dd3ccc
BroadcastManager is a class now (replaced Braodcast object)
2012-08-05 01:10:51 -07:00
Mosharaf Chowdhury
b4804119f9
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-04 20:42:12 -07:00
Matei Zaharia
88b016db2a
Merge pull request #160 from dennybritz/clusterscripts
...
Standalone cluster scripts
2012-08-04 17:45:20 -07:00
Matei Zaharia
1c5ae3edf2
Merge pull request #151 from dennybritz/fix/examples_jar
...
Examples ship to to cluster
2012-08-04 17:39:41 -07:00
Denny
8fb955fd40
Add Apache license to non-trivial scripts taken from Hadoop.
2012-08-04 17:04:33 -07:00
Denny
38d86d2616
updated readme
2012-08-04 16:58:47 -07:00
Denny
48cac4171c
Renamed EXAMPLES_JAR to SPARK_EXAMPLES_JAR
2012-08-04 16:56:32 -07:00
Denny
63c2020f93
Merge branch 'master' into fix/examples_jar
2012-08-04 16:55:18 -07:00
Mosharaf Chowdhury
1b0534af8f
Merge branch 'dev' into bc-bm
2012-08-04 00:30:08 -07:00
Mosharaf Chowdhury
d11b457e67
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-04 00:28:10 -07:00