Matei Zaharia
a96558caa3
Performance improvements to shuffle operations: in particular, preserve
...
RDD partitioning in more cases where it's possible, and use iterators
instead of materializing collections when doing joins.
2012-06-09 14:44:18 -07:00
Matei Zaharia
048276799a
Commit task outputs to Hadoop-supported storage systems in parallel on the
...
cluster instead of on the master. Fixes #110 .
2012-06-06 16:46:53 -07:00
Matei Zaharia
6ae2746d1e
Handle arrays that contain the same element many times better in
...
SizeEstimator. Also added a test for SizeEstimator. Fixes #136 .
2012-06-06 16:13:02 -07:00
Matei Zaharia
0a617958d1
Some refactoring to make BoundedMemoryCache test similar to others
2012-06-06 16:12:08 -07:00
Matei Zaharia
28fed4ce3b
Add System.exit(0) at the end of all the example programs.
2012-06-05 23:31:28 -07:00
Reynold Xin
1dd7d3dfff
Merge branch 'master' of github.com:mesos/spark
2012-05-30 18:41:07 -07:00
Reynold Xin
d176422586
Make spark.repl.Main.interp_ publicly accessible (so Shark can get rid
...
of a weird file dediated to accessing this variable).
2012-05-30 18:40:10 -07:00
Matei Zaharia
e141f644ca
Merge pull request #132 from Benky/rb-first-iteration
...
Little refactoring and unit tests for CacheTrackerActor
2012-05-26 13:15:06 -07:00
Matei Zaharia
69372b48f0
Merge pull request #133 from Benky/565245871f666c22aebb2c534f4fb7e947fca9f5
...
BoundedMemoryCache.put should fail when estimated size of 'value' is larger than cache capacity
2012-05-26 12:59:39 -07:00
Richard Benkovsky
ae64920337
MesosScheduler refactoring
2012-05-22 11:04:54 +02:00
Richard Benkovsky
3a1bcd4028
Added tests for CacheTrackerActor
2012-05-22 11:04:54 +02:00
Richard Benkovsky
8f2f736d53
Little refactoring
2012-05-22 11:04:54 +02:00
Richard Benkovsky
518506a7c5
Added tests for Utils.copyStream
2012-05-22 11:04:51 +02:00
Richard Benkovsky
f162fc2beb
Formating fixed
2012-05-22 09:45:38 +02:00
Richard Benkovsky
565245871f
BoundedMemoryCache.put fails when estimated size of 'value' is larger than cache capacity
2012-05-20 22:13:35 +02:00
Richard Benkovsky
822a4be37d
Utils.memoryBytesToString fixed
2012-05-19 15:13:20 +02:00
Matei Zaharia
10716b1766
Merge pull request #131 from rxin/master
...
Return size estimation, cache usage, and cache capacity from slave nodes to CacheTracker
2012-05-18 15:21:30 -07:00
Reynold Xin
d0c6e9f639
Made some RDD dependencies transient to reduce the amount of data needed
...
to be serialized in closure serialization. This can significantly reduce
the task setup time in Shark when the query involves a large number of
(Hive) partitions.
2012-05-16 14:16:55 -07:00
Reynold Xin
16461e2eda
Updated Cache's put method to use a case class for response. Previously
...
it was pretty ugly that put() should return -1 for failures.
2012-05-15 00:31:52 -07:00
Reynold Xin
019e48833f
Added the capacity to report cache usage status back to the cache
...
trackor. This is essential for building a dashboard to see the status of
caches on all slaves.
2012-05-14 18:39:04 -07:00
Matei Zaharia
f48742683a
Made caches dataset-aware so that they won't cyclically evict partitions
...
from the same dataset.
2012-05-06 20:14:40 -07:00
Matei Zaharia
32a4f4623c
Merge pull request #129 from mesos/rxin
...
Force serialize/deserialize task results in local execution mode.
2012-04-24 16:18:39 -07:00
Matei Zaharia
0b70dae2da
Merge pull request #127 from alupher/master
...
End task instead of just exiting in LocalScheduler for tasks that throw exceptions
2012-04-24 16:18:02 -07:00
Reynold Xin
761ea65a98
Added a test for the previous commit (failing to serialize task results
...
would throw an exception for local tasks).
2012-04-24 15:14:35 -07:00
Reynold Xin
9821cd4d42
Force serialize/deserialize task results in local execution mode.
2012-04-24 14:55:28 -07:00
Antonio
3e48818993
Removed commented-out System.exit call
2012-04-23 11:42:58 -07:00
Antonio
39d99168dc
Added exception handling instead of just exiting in LocalScheduler for tasks that throw exceptions
2012-04-20 14:46:43 -07:00
Matei Zaharia
f709b3ad8f
Merge pull request #124 from mesos/rxin
...
Added the ability to set environmental variables in piped rdd.
2012-04-20 12:58:26 -07:00
Reynold Xin
e601b3b9e5
Added the ability to set environmental variables in piped rdd.
2012-04-17 16:40:56 -07:00
Matei Zaharia
3b745176e0
Bug fix to pluggable closure serialization change
2012-04-12 17:53:02 +00:00
Matei Zaharia
112655f032
Merge pull request #121 from rxin/kryo-closure
...
Added an option (spark.closure.serializer) to specify the serializer for closures.
2012-04-10 14:21:02 -07:00
Reynold Xin
d295ccb43c
Added a closureSerializer field in SparkEnv and use it to serialize
...
tasks.
2012-04-10 13:29:46 -07:00
Reynold Xin
968f75f6af
Added an option (spark.closure.serializer) to specify the serializer for
...
closures. This enables using Kryo as the closure serializer.
2012-04-09 21:59:56 -07:00
Matei Zaharia
a633974143
Merge branch 'master' of github.com:mesos/spark
2012-04-08 23:41:25 -07:00
Matei Zaharia
d401e1b3e8
Fix a possible deadlock in MesosScheduler
2012-04-08 23:38:49 -07:00
Matei Zaharia
a7d6ffc5fd
Merge pull request #119 from mesos/report-cache-events
...
Report entry dropping in BoundedMemoryCache
2012-04-06 15:59:29 -07:00
Ankur Dave
7be1c7b331
Report entry dropping in BoundedMemoryCache
2012-04-06 15:49:32 -07:00
Matei Zaharia
816d4e5840
Pass local IP address instead of hostname in spark.master.host. Fixes #117 .
2012-04-05 14:53:17 -07:00
Matei Zaharia
335a6036ad
Converted some tabs to spaces
2012-04-05 11:58:01 -07:00
Matei Zaharia
8c95a85438
Use Runtime.maxMemory instead of Runtime.totalMemory in
...
BoundedMemoryCache, in case the JVM was not started with its initial
heap size equaling its maximum one (-Xms == -Xmx).
2012-03-30 13:39:35 -04:00
Reynold Xin
d46f662c5b
Merge branch 'master' of github.com:mesos/spark
2012-03-29 15:22:17 -07:00
Reynold Xin
42dcdbcb2f
Removed the extra spaces in OrderedRDDFunctions and SortedRDD.
2012-03-29 15:21:57 -07:00
Matei Zaharia
ca5c19c1ba
Remove dependency on Akka
2012-03-29 01:03:34 -04:00
Reynold Xin
90418b70ff
Added sbt-assembly for spark-repl project so we can generate an
...
assembled jar for Shark.
2012-03-22 18:46:31 -07:00
Matei Zaharia
c7af538ac1
Some fixes to sorting for when the RDD has fewer elements than the
...
number of partitions we ask to partition it into. Also, removed a test
that was taking way too long to run.
2012-03-17 13:08:36 -07:00
Matei Zaharia
a5e2b6a6bd
Merge pull request #112 from cengle/master
...
Changed HadoopRDD to get key and value containers from the RecordReader instead of through reflection
2012-03-06 13:38:32 -08:00
Matei Zaharia
97eee50825
Fixes a nasty bug that could happen when tasks fail, because calling
...
wait() with a timeout of 0 on a Java object means "wait forever".
2012-03-01 13:43:17 -08:00
Cliff Engle
dd68cb6099
Get key and value container from RecordReader
2012-02-29 16:33:23 -08:00
Matei Zaharia
1e10df0a46
Merge pull request #111 from alupher/master
...
Adding sorting to RDDs
2012-02-24 15:50:14 -08:00
Antonio
0d93d95bcf
Removed unnecessary import
2012-02-21 19:57:12 -08:00