Shivaram Venkataraman
674fcf56bf
Use JavaConversion to get a scala iterator
2012-08-08 14:10:23 -07:00
Matei Zaharia
bec4d362c8
Merge pull request #166 from shivaram/dev
...
Avoid a copy in ShuffleMapTask
2012-08-08 09:11:19 -07:00
Shivaram Venkataraman
f4aaec7a48
Avoid a copy in ShuffleMapTask by creating an iterator that will be used by the
...
block manager.
2012-08-08 00:47:02 -07:00
Tathagata Das
cae894ee7a
Added new Clock interface that is used by RecurringTimer to scheduler events on system time or manually-configured time.
2012-08-06 14:52:46 -07:00
Mosharaf Chowdhury
d821dd3ccc
BroadcastManager is a class now (replaced Braodcast object)
2012-08-05 01:10:51 -07:00
Mosharaf Chowdhury
b4804119f9
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-04 20:42:12 -07:00
Matei Zaharia
88b016db2a
Merge pull request #160 from dennybritz/clusterscripts
...
Standalone cluster scripts
2012-08-04 17:45:20 -07:00
Matei Zaharia
1c5ae3edf2
Merge pull request #151 from dennybritz/fix/examples_jar
...
Examples ship to to cluster
2012-08-04 17:39:41 -07:00
Denny
8fb955fd40
Add Apache license to non-trivial scripts taken from Hadoop.
2012-08-04 17:04:33 -07:00
Denny
38d86d2616
updated readme
2012-08-04 16:58:47 -07:00
Denny
48cac4171c
Renamed EXAMPLES_JAR to SPARK_EXAMPLES_JAR
2012-08-04 16:56:32 -07:00
Denny
63c2020f93
Merge branch 'master' into fix/examples_jar
2012-08-04 16:55:18 -07:00
Mosharaf Chowdhury
1b0534af8f
Merge branch 'dev' into bc-bm
2012-08-04 00:30:08 -07:00
Mosharaf Chowdhury
d11b457e67
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-04 00:28:10 -07:00
Mosharaf Chowdhury
24b7eb872c
Bug fixed. Broadcast now works with BlockManager.
2012-08-04 00:27:28 -07:00
Matei Zaharia
5cefda9984
Merge pull request #165 from shivaram/dev
...
Fix test checkpoint to reuse spark context defined in the class
2012-08-03 19:17:50 -07:00
Shivaram Venkataraman
ce3444d2cb
Fix testcheckpoint to reuse spark context defined in the class
2012-08-03 18:52:26 -07:00
Matei Zaharia
62898b631f
Made range partition balance tests more aggressive.
...
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
2012-08-03 16:46:48 -04:00
Matei Zaharia
abca699378
Made range partition balance tests more aggressive.
...
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
2012-08-03 16:44:17 -04:00
Matei Zaharia
6601a6212b
Added a unit test for cross-partition balancing in sort, and changes to
...
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
2012-08-03 16:40:45 -04:00
Harvey
1170de3757
Fix for partitioning when sorting in descending order
2012-08-03 16:40:38 -04:00
Paul Cavallaro
d05c0f97ca
Logging Throwables in Info and Debug
...
Logging Throwables in logInfo and logDebug instead of swallowing them.
Conflicts:
core/src/main/scala/spark/Logging.scala
2012-08-03 16:40:21 -04:00
Matei Zaharia
6da2bcdba1
Added a unit test for cross-partition balancing in sort, and changes to
...
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
2012-08-03 16:37:35 -04:00
Matei Zaharia
508221b8e6
Fix to #154 (CacheTracker trying to cast a broadcast variable's ID to int)
2012-08-03 15:57:43 -04:00
Matei Zaharia
c0d5bd6553
Merge pull request #164 from HarveyFeng/master
...
Bug fix in RangePartitioner for partitioning when sorting in descending order.
2012-08-03 12:27:17 -07:00
Harvey
5ec13327d4
Fix for partitioning when sorting in descending order
2012-08-03 12:22:07 -07:00
Denny
c90c9ec208
Read config variables before to get the master port
2012-08-02 16:12:40 -07:00
Denny
0008994044
merged dev branch
2012-08-02 16:00:33 -07:00
Denny
53008c2d8a
Settings variables and bugfix for stop script.
2012-08-02 15:59:39 -07:00
Denny
aaed039e36
Merged standalone and mesos EC2 scripts
2012-08-02 15:23:52 -07:00
Matei Zaharia
71a958b0b7
Merge branch 'dev' of github.com:mesos/spark into dev
...
Conflicts:
project/SparkBuild.scala
2012-08-02 17:23:13 -04:00
Denny
7312a5c30f
Use spray's implicit Marshaller for Futures.
2012-08-02 14:11:27 -07:00
Denny
ba7e30fb5e
Mostly stlyistic changes.
2012-08-02 13:55:09 -07:00
Matei Zaharia
b8fe672399
Merge pull request #162 from shivaram/dev
...
Use maxMemory to better estimate memory available for BlockManager cache
2012-08-02 12:33:23 -07:00
Shivaram Venkataraman
1a07bb9ba4
Avoid an extra partition copy by passing an iterator to blockManager.put
2012-08-02 12:22:33 -07:00
Shivaram Venkataraman
6790908b11
Use maxMemory to better estimate memory available for BlockManager cache
2012-08-02 12:05:05 -07:00
Matei Zaharia
43b81eb271
Renamed RDS to DStream, plus minor style fixes
2012-08-02 14:05:51 -04:00
Matei Zaharia
b980eabd86
Merge pull request #161 from JoshRosen/fix/assembly-akka-config
...
Use sbt mergeStrategy for reference.conf files.
2012-08-02 10:37:50 -07:00
Josh Rosen
039b41cb54
Use sbt mergeStrategy for reference.conf files.
...
Cleans up #158 / 509b721
2012-08-02 10:21:50 -07:00
Denny
863c31b7c1
Moved resources into static folder
2012-08-02 09:48:36 -07:00
Matei Zaharia
29bf44473c
Added an RDS that repeatedly returns the same input
2012-08-02 11:43:04 -04:00
Matei Zaharia
650d11817e
Added a WordCount for external data and fixed bugs in file streams
2012-08-02 11:09:43 -04:00
Matei Zaharia
4a9c58913d
Merge pull request #157 from paulcavallaro/log-throwables
...
Logging Throwables in Info and Debug
2012-08-02 06:26:06 -07:00
Tathagata Das
ed897ac5e1
Moved streaming files not immediately necessary to spark.streaming.util.
2012-08-01 22:28:54 -07:00
Tathagata Das
1c0aeee960
Merge branch 'dev' of github.com:radlab/spark into dev
2012-08-01 22:11:41 -07:00
Tathagata Das
3be54c2a8a
1. Refactored SparkStreamContext, Scheduler, InputRDS, FileInputRDS and a few other files.
...
2. Modified Time class to represent milliseconds (long) directly, instead of LongTime.
3. Added new files QueueInputRDS, RecurringTimer, etc.
4. Added RDDSuite as the skeleton for testcases.
5. Added two examples in spark.streaming.examples.
6. Removed all past examples and a few unnecessary files. Moved a number of files to spark.streaming.util.
2012-08-01 22:09:27 -07:00
Denny
0ee44c225e
Spark standalone mode cluster scripts.
...
Heavily inspired by Hadoop cluster scripts ;-)
2012-08-01 20:38:52 -07:00
Denny
6c670c37dd
Webui improvements.
2012-08-01 19:47:57 -07:00
Denny
1b29e90a79
merge dev branch
2012-08-01 14:06:09 -07:00
Denny
011220fa55
Compact job page.
2012-08-01 11:26:45 -07:00
