Commit graph

5050 commits

Author SHA1 Message Date
Matei Zaharia 8d528af829 Merge pull request #71 from aarondav/scdefaults
Spark shell exits if it cannot create SparkContext

Mainly, this occurs if you provide a messed up MASTER url (one that doesn't match one
of our regexes). Previously, we would default to Mesos, fail, and then start the shell
anyway, except that any Spark command would fail. Simply exiting seems clearer.
2013-10-18 20:24:10 -07:00
Hossein Falaki 2d511ab320 Made SerializableHyperLogLog Externalizable and added Kryo tests 2013-10-18 15:30:45 -07:00
Hossein Falaki 13227aaa28 Added stream-lib dependency to Maven build 2013-10-18 14:10:24 -07:00
Prabeesh K 6ec39829e9 Update MQTTWordCount.scala 2013-10-18 17:00:28 +05:30
Hossein Falaki 79868fe724 Improved code style. 2013-10-17 23:39:20 -07:00
Mosharaf Chowdhury 08391dbcb8 Should compile now. 2013-10-17 23:06:17 -07:00
Hossein Falaki b611d9a65c Fixed document typo 2013-10-17 23:05:22 -07:00
Mosharaf Chowdhury 8612641362 Added an after block to reset spark.broadcast.factory 2013-10-17 22:44:04 -07:00
Hossein Falaki 654d60b6ee Added dependency on stream-lib version 2.4.0 for approximate distinct count support. 2013-10-17 22:29:10 -07:00
Hossein Falaki ec5df800fd Added countDistinctByKey to PairRDDFunctions that counts the approximate number of unique values for each key in the RDD. 2013-10-17 22:26:00 -07:00
Hossein Falaki 1a701358c0 Added a countDistinct method to RDD that takes takes an accuracy parameter and returns the (approximate) number of distinct elements in the RDD. 2013-10-17 22:24:48 -07:00
Hossein Falaki 843727af99 Added a serializable wrapper for HyperLogLog 2013-10-17 22:17:06 -07:00
Prabeesh K d223d38933 Update MQTTInputDStream.scala 2013-10-18 09:09:49 +05:30
Aaron Davidson 74737264c4 Spark shell exits if it cannot create SparkContext
Mainly, this occurs if you provide a messed up MASTER url (one that doesn't match one
of our regexes). Previously, we would default to Mesos, fail, and then start the shell
anyway, except that any Spark command would fail.
2013-10-17 18:51:19 -07:00
Mosharaf Chowdhury 90ab55fd37 Merge remote-tracking branch 'upstream/master' 2013-10-17 18:12:28 -07:00
Mosharaf Chowdhury e178ae4e9b BroadcastSuite updated to test both HttpBroadcast and TorrentBroadcast in local, local[N], local-cluster settings. 2013-10-17 16:38:43 -07:00
Matei Zaharia fc26e5b832 Merge pull request #69 from KarthikTunga/master
Fix for issue SPARK-627. Implementing --config argument in the scripts.

This code fix is for issue SPARK-627. I added code to consider --config arguments in the scripts. In case the  <conf-dir> is not a directory the scripts exit. I removed the --hosts argument. It can be achieved by giving a different config directory. Let me know if an explicit --hosts argument is required.
2013-10-17 13:21:07 -07:00
Mosharaf Chowdhury 6a84e40efe Merge remote-tracking branch 'upstream/master' 2013-10-17 13:14:33 -07:00
Mosharaf Chowdhury 35b2415fb3 Code styling. Updated doc. 2013-10-17 13:14:12 -07:00
Matei Zaharia cf64f63f8a Merge pull request #67 from kayousterhout/remove_tsl
Removed TaskSchedulerListener interface.

The interface was used only by the DAG scheduler (so it wasn't necessary
to define the additional interface), and the naming makes it very
confusing when reading the code (because "listener" was used
to describe the DAG scheduler, rather than SparkListeners, which
implement a nearly-identical interface but serve a different
function).

@mateiz - is there a reason for this interface that I'm missing?
2013-10-17 11:12:28 -07:00
Mosharaf Chowdhury e663750488 Removed unused code.
Changes to match Spark coding style.
2013-10-17 00:19:50 -07:00
Kay Ousterhout 809f547633 Fixed unit tests 2013-10-16 23:16:12 -07:00
KarthikTunga 8537f19268 SPARK-627 , Implementing --config arguments in the scripts 2013-10-16 23:00:33 -07:00
Reynold Xin 3e7df8f6c6 Added a number of very fast, memory-efficient data structures: BitSet, OpenHashSet, OpenHashMap, PrimitiveKeyOpenHashMap. 2013-10-16 22:58:52 -07:00
KarthikTunga ff4fb1f7ee SPARK-627 , Implementing --config arguments in the scripts 2013-10-16 22:55:15 -07:00
KarthikTunga a32aa6b351 Implementing --config argument in the scripts 2013-10-16 22:51:09 -07:00
Mosharaf Chowdhury e96bd0068f BroadcastTest2 --> BroadcastTest 2013-10-16 21:33:33 -07:00
Mosharaf Chowdhury a8d0981832 Fixes for the new BlockId naming convention. 2013-10-16 21:33:33 -07:00
Mosharaf Chowdhury feb45d391f Default blockSize is 4MB.
BroadcastTest2 example added for testing broadcasts.
2013-10-16 21:33:33 -07:00
Mosharaf Chowdhury 6e5a60fab4 Removed unnecessary code, and added comment of memory-latency tradeoff. 2013-10-16 21:33:33 -07:00
Mosharaf Chowdhury 4602e2bf6e Torrent-ish broadcast based on BlockManager. 2013-10-16 21:33:33 -07:00
prabeesh 890f8fe439 modify code, use Spark Logging Class 2013-10-17 10:00:40 +05:30
prabeesh ee4178f144 remove unused dependency 2013-10-17 09:57:48 +05:30
prabeesh 29245605bf remove unused dependency 2013-10-17 09:57:30 +05:30
Shivaram Venkataraman 0a4b76fcc2 Rename SBT target to assemble-deps. 2013-10-16 17:05:46 -07:00
Kay Ousterhout ec512583ab Removed TaskSchedulerListener interface.
The interface was used only by the DAG scheduler (so it wasn't necessary
to define the additional interface), and the naming makes it very
confusing when reading the code (because "listener" was used
to describe the DAG scheduler, rather than SparkListeners, which
implement a nearly-identical interface but serve a different
function).
2013-10-16 16:57:42 -07:00
Matei Zaharia f9973cae3a Merge pull request #65 from tgravescs/fixYarn
Fix yarn build

Fix the yarn build after renaming StandAloneX to CoarseGrainedX from pull request 34.
2013-10-16 15:58:41 -07:00
Shivaram Venkataraman 1dcded45e2 Exclude assembly jar from classpath if using deps 2013-10-16 13:43:41 -07:00
tgravescs cc7df2b3cc Fix yarn build 2013-10-16 10:09:16 -05:00
prabeesh 9a7575728d add maven dependencies for mqtt 2013-10-16 13:41:49 +05:30
prabeesh 7d36a117c1 add maven dependencies for mqtt 2013-10-16 13:41:26 +05:30
prabeesh 9eaf68fd40 added mqtt adapter wordcount example 2013-10-16 13:40:38 +05:30
prabeesh 06de3d516d added mqtt adapter library dependencies 2013-10-16 13:38:37 +05:30
prabeesh 2e48b23eae added mqtt adapter 2013-10-16 13:36:25 +05:30
prabeesh 742ada91e0 mqttinputdstream for mqttstreaming adapter 2013-10-16 13:35:29 +05:30
Matei Zaharia 28e9c2abc0 Merge pull request #63 from pwendell/master
Fixing spark streaming example and a bug in examples build.

- Examples assembly included a log4j.properties which clobbered Spark's
- Example had an error where some classes weren't serializable
- Did some other clean-up in this example
2013-10-15 23:59:56 -07:00
Matei Zaharia 4e46fde818 Merge pull request #62 from harveyfeng/master
Make TaskContext's stageId publicly accessible.
2013-10-15 23:14:27 -07:00
Patrick Wendell 35befe07bb Fixing spark streaming example and a bug in examples build.
- Examples assembly included a log4j.properties which clobbered Spark's
- Example had an error where some classes weren't serializable
- Did some other clean-up in this example
2013-10-15 22:55:43 -07:00
Harvey Feng 65b46236e7 Proper formatting for SparkHadoopWriter class extensions. 2013-10-15 21:51:52 -07:00
Matei Zaharia b5346064d6 Merge pull request #8 from vchekan/checkpoint-ttl-restore
Serialize and restore spark.cleaner.ttl to savepoint

In accordance to conversation in spark-dev maillist, preserve spark.cleaner.ttl parameter when serializing checkpoint.
2013-10-15 21:25:03 -07:00