Matei Zaharia
4223be3aa4
Merge pull request #503 from pwendell/bug-fix
...
createNewSparkContext should use sparkHome/jars/environment.
2013-02-25 19:43:05 -08:00
Patrick Wendell
284ba90958
createNewSparkContext should use sparkHome/jars/environment.
...
This fixes a bug introduced by Matei's recent change.
2013-02-25 19:40:52 -08:00
Matei Zaharia
5d7b591cfe
Pass a code JAR to SparkContext in our examples. Fixes SPARK-594.
2013-02-25 19:34:32 -08:00
Matei Zaharia
7b88534932
Merge pull request #502 from tdas/master
...
Very minor change in a testcase
2013-02-25 18:11:00 -08:00
Tathagata Das
bc4a6eb850
Changed Flume test to use the same port as other tests, so that can be controlled centrally.
2013-02-25 18:04:21 -08:00
Matei Zaharia
c45d6f1248
Merge pull request #501 from tdas/master
...
Fixed bug in BlockManager and added a testcase
2013-02-25 17:35:31 -08:00
Tathagata Das
c02e064938
Fixed replication bug in BlockManager
2013-02-25 17:27:46 -08:00
Matei Zaharia
4d480ec59e
Fixed something that was reported as a compile error in ScalaDoc.
...
For some reason, ScalaDoc complained about no such constructor for
StreamingContext; it doesn't seem like an actual Scala error but it
prevented sbt publish and from working because docs weren't built.
2013-02-25 15:53:43 -08:00
Matei Zaharia
6494cab19d
Update Hadoop dependency to 1.0.4
2013-02-25 15:38:21 -08:00
Tathagata Das
6a78ef0578
Merge pull request #500 from pwendell/streaming-docs
...
Minor changes based on feedback
2013-02-25 15:28:45 -08:00
Patrick Wendell
8316534eef
meta-data
2013-02-25 15:27:04 -08:00
Patrick Wendell
918ee25867
One more change done with TD
2013-02-25 15:24:17 -08:00
Matei Zaharia
351ac5233e
Some tweaks to docs
2013-02-25 15:19:05 -08:00
Matei Zaharia
2ae15353a1
Merge branch 'master' of github.com:mesos/spark
2013-02-25 15:14:16 -08:00
Matei Zaharia
490f056cdd
Allow passing sparkHome and JARs to StreamingContext constructor
...
Also warns if spark.cleaner.ttl is not set in the version where you pass
your own SparkContext.
2013-02-25 15:13:30 -08:00
Patrick Wendell
07f2618769
Minor changes based on feedback
2013-02-25 15:09:59 -08:00
Matei Zaharia
271a40f0e4
Merge pull request #499 from pwendell/streaming-docs
...
Some changes to streaming failure docs.
2013-02-25 14:47:32 -08:00
Patrick Wendell
50ce0516e6
Some changes to streaming failure docs.
...
TD gave me the go-ahead to just make these changes:
- Define stateful dstream
- Some minor wording fixes
2013-02-25 14:38:39 -08:00
Matei Zaharia
568bdaf8ae
Set spark.deploy.spreadOut to true by default in 0.7 (improves locality)
2013-02-25 14:34:55 -08:00
Matei Zaharia
5d4a0ac794
Some tweaks to docs
2013-02-25 14:23:03 -08:00
Matei Zaharia
1ef58dadcc
Add a config property for Akka lifecycle event logging
2013-02-25 14:01:24 -08:00
Matei Zaharia
6b87ef7c86
Fix compile error
2013-02-25 14:01:16 -08:00
Matei Zaharia
01bd136ba5
Use public method sparkContext instead of protected sc in streaming examples
2013-02-25 13:27:11 -08:00
Matei Zaharia
848321f910
Change doc color scheme slightly for Spark 0.7 (to differ from 0.6)
2013-02-25 13:15:30 -08:00
Matei Zaharia
3c7dcb61ab
Use a single setting for disabling API doc build
2013-02-25 13:15:12 -08:00
Matei Zaharia
0cb8ee93cd
Merge branch 'master' of github.com:mesos/spark
2013-02-25 13:10:35 -08:00
Matei Zaharia
ceaec4a675
Merge pull request #498 from pwendell/shutup-akka
...
Disable remote lifecycle logging from Akka.
2013-02-25 12:31:24 -08:00
Patrick Wendell
85a85646d9
Disable remote lifecycle logging from Akka.
...
This changes the default setting to `off` for remote lifecycle events. When this is on, it is very chatty at the INFO level. It also prints out several ERROR messages sometimes when sc.stop() is called.
2013-02-25 12:25:43 -08:00
Matei Zaharia
25f737804a
Change tabs to spaces
2013-02-25 11:53:55 -08:00
Imran Rashid
8f17387d97
remove bogus comment
2013-02-25 10:31:06 -08:00
Matei Zaharia
6ae9a22c3e
Get spark.default.paralellism on each call to defaultPartitioner,
...
instead of only once, in case the user changes it across Spark uses
2013-02-25 10:28:08 -08:00
Matei Zaharia
d6e6abece3
Merge pull request #459 from stephenh/bettersplits
...
Change defaultPartitioner to use upstream split size.
2013-02-25 09:22:04 -08:00
Matei Zaharia
fb76250598
Merge pull request #497 from ScrapCodes/dep-resolution-fix
...
Moving akka dependency resolver to shared.
2013-02-25 00:16:52 -08:00
Prashant Sharma
254acb1666
Moving akka dependency resolver to shared.
2013-02-25 13:37:07 +05:30
Stephen Haberman
c44ccf2862
Use default parallelism if its set.
2013-02-24 23:54:03 -06:00
Stephen Haberman
44032bc476
Merge branch 'master' into bettersplits
...
Conflicts:
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/test/scala/spark/ShuffleSuite.scala
2013-02-24 22:08:14 -06:00
Matei Zaharia
22f68b7e6b
Merge pull request #494 from mesos/streaming
...
Final set of Spark Streaming changes
2013-02-24 17:48:17 -08:00
Tathagata Das
f9881b10c6
Merge pull request #495 from tdas/streaming
...
Changed class paths and dependencies based on Matei's comments
2013-02-24 16:39:04 -08:00
Tathagata Das
abb5471865
Removing duplicate doc.
2013-02-24 16:32:44 -08:00
Tathagata Das
5ab37be983
Fixed class paths and dependencies based on Matei's comments.
2013-02-24 16:24:52 -08:00
Tathagata Das
32f1a2d540
Merge pull request #493 from tdas/streaming
...
Streaming bug fixes
2013-02-24 14:46:19 -08:00
Matei Zaharia
066ce0ffa4
Merge pull request #492 from JoshRosen/pyspark-split-rename
...
Change numSplits to numPartitions in PySpark
2013-02-24 14:11:07 -08:00
Josh Rosen
2c966c98fb
Change numSplits to numPartitions in PySpark.
2013-02-24 13:25:09 -08:00
Tathagata Das
28f8b721f6
Added back the initial spark job before starting streaming receivers
2013-02-24 13:01:54 -08:00
Tathagata Das
f282bc4960
Changed Algebird from 0.1.9 to 0.1.8
2013-02-24 12:44:12 -08:00
Tathagata Das
dff53d1b94
Merge branch 'mesos-master' into streaming
2013-02-24 12:17:22 -08:00
Tathagata Das
68c7934b1a
Fixed missing dependencies in streaming/pom.xml
2013-02-24 11:51:45 -08:00
Tathagata Das
c1a040db3a
Fixed bugs in examples.
2013-02-24 11:00:30 -08:00
Tathagata Das
b4eb24de96
Updated streaming programming guide with Java API info, and comments from Patrick.
2013-02-23 23:59:45 -08:00
Matei Zaharia
3b9f929467
Merge pull request #468 from haitaoyao/master
...
support customized java options for Master, Worker, Executor, and Repl
2013-02-23 23:38:15 -08:00