Matei Zaharia
4358acfe07
Initialize Twitter4J OAuth from system properties instead of prompting
2013-06-29 15:25:06 -07:00
Matei Zaharia
1667158544
Merge remote-tracking branch 'mrpotes/master'
2013-06-29 14:36:09 -07:00
James Phillpotts
366572edca
Include a default OAuth implementation, and update examples and JavaStreamingContext
2013-06-25 22:59:34 +01:00
Tathagata Das
c89af0a7f9
Merge branch 'master' into streaming
...
Conflicts:
.gitignore
2013-06-24 23:57:47 -07:00
Tathagata Das
48c7e373c6
Minor formatting fixes
2013-06-24 23:11:04 -07:00
Tathagata Das
1249e9153b
Merge pull request #572 from Reinvigorate/sm-block-interval
...
Adding spark.streaming.blockInterval property
2013-06-24 21:46:33 -07:00
Tathagata Das
cfcda95f86
Merge pull request #571 from Reinvigorate/sm-kafka-serializers
...
Surfacing decoders on KafkaInputDStream
2013-06-24 21:44:50 -07:00
James Phillpotts
8955787a59
Twitter API v1 is retired - username/password auth no longer possible
2013-06-24 09:15:17 +01:00
James Phillpotts
93a1643405
Allow other twitter authorizations than username/password
2013-06-21 14:21:52 +01:00
Jey Kottalam
e7982c798e
Exclude old versions of Netty from Maven-based build
2013-05-18 21:24:58 -07:00
seanm
f25282def5
fixing kafkaStream Java API and adding test
2013-05-10 17:34:28 -06:00
seanm
3632980b1b
fixing indentation
2013-05-10 15:54:26 -06:00
seanm
b95c1bdbba
count() now uses a transform instead of ConstantInputDStream
2013-05-10 12:47:24 -06:00
seanm
d761e7359d
adding kafkaStream API tests
2013-05-10 12:05:10 -06:00
Reynold Xin
90577ada69
Merge branch 'shuffle-performance-fix-0.7' of github.com:shane-huang/spark into shufflemerge
...
Conflicts:
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/DiskStore.scala
project/SparkBuild.scala
2013-05-07 15:56:19 -07:00
Mridul Muralidharan
430c531464
Remove debug statements
2013-04-29 00:24:30 +05:30
Mridul Muralidharan
3a89a76b87
Make log message more descriptive to aid in debugging
2013-04-29 00:04:12 +05:30
Mridul Muralidharan
7fa6978a1e
Allow CheckpointWriter pending tasks to finish
2013-04-28 23:08:10 +05:30
Mridul Muralidharan
afee902443
Attempt to fix streaming test failures after yarn branch merge
2013-04-28 22:26:45 +05:30
Mridul Muralidharan
dd515ca3ee
Attempt at fixing merge conflict
2013-04-24 09:24:17 +05:30
seanm
7e56e99573
Surfacing decoders on KafkaInputDStream
2013-04-16 17:17:16 -06:00
seanm
ab0f834dbb
adding spark.streaming.blockInterval property
2013-04-16 11:57:05 -06:00
seanm
b42d68c8ce
fixing Spark Streaming count() so that 0 will be emitted when there is nothing to count
2013-04-15 12:54:55 -06:00
Matei Zaharia
65caa8f711
Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0'
...
Conflicts:
docs/_config.yml
project/SparkBuild.scala
2013-04-08 12:43:17 -04:00
Mridul Muralidharan
6798a09df8
Add support for building against hadoop2-yarn : adding new maven profile for it
2013-04-07 17:47:38 +05:30
shane-huang
df47b40b76
Shuffle Performance fix: Use netty embeded OIO file server instead of ConnectionManager
...
Shuffle Performance Optimization: do not send 0-byte block requests to reduce network messages
change reference from io.Source to scala.io.Source to avoid looking into io.netty package
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
2013-04-07 14:37:12 +08:00
Jey Kottalam
bc8ba222ff
Bump development version to 0.8.0
2013-03-28 15:42:01 -07:00
Jey Kottalam
b569b3f200
Move streaming test initialization into 'before' blocks
2013-03-28 15:08:41 -07:00
seanm
329ef34c2e
fixing autooffset.reset behavior when set to 'largest'
2013-03-26 23:56:15 -06:00
Holden Karau
1f5381119f
method first in trait IterableLike is deprecated: use `head' instead
2013-03-24 19:19:40 -07:00
seanm
d61978d0ab
keeping JavaStreamingContext in sync with StreamingContext + adding comments for better clarity
2013-03-15 23:36:52 -06:00
Mikhail Bautin
7fd2708eda
Add a log4j compile dependency to fix build in IntelliJ
...
Also rename parent project to spark-parent (otherwise it shows up as
"parent" in IntelliJ, which is very confusing).
2013-03-15 11:41:51 -07:00
seanm
33fa1e7e4a
removing dependency on ZookeeperConsumerConnector + purging last relic of kafka reliability that never solidified (ie- setOffsets)
2013-03-15 00:10:13 -06:00
seanm
d069283211
fixing memory leak in kafka MessageHandler
2013-03-14 23:45:33 -06:00
seanm
cfa8e769a8
KafkaInputDStream improvements. Allows more Kafka configurability
2013-03-14 23:45:19 -06:00
Stephen Haberman
0cf320485d
Forgot equals.
2013-03-12 00:05:35 -05:00
Stephen Haberman
9e68f48625
More quickly call close in HadoopRDD.
...
This also refactors out the common "gotNext" iterator pattern into
a shared utility class.
2013-03-11 23:59:17 -05:00
Mark Hamstra
b409073102
Instead of failing to bind to a fixed, already-in-use port, let the OS choose an available port for TestServer.
2013-03-01 15:05:07 -08:00
Mark Hamstra
8b06b359da
bump version to 0.7.1-SNAPSHOT in the subproject poms to keep the maven build building.
2013-02-28 23:34:34 -08:00
Matei Zaharia
4223be3aa4
Merge pull request #503 from pwendell/bug-fix
...
createNewSparkContext should use sparkHome/jars/environment.
2013-02-25 19:43:05 -08:00
Patrick Wendell
284ba90958
createNewSparkContext should use sparkHome/jars/environment.
...
This fixes a bug introduced by Matei's recent change.
2013-02-25 19:40:52 -08:00
Matei Zaharia
5d7b591cfe
Pass a code JAR to SparkContext in our examples. Fixes SPARK-594.
2013-02-25 19:34:32 -08:00
Tathagata Das
bc4a6eb850
Changed Flume test to use the same port as other tests, so that can be controlled centrally.
2013-02-25 18:04:21 -08:00
Matei Zaharia
4d480ec59e
Fixed something that was reported as a compile error in ScalaDoc.
...
For some reason, ScalaDoc complained about no such constructor for
StreamingContext; it doesn't seem like an actual Scala error but it
prevented sbt publish and from working because docs weren't built.
2013-02-25 15:53:43 -08:00
Matei Zaharia
490f056cdd
Allow passing sparkHome and JARs to StreamingContext constructor
...
Also warns if spark.cleaner.ttl is not set in the version where you pass
your own SparkContext.
2013-02-25 15:13:30 -08:00
Tathagata Das
5ab37be983
Fixed class paths and dependencies based on Matei's comments.
2013-02-24 16:24:52 -08:00
Tathagata Das
28f8b721f6
Added back the initial spark job before starting streaming receivers
2013-02-24 13:01:54 -08:00
Tathagata Das
68c7934b1a
Fixed missing dependencies in streaming/pom.xml
2013-02-24 11:51:45 -08:00
Tathagata Das
b4eb24de96
Updated streaming programming guide with Java API info, and comments from Patrick.
2013-02-23 23:59:45 -08:00
Tathagata Das
41285eaae3
Fixed differences in APIs of StreamingContext and JavaStreamingContext. Change rawNetworkStream to rawSocketStream, and added twitter, actor, zeroMQ streams to JavaStreamingContext. Also added them to JavaAPISuite.
2013-02-23 16:25:07 -08:00
