Patrick Wendell
5fecd2516d
Merge pull request #441 from pwendell/graphx-build
...
GraphX shouldn't list Spark as provided.
I noticed this when building an application against GraphX to audit the released artifacts.
2014-01-15 11:15:07 -08:00
Patrick Wendell
00a3f7eec5
Workers should use working directory as spark home if it's not specified
2014-01-15 11:05:36 -08:00
Patrick Wendell
9259d706be
GraphX shouldn't list Spark as provided
2014-01-15 10:46:37 -08:00
Patrick Wendell
494d3c0774
Merge pull request #433 from markhamstra/debFix
...
Updated Debian packaging
2014-01-15 10:00:50 -08:00
Thomas Graves
cef2af9c7d
Merge pull request #366 from colorant/yarn-dev
...
More yarn code refactor
Try to retrive common code in yarn alpha/stable for client and workerRunnable to reduce duplicated codes. By put them into a trait in common dir and extends with them.
Same works could be done for the remaining files in alpha/stable , while the remainning files have much more overlapping codes with different API call here and there within functions, and will need much more close review , aslo it might divide functions into too small trifle ones, thus might not deserve to be done in this way.
So just make it run for these two files firstly.
2014-01-15 10:06:17 -06:00
CrazyJvm
263933da97
remove "-XX:+UseCompressedStrings" option
...
remove "-XX:+UseCompressedStrings" option from tuning guide since jdk7 no longer supports this.
2014-01-15 22:26:15 +08:00
Reynold Xin
3d9e66d92a
Merge pull request #436 from ankurdave/VertexId-case
...
Rename VertexID -> VertexId in GraphX
2014-01-14 23:17:05 -08:00
Mridul Muralidharan
0aea33d39e
Expose method and class - so that we can use it from user code (particularly since checkpoint directory is autogenerated now
2014-01-15 12:44:44 +05:30
Patrick Wendell
139c24ef08
Merge pull request #435 from tdas/filestream-fix
...
Fixed the flaky tests by making SparkConf not serializable
SparkConf was being serialized with CoGroupedRDD and Aggregator, which somehow caused OptionalJavaException while being deserialized as part of a ShuffleMapTask. SparkConf should not even be serializable (according to conversation with Matei). This change fixes that.
@mateiz @pwendell
2014-01-14 23:07:55 -08:00
Patrick Wendell
087487e90e
Merge pull request #434 from rxin/graphxmaven
...
Fixed SVDPlusPlusSuite in Maven build.
This should go into 0.9.0 also.
2014-01-14 22:50:36 -08:00
Tathagata Das
0e15bd7827
Merge remote-tracking branch 'apache/master' into filestream-fix
2014-01-14 22:21:20 -08:00
Tathagata Das
1f4718c480
Changed SparkConf to not be serializable. And also fixed unit-test log paths in log4j.properties of external modules.
2014-01-14 22:20:14 -08:00
Reynold Xin
dfb152446d
Fixed SVDPlusPlusSuite in Maven build.
2014-01-14 22:18:43 -08:00
Mark Hamstra
147a943df0
Removed repl-bin and updated maven build doc.
2014-01-14 22:17:24 -08:00
Ankur Dave
f4d9019aa8
VertexID -> VertexId
2014-01-14 22:17:18 -08:00
Mark Hamstra
148757e88c
Add deb profile to assembly/pom.xml
2014-01-14 22:05:42 -08:00
Reynold Xin
3a386e2389
Merge pull request #424 from jegonzal/GraphXProgrammingGuide
...
Additional edits for clarity in the graphx programming guide.
Added an overview of the Graph and GraphOps functions and fixed numerous typos.
2014-01-14 21:52:50 -08:00
Reynold Xin
ad294db326
Merge pull request #431 from ankurdave/graphx-caching-doc
...
Describe caching and uncaching in GraphX programming guide
2014-01-14 21:51:06 -08:00
Ankur Dave
1210ec2945
Describe GraphX caching and uncaching in guide
2014-01-14 17:25:38 -08:00
Reynold Xin
74b46acdc5
Merge pull request #428 from pwendell/writeable-objects
...
Don't clone records for text files
2014-01-14 14:59:13 -08:00
Reynold Xin
193a0757c8
Merge pull request #429 from ankurdave/graphx-examples-pom.xml
...
Add GraphX dependency to examples/pom.xml
2014-01-14 14:53:24 -08:00
Reynold Xin
d601a76d1f
Merge pull request #427 from pwendell/deprecate-aggregator
...
Deprecate rather than remove old combineValuesByKey function
2014-01-14 14:52:24 -08:00
Ankur Dave
8ea056d721
Add GraphX dependency to examples/pom.xml
2014-01-14 13:58:48 -08:00
Patrick Wendell
b1b22b7a13
Style fix
2014-01-14 13:56:27 -08:00
Patrick Wendell
8ea2cd56e4
Adding fix covering combineCombinersByKey as well
2014-01-14 13:52:23 -08:00
Reynold Xin
2ce23a55a3
Merge pull request #425 from rxin/scaladoc
...
API doc update & make Broadcast public
In #413 Broadcast was mistakenly made private[spark]. I changed it to public again. Also exposing id in public given the R frontend requires that.
Copied some of the documentation from the programming guide to API Doc for Broadcast and Accumulator.
This should be cherry picked into branch-0.9 as well for 0.9.0 release.
2014-01-14 13:28:44 -08:00
Matei Zaharia
5b3a3e28d7
Complain if Python and NumPy versions are too old for MLlib
2014-01-14 12:27:58 -08:00
Patrick Wendell
b683608c9f
Deprecate rather than remove old combineValuesByKey function
2014-01-14 12:15:10 -08:00
Matei Zaharia
938e4a0e16
Re-enable Python MLlib tests (require Python 2.7 and NumPy 1.7+)
2014-01-14 12:14:48 -08:00
Patrick Wendell
6f965a46a9
Don't clone records for text files
2014-01-14 11:57:53 -08:00
Reynold Xin
f12e506c9e
Fixed a typo in JavaSparkContext's API doc.
2014-01-14 11:42:28 -08:00
Reynold Xin
1b5623fd0b
Maintain Serializable API compatibility by reverting back to java.io.Serializable for Broadcast and Accumulator.
2014-01-14 11:30:59 -08:00
Reynold Xin
55db77416b
Added license header for package.scala in the Java API package.
2014-01-14 11:20:12 -08:00
Reynold Xin
f8c12e9457
Added package doc for the Java API.
2014-01-14 11:16:25 -08:00
Reynold Xin
6a12b9ebc5
Updated API doc for Accumulable and Accumulator.
2014-01-14 11:16:08 -08:00
Reynold Xin
71b3007dbd
Broadcast variable visibility change & doc update.
...
Note that previously Broadcast class was accidentally marked as private[spark]. It needs to be public
for broadcast variables to work. Also exposing the broadcast varaible id.
2014-01-14 11:15:21 -08:00
Joseph E. Gonzalez
0bba7738a2
Additional edits for clarity in the graphx programming guide.
2014-01-14 10:31:54 -08:00
Reynold Xin
3fcc68bfa5
Merge pull request #423 from jegonzal/GraphXProgrammingGuide
...
Improving the graphx-programming-guide
This PR will track a few minor improvements to the content and formatting of the graphx-programming-guide.
2014-01-14 09:44:43 -08:00
Joseph E. Gonzalez
486f37c59c
Improving the graphx-programming-guide.
2014-01-14 09:43:33 -08:00
Frank Dai
57fcfc75b3
Added parentheses for that getDouble() also has side effect
2014-01-14 18:56:11 +08:00
Patrick Wendell
fa75e5e1c5
Merge pull request #420 from pwendell/header-files
...
Add missing header files
2014-01-14 01:18:34 -08:00
Patrick Wendell
23034798d7
Add missing header files
2014-01-14 01:17:13 -08:00
Saurabh Rawat
1442cd5d50
Modifications as suggested in PR feedback-
...
- more variants of mapPartitions added to JavaRDDLike
- move setGenerator to JavaRDDLike
- clean up
2014-01-14 14:19:02 +05:30
Patrick Wendell
980250b1ee
Merge pull request #416 from tdas/filestream-fix
...
Removed unnecessary DStream operations and updated docs
Removed StreamingContext.registerInputStream and registerOutputStream - they were useless. InputDStream has been made to register itself, and just registering a DStream as output stream cause RDD objects to be created but the RDDs will not be computed at all.. Also made DStream.register() private[streaming] for the same reasons.
Updated docs, specially added package documentation for streaming package.
Also, changed NetworkWordCount's input storage level to use MEMORY_ONLY, replication on the local machine causes warning messages (as replication fails) which is scary for a new user trying out his/her first example.
2014-01-14 00:05:37 -08:00
Tathagata Das
f8bd828c7c
Fixed loose ends in docs.
2014-01-14 00:03:46 -08:00
Tathagata Das
f8e239e058
Merge remote-tracking branch 'apache/master' into filestream-fix
...
Conflicts:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
2014-01-13 23:57:27 -08:00
Reza Zadeh
845e568fad
Merge remote-tracking branch 'upstream/master' into sparsesvd
2014-01-13 23:52:34 -08:00
Frank Dai
a3da468d8b
Merge remote-tracking branch 'upstream/master' into code-style
2014-01-14 15:29:17 +08:00
Patrick Wendell
055be5c694
Merge pull request #415 from pwendell/shuffle-compress
...
Enable compression by default for spills
2014-01-13 23:26:44 -08:00
Patrick Wendell
0984647aae
Enable compression by default for spills
2014-01-13 23:25:25 -08:00
