Commit graph

332 commits

Author SHA1 Message Date
Sandy Ryza a99fb3747a SPARK-1193. Fix indentation in pom.xmls
Author: Sandy Ryza <sandy@cloudera.com>

Closes #91 from sryza/sandy-spark-1193 and squashes the following commits:

a878124 [Sandy Ryza] SPARK-1193. Fix indentation in pom.xmls
2014-03-07 23:10:35 -08:00
Patrick Wendell c3f5e07533 SPARK-1121: Include avro for yarn-alpha builds
This lets us explicitly include Avro based on a profile for 0.23.X
builds. It makes me sad how convoluted it is to express this logic
in Maven. @tgraves and @sryza curious if this works for you.

I'm also considering just reverting to how it was before. The only
real problem was that Spark advertised a dependency on Avro
even though it only really depends transitively on Avro through
other deps.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #49 from pwendell/avro-build-fix and squashes the following commits:

8d6ee92 [Patrick Wendell] SPARK-1121: Add avro to yarn-alpha profile
2014-03-02 15:18:19 -08:00
Patrick Wendell 1fd2bfd3dd Remove remaining references to incubation
This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #51 from pwendell/tlp and squashes the following commits:

d553b1b [Patrick Wendell] Remove remaining references to incubation
2014-03-02 01:00:16 -08:00
Semih Salihoglu 1f4c7f7ecc Graph primitives2
Hi guys,

I'm following Joey and Ankur's suggestions to add collectEdges and pickRandomVertex. I'm also adding the tests for collectEdges and refactoring one method getCycleGraph in GraphOpsSuite.scala.

Thank you,

semih

Author: Semih Salihoglu <semihsalihoglu@gmail.com>

Closes #580 from semihsalihoglu/GraphPrimitives2 and squashes the following commits:

937d3ec [Semih Salihoglu] - Fixed the scalastyle errors.
a69a152 [Semih Salihoglu] - Adding collectEdges and pickRandomVertices. - Adding tests for collectEdges. - Refactoring a getCycle utility function for GraphOpsSuite.scala.
41265a6 [Semih Salihoglu] - Adding collectEdges and pickRandomVertex. - Adding tests for collectEdges. - Recycling a getCycle utility test file.
2014-02-24 22:42:30 -08:00
Prashant Sharma 919bd7f669 Merge pull request #567 from ScrapCodes/style2.
SPARK-1058, Fix Style Errors and Add Scala Style to Spark Build. Pt 2

Continuation of PR #557

With this all scala style errors are fixed across the code base !!

The reason for creating a separate PR was to not interrupt an already reviewed and ready to merge PR. Hope this gets reviewed soon and merged too.

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #567 and squashes the following commits:

3b1ec30 [Prashant Sharma] scala style fixes
2014-02-09 22:17:52 -08:00
Patrick Wendell b69f8b2a01 Merge pull request #557 from ScrapCodes/style. Closes #557.
SPARK-1058, Fix Style Errors and Add Scala Style to Spark Build.

Author: Patrick Wendell <pwendell@gmail.com>
Author: Prashant Sharma <scrapcodes@gmail.com>

== Merge branch commits ==

commit 1a8bd1c059b842cb95cc246aaea74a79fec684f4
Author: Prashant Sharma <scrapcodes@gmail.com>
Date:   Sun Feb 9 17:39:07 2014 +0530

    scala style fixes

commit f91709887a8e0b608c5c2b282db19b8a44d53a43
Author: Patrick Wendell <pwendell@gmail.com>
Date:   Fri Jan 24 11:22:53 2014 -0800

    Adding scalastyle snapshot
2014-02-09 10:09:19 -08:00
Mark Hamstra c2341c92bb Merge pull request #542 from markhamstra/versionBump. Closes #542.
Version number to 1.0.0-SNAPSHOT

Since 0.9.0-incubating is done and out the door, we shouldn't be building 0.9.0-incubating-SNAPSHOT anymore.

@pwendell

Author: Mark Hamstra <markhamstra@gmail.com>

== Merge branch commits ==

commit 1b00a8a7c1a7f251b4bb3774b84b9e64758eaa71
Author: Mark Hamstra <markhamstra@gmail.com>
Date:   Wed Feb 5 09:30:32 2014 -0800

    Version number to 1.0.0-SNAPSHOT
2014-02-08 16:00:43 -08:00
Jianping J Wang a5a513e25e Add jblas dependency 2014-01-23 19:48:39 +08:00
Jianping J Wang cc0fd33177 Replace commons-math with jblas 2014-01-23 19:44:30 +08:00
Sean Owen fd0c5b8c57 Depend on Commons Math explicitly instead of accidentally getting it from Hadoop (which stops working in 2.2.x) and also use the newer commons-math3 2014-01-22 22:25:49 +00:00
Patrick Wendell 5fecd2516d Merge pull request #441 from pwendell/graphx-build
GraphX shouldn't list Spark as provided.

I noticed this when building an application against GraphX to audit the released artifacts.
2014-01-15 11:15:07 -08:00
Patrick Wendell 9259d706be GraphX shouldn't list Spark as provided 2014-01-15 10:46:37 -08:00
Reynold Xin 3d9e66d92a Merge pull request #436 from ankurdave/VertexId-case
Rename VertexID -> VertexId in GraphX
2014-01-14 23:17:05 -08:00
Reynold Xin dfb152446d Fixed SVDPlusPlusSuite in Maven build. 2014-01-14 22:18:43 -08:00
Ankur Dave f4d9019aa8 VertexID -> VertexId 2014-01-14 22:17:18 -08:00
Patrick Wendell 23034798d7 Add missing header files 2014-01-14 01:17:13 -08:00
Joseph E. Gonzalez 80e73ed000 Adding minimal additional functionality to EdgeRDD 2014-01-13 22:56:57 -08:00
Ankur Dave af645be5b8 Fix all code examples in guide 2014-01-13 22:29:45 -08:00
Ankur Dave 76ebdae798 Fix bug in GraphLoader.edgeListFile that caused srcId > dstId 2014-01-13 22:20:45 -08:00
Ankur Dave c6dbfd1694 Edge object must be public for Edge case class 2014-01-13 22:08:44 -08:00
Ankur Dave c28e5a08ee Improve scaladoc links 2014-01-13 21:11:39 -08:00
Ankur Dave c6023bee60 Fix infinite loop in GraphGenerators.generateRandomEdges
The loop occurred when numEdges < numVertices. This commit fixes it by
allowing generateRandomEdges to generate a multigraph.
2014-01-13 21:02:37 -08:00
Ankur Dave 84d6af8021 Make Graph{,Impl,Ops} serializable to work around capture 2014-01-13 21:02:37 -08:00
Ankur Dave d4d9ece1af Remove Graph.statistics and GraphImpl.printLineage 2014-01-13 21:02:37 -08:00
Reynold Xin 0b18bfba1a Updated doc for PageRank. 2014-01-13 18:51:04 -08:00
Reynold Xin 9317286b72 More cleanup. 2014-01-13 18:45:35 -08:00
Reynold Xin 8e5c732430 Moved SVDPlusPlusConf into SVDPlusPlus object itself. 2014-01-13 18:45:20 -08:00
Reynold Xin 1dce9ce446 Moved PartitionStrategy's into an object. 2014-01-13 18:32:04 -08:00
Reynold Xin ae06d2c22f Updated GraphGenerator. 2014-01-13 18:31:49 -08:00
Reynold Xin 87f335db78 Made more things private. 2014-01-13 18:30:26 -08:00
Reynold Xin a4e12af7aa Merge branch 'graphx' of github.com:ankurdave/incubator-spark into graphx
Conflicts:
	graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
2014-01-13 17:42:59 -08:00
Reynold Xin 02a8f54bfa Miscel doc update. 2014-01-13 17:40:36 -08:00
Joseph E. Gonzalez cfe4a29dcb Improvements in example code for the programming guide as well as adding serialization support for GraphImpl to address issues with failed closure capture. 2014-01-13 17:18:31 -08:00
Ankur Dave ae4b75d94a Add EdgeDirection.Either and use it to fix CC bug
The bug was due to a misunderstanding of the activeSetOpt parameter to
Graph.mapReduceTriplets. Passing EdgeDirection.Both causes
mapReduceTriplets to run only on edges with *both* vertices in the
active set. This commit adds EdgeDirection.Either, which causes
mapReduceTriplets to run on edges with *either* vertex in the active
set. This is what connected components needed.
2014-01-13 17:03:03 -08:00
Ankur Dave 1bd5cefcae Remove aggregateNeighbors 2014-01-13 17:03:03 -08:00
Ankur Dave 8038da2328 Merge pull request #2 from jegonzal/GraphXCCIssue
Improving documentation and identifying potential bug in CC calculation.
2014-01-13 14:59:30 -08:00
Joseph E. Gonzalez 80e4d98dc6 Improving documentation and identifying potential bug in CC calculation. 2014-01-13 13:40:16 -08:00
Ankur Dave 9fe88627b5 Improve EdgeRDD scaladoc 2014-01-13 13:16:41 -08:00
Ankur Dave ea69cff711 Further improve VertexRDD scaladocs 2014-01-13 12:52:52 -08:00
Ankur Dave d691e9f47e Move algorithms to GraphOps 2014-01-12 21:47:16 -08:00
Ankur Dave 20c509b805 Add TriangleCount example 2014-01-12 21:41:32 -08:00
Joseph E. Gonzalez 2216319f48 adding Pregel as an operator in GraphOps and cleaning up documentation of GraphOps 2014-01-12 21:26:37 -08:00
Ankur Dave 7a4bb863c7 Add connected components example to doc 2014-01-12 16:58:18 -08:00
Ankur Dave 5e35d39e0f Add PageRank example and data 2014-01-12 13:10:53 -08:00
Ankur Dave f096f4eaf1 Link methods in programming guide; document VertexID 2014-01-12 10:55:29 -08:00
Ankur Dave 02771aa087 Make EdgeDirection val instead of case object for Java compat. 2014-01-11 13:15:46 -08:00
Ankur Dave 574c0d28c2 Use SparkConf in GraphX tests (via LocalSparkContext) 2014-01-11 12:39:30 -08:00
Ankur Dave 55101f5821 One-line Scaladoc comments in Edge and EdgeDirection 2014-01-11 12:35:41 -08:00
Ankur Dave 64f73f73a0 Fix indent and use SparkConf in Analytics 2014-01-11 12:33:06 -08:00
Ankur Dave 732333d78e Remove GraphLab 2014-01-11 11:49:35 -08:00
Ankur Dave 0b5c49ebad Make nullValue and VertexSet package-private 2014-01-11 11:49:35 -08:00
Ankur Dave feaa078022 algorithms -> lib 2014-01-11 00:30:10 -08:00
Ankur Dave 4f7ddf40fc Optimize Edge.lexicographicOrdering 2014-01-11 00:15:01 -08:00
Ankur Dave 34496d6a9f Move Analytics to algorithms and fix doc 2014-01-11 00:08:36 -08:00
Ankur Dave 2d7e8d8c48 Add GC note to GraphLab 2014-01-10 23:46:02 -08:00
Ankur Dave c4fb6a87d3 Fix scaladoc warnings 2014-01-10 18:36:42 -08:00
Ankur Dave 41d6586e8e Revert changes to Spark's (PrimitiveKey)OpenHashMap; copy PKOHM to graphx 2014-01-10 18:00:54 -08:00
Ankur Dave 85a6645d31 Add doc for Algorithms 2014-01-10 16:08:58 -08:00
Ankur Dave 04c20e7f4f Minor cleanup to docs 2014-01-10 15:58:30 -08:00
Ankur Dave 1788729273 Move VertexIdToIndexMap into impl 2014-01-10 15:58:18 -08:00
Ankur Dave 57d7487d3d Improve docs for VertexRDD 2014-01-10 15:48:20 -08:00
Ankur Dave 11dd35c28b Clean up GraphGenerators 2014-01-10 15:23:32 -08:00
Ankur Dave 9e48af6dba Remove unused HashUtils class 2014-01-10 15:22:57 -08:00
Ankur Dave 7bda997785 Improve docs for PartitionStrategy 2014-01-10 13:00:28 -08:00
Ankur Dave eb4b46f8d1 Improve docs for GraphOps 2014-01-10 12:46:00 -08:00
Ankur Dave 9454fa1f6c Remove duplicate method in GraphLoader and improve docs 2014-01-10 12:37:20 -08:00
Ankur Dave 37611e57f6 Improve docs for EdgeRDD, EdgeTriplet, and GraphLab 2014-01-10 12:37:03 -08:00
Ankur Dave eee9bc0958 Remove commented-out perf files 2014-01-10 12:36:15 -08:00
Ankur Dave c39ec3017f Remove some commented code 2014-01-10 12:17:17 -08:00
Ankur Dave 5fcd2a61b4 Finish cleaning up Graph docs 2014-01-10 12:17:04 -08:00
Ankur Dave 4c114a7556 Start cleaning up Scaladocs in Graph and EdgeRDD 2014-01-10 11:37:54 -08:00
Ankur Dave cfc10c74a3 Remove EdgeTriplet.{src,dst}Stale, which were unused 2014-01-10 10:43:23 -08:00
Ankur Dave bf50e8c6cd Remove commented code from Analytics 2014-01-10 10:37:04 -08:00
Ankur Dave 1b2aad918c Update graphx/pom.xml to mirror mllib/pom.xml 2014-01-10 10:34:40 -08:00
Ankur Dave 729277ebc4 Undo 8b6b8ac87f
Getting unpersist right in GraphLab is tricky.
2014-01-10 01:53:28 -08:00
Ankur Dave 4cc550909a graph -> graphx in log4j.properties 2014-01-10 00:59:59 -08:00
Ankur Dave ba511f890e Avoid recomputation by caching all multiply-used RDDs 2014-01-10 00:35:02 -08:00
Ankur Dave 8b6b8ac87f Unpersist previous iterations in GraphLab 2014-01-10 00:34:08 -08:00
Ankur Dave 2578332f97 Add Graph.unpersistVertices() 2014-01-09 23:34:35 -08:00
Ankur Dave 8ae108f6c4 Unpersist previous iterations in Pregel 2014-01-09 23:25:35 -08:00
Ankur Dave b7c92dded3 Add implicit algorithm methods for Graph; remove standalone PageRank 2014-01-09 20:44:28 -08:00
Ankur Dave 731f56f309 graph -> graphx 2014-01-09 14:31:33 -08:00