ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5025 commits 6 branches 181 tags 397 MiB
Commit graph

319 commits

Author SHA1 Message Date
Ankur Dave e8ba51d644 Add standalone PageRank using only GraphX operators 2013-12-12 15:42:27 -08:00
Dan Crankshaw 12483d4ae6 Added BoundedPriorityQueue kryo registrator. Fixes top issue. 2013-12-11 20:56:21 -08:00
Ankur Dave 84d0e1a334 Set RDD names for easy debugging 2013-12-07 04:05:45 -08:00
Reynold Xin 41721b1494 Fixed a bug in VTableReplicated that we only process the first block. 2013-12-06 00:51:12 -08:00
Reynold Xin 3b0ee53eda Minor update. 2013-12-05 23:30:56 -08:00
Reynold Xin 15168d6c4d Fixed a bug in VTableReplicated that we are always broadcasting all the vertices. 2013-12-05 23:25:53 -08:00
Reynold Xin a6075ba11f Merge branch 'pregel-replicate-changed' of github.com:ankurdave/graphx into pregel-replicate-changed 2013-12-05 22:35:16 -08:00
Ankur Dave b707861ba0 Simplify GraphImpl.deltaJoinVertices 2013-12-05 20:01:32 -08:00
Ankur Dave 67bfa7fd65 Test prevVTableReplicated code path 2013-12-05 19:16:00 -08:00
Reynold Xin 920282c36a Merge branch 'pregel-replicate-changed' of github.com:ankurdave/graphx into pregel-replicate-changed 2013-12-05 18:57:36 -08:00
Ankur Dave 39b0256668 Use mask for dynamic Pregel 2013-12-05 18:55:59 -08:00
Ankur Dave 7457abe282 Re-roll last iteration of static Pregel loop 2013-12-05 16:59:49 -08:00
Reynold Xin 4f80dd22bd Fixed a bug that variable encoding doesn't work for ints that use all 64 bits. 2013-12-05 16:19:37 -08:00
Ankur Dave 4d4c1c931e Test Graph.deltaJoinVertices and VertexRDD.diff 2013-12-05 16:15:56 -08:00
Ankur Dave 15374fed97 In static Pregel, replicate only changed vertices 2013-12-05 15:55:33 -08:00
Reynold Xin 3fc4534d19 wip delta join. 2013-12-05 14:55:26 -08:00
Ankur Dave 6a7b396e5d Finish work on #85 2013-12-05 12:35:03 -08:00
Ankur Dave a3bb98b88a Always write Vids using variable encoding 
Also, autoformat Serializers.scala.
 2013-12-05 12:06:07 -08:00
Akihiro Matsukawa c49a6a7954 vertex should be inactive if no vertices scatter to it 2013-12-04 18:18:49 -08:00
Akihiro Matsukawa 38c6f5f612 add a predicate to GraphLab to indicate active vertices at start 2013-12-04 18:17:14 -08:00
Ankur Dave 2e583d2de4 Declare Vids explicitly to avoid ClassCastException 2013-12-04 17:34:14 -08:00
Ankur Dave 92e96f727e Fix compile errors in GraphSuite and SerializerSuite 2013-12-04 17:29:52 -08:00
Akihiro Matsukawa cb20175f97 add filter operation to GraphOps 2013-12-04 16:48:51 -08:00
Akihiro Matsukawa c888b611da add mask operation to Graph 2013-12-04 16:48:32 -08:00
Reynold Xin 8701cb55e6 Use specialized shuffler for aggregation. 2013-12-01 21:55:50 -08:00
Reynold Xin 55edbb4209 Created an algorithms package and put all algorithms there. 2013-12-01 20:17:26 -08:00
Reynold Xin 583a389e3f Removed PartitionStrategy from GraphImpl. 2013-11-30 17:00:54 -08:00
Reynold Xin 6eeadb667d Created EdgeRDD. 2013-11-30 16:53:54 -08:00
Reynold Xin 34ee81415e Merged Ankur's pull request #80 and fixed subgraph. 2013-11-30 15:10:30 -08:00
Reynold Xin 8e790b7f7a Merge branch 'subgraph-test' of github.com:ankurdave/graphx into clean1 
Conflicts:
	graph/src/main/scala/org/apache/spark/graph/impl/VertexPartition.scala
 2013-11-30 14:48:43 -08:00
Reynold Xin 229022891f Made all VertexPartition internal data structures private. 2013-11-30 14:45:56 -08:00
Reynold Xin b30e0ae035 Added an optimized count to VertexSetRDD. 2013-11-30 14:24:18 -08:00
Reynold Xin 689f757f7a Merge branch 'clean1' of github.com:amplab/graphx into clean1 2013-11-30 14:16:06 -08:00
Reynold Xin 4d3d68b8fb Minor update to tests. 2013-11-30 14:15:47 -08:00
Ankur Dave 3292cb0f9c Revert "Fix join error by caching vTable in mapReduceTriplets" 
This reverts commit dee1318d3d, which is
unnecessary due to 7528e6d5f1.
 2013-11-30 14:05:32 -08:00
Reynold Xin e72bd91590 Merge branch 'clean1' of github.com:amplab/graphx into clean1 2013-11-30 14:04:45 -08:00
Reynold Xin 7528e6d5f1 Enable joining arbitrary VertexPartitions (with different indexes). 2013-11-30 14:04:16 -08:00
Ankur Dave eed3195038 Fix VertexSetRDD test by enabling index reuse 2013-11-30 13:50:37 -08:00
Ankur Dave dee1318d3d Fix join error by caching vTable in mapReduceTriplets 2013-11-30 13:37:19 -08:00
Reynold Xin 10c0f9b0bb Added a log4j properties file for graphx unit tests. 2013-11-30 13:18:43 -08:00
Reynold Xin 95e83af209 More, bigger cleaning for better encapsulation of VertexSetRDD and VertexPartition. This is work in progress as stuff doesn't really run. 2013-11-27 00:30:26 -08:00
Reynold Xin caba162861 Added join and aggregateUsingIndex to VertexPartition. 2013-11-26 21:02:39 -08:00
Ankur Dave 9e896be375 Test edge filtering in subgraph (test fails) 2013-11-26 15:58:55 -08:00
Ankur Dave 137294e2ab Test GraphImpl.subgraph and fix bug 2013-11-26 15:32:47 -08:00
Reynold Xin 2d19d0381b Merge branch 'simplify' into clean 2013-11-26 13:55:26 -08:00
Reynold Xin d58bfa8573 Code cleaning to improve readability. 2013-11-26 13:54:46 -08:00
Dan Crankshaw 4b6b15dadd Actually use partitioner command line args in Analytics. 2013-11-24 16:38:38 -08:00
Ankur Dave fad6e70add Simplify GraphImpl internals 2013-11-23 02:59:56 -08:00
Reynold Xin 18ce7e940b Merge pull request #73 from jegonzal/TriangleCount 
Triangle count
 2013-11-22 17:02:40 -08:00
Joseph E. Gonzalez de3d6ee5a7 Fixing build after merging upstream changes. 2013-11-19 22:03:49 -08:00
Joseph E. Gonzalez 12cb19b1c1 Adding comments and addressing comments. 2013-11-19 21:37:29 -08:00
Joseph E. Gonzalez ae4ffc319a Setting the initial vertex set size to be small. 2013-11-19 21:36:15 -08:00
Joseph E. Gonzalez 18700b6e74 Switching mapReduceTriplets mapFunction to return iterator instead of array to allow optimizations of the returned object. 2013-11-19 21:36:15 -08:00
Joseph E. Gonzalez 983810ad69 Now with style. Addressing most of Reynolds comments. 2013-11-19 21:35:03 -08:00
Joseph E. Gonzalez 2093a17ff3 Adding triangle count code 2013-11-19 21:35:03 -08:00
Joseph E. Gonzalez 8719ba83c8 Modifying graph loaders to create initial vertex sets more efficiently and load undirected graphs. 2013-11-19 21:35:02 -08:00
Joseph E. Gonzalez 288ae310e7 adding test for collectNeighborIds 2013-11-19 21:03:00 -08:00
Joseph E. Gonzalez 2fc6f5bd47 Switching collectNeighborIds to use mapReduceTriplets directly 2013-11-19 21:03:00 -08:00
Dan Crankshaw 96fafdbd4b Removed sleep from pagerank in Analytics. 2013-11-19 20:39:34 -08:00
Dan Crankshaw 37a524d91c Addressed code review comments. 2013-11-19 16:39:39 -08:00
Dan Crankshaw 5f3ee53751 Added accessVertexAttr func which somehow got lost in a merge. 2013-11-18 19:34:02 -08:00
Dan Crankshaw 8a460e1811 Added partitioner to GraphImpl constructor args. 2013-11-18 19:32:03 -08:00
Dan Crankshaw 1022e9bf17 Fixed code review changes. 2013-11-18 18:08:32 -08:00
Dan Crankshaw 2aaa095687 Merge branch 'master' of github.com:amplab/graphx 2013-11-17 19:35:43 -08:00
Ankur Dave 62a2a71c37 Merge pull request #65 from amplab/varenc 
Use variable encoding for ints, longs, and doubles in the specialized serializers.
 2013-11-15 13:12:07 -08:00
Ankur Dave 3558e8bda1 During graph creation, create eTable earlier 2013-11-13 17:07:23 -08:00
Joseph E. Gonzalez 5a9b07ead2 Fixing documentation 2013-11-13 10:45:25 -08:00
Joseph E. Gonzalez 266eb01ce8 Addressing issue in Graph creation where a graph created with a vertex set that does not span all of the vertices in the edges will crash on triplet construction. 2013-11-13 10:45:25 -08:00
Reynold Xin 882d069189 Fixed the bug in variable encoding for longs. 2013-11-12 18:50:03 -08:00
Reynold Xin 1e5c17812d Use variable encoding for ints, longs, and doubles in the specialized serializers. 2013-11-12 15:30:27 -08:00
Dan Crankshaw a13460bb64 Updated documentation 2013-11-11 23:42:02 -08:00
Dan Crankshaw 7c573a8b43 Added PartitionStrategy option 2013-11-11 23:42:01 -08:00
Dan Crankshaw 8d8056da14 Fixed issue with canonical edge partitioner. 2013-11-11 23:40:23 -08:00
Dan Crankshaw 4a670ef0ba Merge branch 'master' of github.com:amplab/graphx 2013-11-11 21:42:08 -08:00
Joseph E. Gonzalez 577092080c Cleanning up documentation of VertexSetRDD.scala 2013-11-11 17:29:22 -08:00
Reynold Xin b8e294a21b Merge pull request #61 from ankurdave/pid2vid 
Shuffle replicated vertex attributes efficiently in columnar format
 2013-11-11 16:25:42 -08:00
Ankur Dave bee1015620 Handle ClassNotFoundException from ByteCodeUtils 
ByteCodeUtils.invokedMethod(), which we use in mapReduceTriplets, throws
a ClassNotFoundException when called with a closure defined in the
console. This commit catches the exception and conservatively assumes
the closure references all edge attributes.
 2013-11-10 23:00:37 -08:00
Dan Crankshaw 60db25bded Fixed merge conflicts. 2013-11-10 15:45:55 -08:00
Ankur Dave d1ff1b7222 Build pid2vid structures only once, in Vid2Pid 2013-11-10 14:47:39 -08:00
Ankur Dave 502c511711 Use pid2vid for creating VTableReplicatedValues 2013-11-10 14:36:14 -08:00
Ankur Dave 53d24a973e Fix typo 2013-11-10 14:24:38 -08:00
Ankur Dave aa24b0bbe8 Add test for mapReduceTriplets in GraphSuite 2013-11-10 14:24:38 -08:00
Ankur Dave bf4e45e685 Factor out VTableReplicatedValues 2013-11-10 14:24:38 -08:00
Ankur Dave cdbd19bbee Create all versions of vid2pid ahead of time 2013-11-10 14:10:23 -08:00
Ankur Dave 27e4355d61 Test no vertex attribute replication 2013-11-10 14:04:12 -08:00
Ankur Dave 80abc28078 Optimize mrTriplets for source-attr-only mapF using bytecode inspection 2013-11-10 14:04:12 -08:00
Reynold Xin 0e813cd483 Fix the hanging bug. 2013-11-09 23:29:37 -08:00
Joseph E. Gonzalez 6083e4350f Adding unit tests to reproduce error. 2013-11-08 15:39:30 -08:00
Joseph E. Gonzalez 161784d0e6 Fixing tests 2013-11-07 20:40:21 -08:00
Joseph E. Gonzalez e523f0d2fb merged and debugged 2013-11-07 20:19:49 -08:00
Joseph E. Gonzalez 908e606473 Additional optimizations 2013-11-07 19:47:30 -08:00
Reynold Xin bac7be30cd Made more specialized messages. 2013-11-07 19:39:48 -08:00
Reynold Xin 64ad3b18d9 Merge branch 'master' into rxin 
Conflicts:
	graph/src/main/scala/org/apache/spark/graph/impl/GraphImpl.scala
 2013-11-07 19:23:42 -08:00
Reynold Xin 2406bf33e4 Use custom serializer for aggregation messages when the data type is int/double. 2013-11-07 19:18:58 -08:00
Joseph E. Gonzalez e9308e0e75 Changing Pregel API to operate directly on edge triplets in SendMessage rather than (Vid, EdgeTriplet) pairs. 2013-11-07 18:04:06 -08:00
Reynold Xin 6fadff2b92 Converted for loops to while loops in EdgePartition. 2013-11-07 16:54:33 -08:00
Dan Crankshaw 384befb208 Merge branch 'master' of github.com:amplab/graphx 2013-11-06 19:50:55 -08:00
Joseph E. Gonzalez 3e504938c2 merging upstream changes 2013-11-05 01:36:48 -08:00
Joseph E. Gonzalez 2dc9ec2387 Reverting to Array based (materialized) output of all VertexSetRDD operations. 2013-11-05 01:15:12 -08:00
Reynold Xin 551a43fd3d Merge branch 'master' of github.com:apache/incubator-spark into mergemerge 
Conflicts:
	README.md
	core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
	core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
	core/src/main/scala/org/apache/spark/util/collection/PrimitiveKeyOpenHashMap.scala
 2013-11-04 21:02:36 -08:00