Reynold Xin
4d3d68b8fb
Minor update to tests.
2013-11-30 14:15:47 -08:00
Ankur Dave
3292cb0f9c
Revert "Fix join error by caching vTable in mapReduceTriplets"
...
This reverts commit dee1318d3d
, which is
unnecessary due to 7528e6d5f1
.
2013-11-30 14:05:32 -08:00
Reynold Xin
e72bd91590
Merge branch 'clean1' of github.com:amplab/graphx into clean1
2013-11-30 14:04:45 -08:00
Reynold Xin
7528e6d5f1
Enable joining arbitrary VertexPartitions (with different indexes).
2013-11-30 14:04:16 -08:00
Ankur Dave
eed3195038
Fix VertexSetRDD test by enabling index reuse
2013-11-30 13:50:37 -08:00
Ankur Dave
dee1318d3d
Fix join error by caching vTable in mapReduceTriplets
2013-11-30 13:37:19 -08:00
Reynold Xin
10c0f9b0bb
Added a log4j properties file for graphx unit tests.
2013-11-30 13:18:43 -08:00
Reynold Xin
95e83af209
More, bigger cleaning for better encapsulation of VertexSetRDD and VertexPartition. This is work in progress as stuff doesn't really run.
2013-11-27 00:30:26 -08:00
Reynold Xin
caba162861
Added join and aggregateUsingIndex to VertexPartition.
2013-11-26 21:02:39 -08:00
Ankur Dave
9e896be375
Test edge filtering in subgraph (test fails)
2013-11-26 15:58:55 -08:00
Ankur Dave
137294e2ab
Test GraphImpl.subgraph and fix bug
2013-11-26 15:32:47 -08:00
Reynold Xin
2d19d0381b
Merge branch 'simplify' into clean
2013-11-26 13:55:26 -08:00
Reynold Xin
d58bfa8573
Code cleaning to improve readability.
2013-11-26 13:54:46 -08:00
Dan Crankshaw
4b6b15dadd
Actually use partitioner command line args in Analytics.
2013-11-24 16:38:38 -08:00
Ankur Dave
fad6e70add
Simplify GraphImpl internals
2013-11-23 02:59:56 -08:00
Reynold Xin
18ce7e940b
Merge pull request #73 from jegonzal/TriangleCount
...
Triangle count
2013-11-22 17:02:40 -08:00
Joseph E. Gonzalez
de3d6ee5a7
Fixing build after merging upstream changes.
2013-11-19 22:03:49 -08:00
Joseph E. Gonzalez
12cb19b1c1
Adding comments and addressing comments.
2013-11-19 21:37:29 -08:00
Joseph E. Gonzalez
ae4ffc319a
Setting the initial vertex set size to be small.
2013-11-19 21:36:15 -08:00
Joseph E. Gonzalez
18700b6e74
Switching mapReduceTriplets mapFunction to return iterator instead of array to allow optimizations of the returned object.
2013-11-19 21:36:15 -08:00
Joseph E. Gonzalez
983810ad69
Now with style. Addressing most of Reynolds comments.
2013-11-19 21:35:03 -08:00
Joseph E. Gonzalez
2093a17ff3
Adding triangle count code
2013-11-19 21:35:03 -08:00
Joseph E. Gonzalez
8719ba83c8
Modifying graph loaders to create initial vertex sets more efficiently and load undirected graphs.
2013-11-19 21:35:02 -08:00
Joseph E. Gonzalez
288ae310e7
adding test for collectNeighborIds
2013-11-19 21:03:00 -08:00
Joseph E. Gonzalez
2fc6f5bd47
Switching collectNeighborIds to use mapReduceTriplets directly
2013-11-19 21:03:00 -08:00
Dan Crankshaw
96fafdbd4b
Removed sleep from pagerank in Analytics.
2013-11-19 20:39:34 -08:00
Dan Crankshaw
37a524d91c
Addressed code review comments.
2013-11-19 16:39:39 -08:00
Dan Crankshaw
5f3ee53751
Added accessVertexAttr func which somehow got lost in a merge.
2013-11-18 19:34:02 -08:00
Dan Crankshaw
8a460e1811
Added partitioner to GraphImpl constructor args.
2013-11-18 19:32:03 -08:00
Dan Crankshaw
1022e9bf17
Fixed code review changes.
2013-11-18 18:08:32 -08:00
Dan Crankshaw
2aaa095687
Merge branch 'master' of github.com:amplab/graphx
2013-11-17 19:35:43 -08:00
Ankur Dave
62a2a71c37
Merge pull request #65 from amplab/varenc
...
Use variable encoding for ints, longs, and doubles in the specialized serializers.
2013-11-15 13:12:07 -08:00
Ankur Dave
3558e8bda1
During graph creation, create eTable earlier
2013-11-13 17:07:23 -08:00
Joseph E. Gonzalez
5a9b07ead2
Fixing documentation
2013-11-13 10:45:25 -08:00
Joseph E. Gonzalez
266eb01ce8
Addressing issue in Graph creation where a graph created with a vertex set that does not span all of the vertices in the edges will crash on triplet construction.
2013-11-13 10:45:25 -08:00
Reynold Xin
882d069189
Fixed the bug in variable encoding for longs.
2013-11-12 18:50:03 -08:00
Reynold Xin
1e5c17812d
Use variable encoding for ints, longs, and doubles in the specialized serializers.
2013-11-12 15:30:27 -08:00
Dan Crankshaw
a13460bb64
Updated documentation
2013-11-11 23:42:02 -08:00
Dan Crankshaw
7c573a8b43
Added PartitionStrategy option
2013-11-11 23:42:01 -08:00
Dan Crankshaw
8d8056da14
Fixed issue with canonical edge partitioner.
2013-11-11 23:40:23 -08:00
Dan Crankshaw
4a670ef0ba
Merge branch 'master' of github.com:amplab/graphx
2013-11-11 21:42:08 -08:00
Joseph E. Gonzalez
577092080c
Cleanning up documentation of VertexSetRDD.scala
2013-11-11 17:29:22 -08:00
Reynold Xin
b8e294a21b
Merge pull request #61 from ankurdave/pid2vid
...
Shuffle replicated vertex attributes efficiently in columnar format
2013-11-11 16:25:42 -08:00
Ankur Dave
bee1015620
Handle ClassNotFoundException from ByteCodeUtils
...
ByteCodeUtils.invokedMethod(), which we use in mapReduceTriplets, throws
a ClassNotFoundException when called with a closure defined in the
console. This commit catches the exception and conservatively assumes
the closure references all edge attributes.
2013-11-10 23:00:37 -08:00
Dan Crankshaw
60db25bded
Fixed merge conflicts.
2013-11-10 15:45:55 -08:00
Ankur Dave
d1ff1b7222
Build pid2vid structures only once, in Vid2Pid
2013-11-10 14:47:39 -08:00
Ankur Dave
502c511711
Use pid2vid for creating VTableReplicatedValues
2013-11-10 14:36:14 -08:00
Ankur Dave
53d24a973e
Fix typo
2013-11-10 14:24:38 -08:00
Ankur Dave
aa24b0bbe8
Add test for mapReduceTriplets in GraphSuite
2013-11-10 14:24:38 -08:00
Ankur Dave
bf4e45e685
Factor out VTableReplicatedValues
2013-11-10 14:24:38 -08:00
Ankur Dave
cdbd19bbee
Create all versions of vid2pid ahead of time
2013-11-10 14:10:23 -08:00
Ankur Dave
27e4355d61
Test no vertex attribute replication
2013-11-10 14:04:12 -08:00
Ankur Dave
80abc28078
Optimize mrTriplets for source-attr-only mapF using bytecode inspection
2013-11-10 14:04:12 -08:00
Reynold Xin
0e813cd483
Fix the hanging bug.
2013-11-09 23:29:37 -08:00
Joseph E. Gonzalez
6083e4350f
Adding unit tests to reproduce error.
2013-11-08 15:39:30 -08:00
Joseph E. Gonzalez
161784d0e6
Fixing tests
2013-11-07 20:40:21 -08:00
Joseph E. Gonzalez
e523f0d2fb
merged and debugged
2013-11-07 20:19:49 -08:00
Joseph E. Gonzalez
908e606473
Additional optimizations
2013-11-07 19:47:30 -08:00
Reynold Xin
bac7be30cd
Made more specialized messages.
2013-11-07 19:39:48 -08:00
Reynold Xin
64ad3b18d9
Merge branch 'master' into rxin
...
Conflicts:
graph/src/main/scala/org/apache/spark/graph/impl/GraphImpl.scala
2013-11-07 19:23:42 -08:00
Reynold Xin
2406bf33e4
Use custom serializer for aggregation messages when the data type is int/double.
2013-11-07 19:18:58 -08:00
Joseph E. Gonzalez
e9308e0e75
Changing Pregel API to operate directly on edge triplets in SendMessage rather than (Vid, EdgeTriplet) pairs.
2013-11-07 18:04:06 -08:00
Reynold Xin
6fadff2b92
Converted for loops to while loops in EdgePartition.
2013-11-07 16:54:33 -08:00
Dan Crankshaw
384befb208
Merge branch 'master' of github.com:amplab/graphx
2013-11-06 19:50:55 -08:00
Joseph E. Gonzalez
3e504938c2
merging upstream changes
2013-11-05 01:36:48 -08:00
Joseph E. Gonzalez
2dc9ec2387
Reverting to Array based (materialized) output of all VertexSetRDD operations.
2013-11-05 01:15:12 -08:00
Reynold Xin
551a43fd3d
Merge branch 'master' of github.com:apache/incubator-spark into mergemerge
...
Conflicts:
README.md
core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
core/src/main/scala/org/apache/spark/util/collection/PrimitiveKeyOpenHashMap.scala
2013-11-04 21:02:36 -08:00
Dan Crankshaw
d87d112b2c
Merge branch 'master' of github.com:amplab/graphx
2013-11-01 12:04:09 -07:00
Joseph E. Gonzalez
e7d37472b8
After some testing I realized that the IndexedSeq is still instantiating the array (not maintaining a view) so I have replaced all IndexedSeq[V] with (Int => V)
2013-10-31 21:09:39 -07:00
Joseph E. Gonzalez
63311d9c72
renamed update to setMerge
2013-10-31 20:12:30 -07:00
Dan Crankshaw
e218e30b52
Merge branch 'master' of github.com:amplab/graphx
2013-10-31 19:54:17 -07:00
Dan Crankshaw
0a61cafba8
Added logging to Graph, GraphLab, and Pregel.
2013-10-31 19:54:06 -07:00
Joseph E. Gonzalez
8381aeffb3
This commit introduces the OpenHashSet and OpenHashMap as indexing primitives.
...
Large parts of the VertexSetRDD were restructured to take advantage of:
1) the OpenHashSet as an index map
2) view based lazy mapValues and mapValuesWithVertices
3) the cogroup code is currently disabled (since it is not used in any of the tests)
The GraphImpl was updated to also use the OpenHashSet and PrimitiveOpenHashMap
wherever possible:
1) the LocalVidMaps (used to track replicated vertices) are now implemented
using the OpenHashSet
2) an OpenHashMap is temporarily constructed to combine the local OpenHashSet
with the local (replicated) vertex attribute arrays
3) because the OpenHashSet constructor grabs a class manifest all operations
that construct OpenHashSets have been moved to the GraphImpl Singleton to prevent
implicit variable capture within closures.
2013-10-31 18:13:02 -07:00
Dan Crankshaw
b3bcfc09c7
Merge branch 'master' of github.com:amplab/graphx
2013-10-31 18:03:00 -07:00
Joseph E. Gonzalez
aeb773fa47
Merging with upstream master.
2013-10-31 10:12:12 -07:00
Reynold Xin
3f3c727bc5
Merge pull request #41 from jegonzal/LineageTracking
...
Optimizing Graph Lineage
2013-10-31 09:52:25 -07:00
Joseph E. Gonzalez
d6b5122532
Switching to the @rxin BitSet implementation for VertexSet Value tables.
2013-10-31 01:44:24 -07:00
Dan Crankshaw
c430d2e21d
Added bitset to kryo register
2013-10-31 01:01:59 -07:00
Dan Crankshaw
37b4afbbf9
Merge branch 'cleanup'
2013-10-30 23:17:50 -07:00
Joseph E. Gonzalez
a3ce484a2c
Adding additional type constraints to VertexSetRDD to help diagnose issues with recent benchmarks.
2013-10-30 21:02:21 -07:00
Joseph E. Gonzalez
09ea661bbb
removing completely unnecessary map operation.
2013-10-30 20:07:26 -07:00
Joseph E. Gonzalez
003f8a505d
Removing potential additional shuffle dependency where an already partitioned RDD[(Vid, VD)] is repartitioned.
2013-10-30 20:06:54 -07:00
Joseph E. Gonzalez
d513addb77
added lineage tracking code
2013-10-30 20:05:29 -07:00
Joseph E. Gonzalez
a4b8ddf417
removing unused commented code
2013-10-30 16:07:05 -07:00
Dan Crankshaw
a0c86c3689
Merge pull request #38 from jegonzal/Documentation
...
Improving Documentation
2013-10-30 15:34:39 -07:00
Dan Crankshaw
e1099f4d89
Fixed issue with canonical edge partitioner.
2013-10-30 15:03:21 -07:00
Joey
06adf636c5
Merge pull request #33 from kellrott/master
...
Fixing graph/pom.xml
2013-10-29 16:43:46 -07:00
Joseph E. Gonzalez
38ec0baf5c
fixing a typo in the VertexSetRDD docs
2013-10-29 16:27:55 -07:00
Joseph E. Gonzalez
d8c8256e52
merging upstream changes
2013-10-29 16:23:26 -07:00
Joseph E. Gonzalez
08c7b040d6
Documented the VertexSetRDD
2013-10-29 15:03:13 -07:00
Joseph E. Gonzalez
ede329336d
Fixing a scaladoc bug in graph generators.
2013-10-29 14:50:12 -07:00
Joseph E. Gonzalez
15958ca65a
Reindenting documentation.
2013-10-29 14:01:24 -07:00
Joseph E. Gonzalez
d316cad9b1
Documented Graph.appy functions.
2013-10-29 13:58:04 -07:00
Joseph E. Gonzalez
19da8820fc
Minor modifications to documentation.
2013-10-29 11:06:06 -07:00
Joseph E. Gonzalez
77626d1507
Adding collect neighbors and documenting GraphOps.
2013-10-29 11:05:42 -07:00
Joseph E. Gonzalez
942de98433
Making suggested changes.
2013-10-29 10:19:49 -07:00
Joseph E. Gonzalez
d6a902f309
Finished updating connected components to used Pregel like abstraction and created a series of tests in the AnalyticsSuite.
2013-10-28 11:52:26 -07:00
Joseph E. Gonzalez
a2287ae138
Implementing connected components on top of pregel like abstraction.
2013-10-27 10:42:11 -07:00
Joseph E. Gonzalez
6a0fbc0374
Updating the GraphLab API to match the changes made to the Pregel API.
2013-10-26 15:44:19 -07:00
Joseph E. Gonzalez
08024c938c
Adding more documentation to the Pregel API as well as additional functionality including the ability to specify the edge direction along which messages are computed.
2013-10-26 15:42:51 -07:00