ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4751 commits 6 branches 181 tags 397 MiB
Commit graph

190 commits

Author SHA1 Message Date
Dan Crankshaw a13460bb64 Updated documentation 2013-11-11 23:42:02 -08:00
Dan Crankshaw 7c573a8b43 Added PartitionStrategy option 2013-11-11 23:42:01 -08:00
Dan Crankshaw 8d8056da14 Fixed issue with canonical edge partitioner. 2013-11-11 23:40:23 -08:00
Joseph E. Gonzalez 577092080c Cleanning up documentation of VertexSetRDD.scala 2013-11-11 17:29:22 -08:00
Reynold Xin b8e294a21b Merge pull request #61 from ankurdave/pid2vid 
Shuffle replicated vertex attributes efficiently in columnar format
 2013-11-11 16:25:42 -08:00
Ankur Dave bee1015620 Handle ClassNotFoundException from ByteCodeUtils 
ByteCodeUtils.invokedMethod(), which we use in mapReduceTriplets, throws
a ClassNotFoundException when called with a closure defined in the
console. This commit catches the exception and conservatively assumes
the closure references all edge attributes.
 2013-11-10 23:00:37 -08:00
Ankur Dave d1ff1b7222 Build pid2vid structures only once, in Vid2Pid 2013-11-10 14:47:39 -08:00
Ankur Dave 502c511711 Use pid2vid for creating VTableReplicatedValues 2013-11-10 14:36:14 -08:00
Ankur Dave 53d24a973e Fix typo 2013-11-10 14:24:38 -08:00
Ankur Dave aa24b0bbe8 Add test for mapReduceTriplets in GraphSuite 2013-11-10 14:24:38 -08:00
Ankur Dave bf4e45e685 Factor out VTableReplicatedValues 2013-11-10 14:24:38 -08:00
Ankur Dave cdbd19bbee Create all versions of vid2pid ahead of time 2013-11-10 14:10:23 -08:00
Ankur Dave 27e4355d61 Test no vertex attribute replication 2013-11-10 14:04:12 -08:00
Ankur Dave 80abc28078 Optimize mrTriplets for source-attr-only mapF using bytecode inspection 2013-11-10 14:04:12 -08:00
Reynold Xin 0e813cd483 Fix the hanging bug. 2013-11-09 23:29:37 -08:00
Joseph E. Gonzalez 6083e4350f Adding unit tests to reproduce error. 2013-11-08 15:39:30 -08:00
Joseph E. Gonzalez 161784d0e6 Fixing tests 2013-11-07 20:40:21 -08:00
Joseph E. Gonzalez e523f0d2fb merged and debugged 2013-11-07 20:19:49 -08:00
Joseph E. Gonzalez 908e606473 Additional optimizations 2013-11-07 19:47:30 -08:00
Reynold Xin bac7be30cd Made more specialized messages. 2013-11-07 19:39:48 -08:00
Reynold Xin 64ad3b18d9 Merge branch 'master' into rxin 
Conflicts:
	graph/src/main/scala/org/apache/spark/graph/impl/GraphImpl.scala
 2013-11-07 19:23:42 -08:00
Reynold Xin 2406bf33e4 Use custom serializer for aggregation messages when the data type is int/double. 2013-11-07 19:18:58 -08:00
Joseph E. Gonzalez e9308e0e75 Changing Pregel API to operate directly on edge triplets in SendMessage rather than (Vid, EdgeTriplet) pairs. 2013-11-07 18:04:06 -08:00
Reynold Xin 6fadff2b92 Converted for loops to while loops in EdgePartition. 2013-11-07 16:54:33 -08:00
Joseph E. Gonzalez 3e504938c2 merging upstream changes 2013-11-05 01:36:48 -08:00
Joseph E. Gonzalez 2dc9ec2387 Reverting to Array based (materialized) output of all VertexSetRDD operations. 2013-11-05 01:15:12 -08:00
Reynold Xin 551a43fd3d Merge branch 'master' of github.com:apache/incubator-spark into mergemerge 
Conflicts:
	README.md
	core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
	core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
	core/src/main/scala/org/apache/spark/util/collection/PrimitiveKeyOpenHashMap.scala
 2013-11-04 21:02:36 -08:00
Joseph E. Gonzalez e7d37472b8 After some testing I realized that the IndexedSeq is still instantiating the array (not maintaining a view) so I have replaced all IndexedSeq[V] with (Int => V) 2013-10-31 21:09:39 -07:00
Joseph E. Gonzalez 63311d9c72 renamed update to setMerge 2013-10-31 20:12:30 -07:00
Joseph E. Gonzalez 8381aeffb3 This commit introduces the OpenHashSet and OpenHashMap as indexing primitives. 
Large parts of the VertexSetRDD were restructured to take advantage of:

  1) the OpenHashSet as an index map
  2) view based lazy mapValues and mapValuesWithVertices
  3) the cogroup code is currently disabled (since it is not used in any of the tests)

The GraphImpl was updated to also use the OpenHashSet and PrimitiveOpenHashMap
wherever possible:

  1) the LocalVidMaps (used to track replicated vertices) are now implemented
     using the OpenHashSet
  2) an OpenHashMap is temporarily constructed to combine the local OpenHashSet
     with the local (replicated) vertex attribute arrays
  3) because the OpenHashSet constructor grabs a class manifest all operations
     that construct OpenHashSets have been moved to the GraphImpl Singleton to prevent
     implicit variable capture within closures.
 2013-10-31 18:13:02 -07:00
Joseph E. Gonzalez aeb773fa47 Merging with upstream master. 2013-10-31 10:12:12 -07:00
Reynold Xin 3f3c727bc5 Merge pull request #41 from jegonzal/LineageTracking 
Optimizing Graph Lineage
 2013-10-31 09:52:25 -07:00
Joseph E. Gonzalez d6b5122532 Switching to the @rxin BitSet implementation for VertexSet Value tables. 2013-10-31 01:44:24 -07:00
Dan Crankshaw c430d2e21d Added bitset to kryo register 2013-10-31 01:01:59 -07:00
Joseph E. Gonzalez a3ce484a2c Adding additional type constraints to VertexSetRDD to help diagnose issues with recent benchmarks. 2013-10-30 21:02:21 -07:00
Joseph E. Gonzalez 09ea661bbb removing completely unnecessary map operation. 2013-10-30 20:07:26 -07:00
Joseph E. Gonzalez 003f8a505d Removing potential additional shuffle dependency where an already partitioned RDD[(Vid, VD)] is repartitioned. 2013-10-30 20:06:54 -07:00
Joseph E. Gonzalez d513addb77 added lineage tracking code 2013-10-30 20:05:29 -07:00
Joseph E. Gonzalez a4b8ddf417 removing unused commented code 2013-10-30 16:07:05 -07:00
Dan Crankshaw a0c86c3689 Merge pull request #38 from jegonzal/Documentation 
Improving Documentation
 2013-10-30 15:34:39 -07:00
Joey 06adf636c5 Merge pull request #33 from kellrott/master 
Fixing graph/pom.xml
 2013-10-29 16:43:46 -07:00
Joseph E. Gonzalez 38ec0baf5c fixing a typo in the VertexSetRDD docs 2013-10-29 16:27:55 -07:00
Joseph E. Gonzalez d8c8256e52 merging upstream changes 2013-10-29 16:23:26 -07:00
Joseph E. Gonzalez 08c7b040d6 Documented the VertexSetRDD 2013-10-29 15:03:13 -07:00
Joseph E. Gonzalez ede329336d Fixing a scaladoc bug in graph generators. 2013-10-29 14:50:12 -07:00
Joseph E. Gonzalez 15958ca65a Reindenting documentation. 2013-10-29 14:01:24 -07:00
Joseph E. Gonzalez d316cad9b1 Documented Graph.appy functions. 2013-10-29 13:58:04 -07:00
Joseph E. Gonzalez 19da8820fc Minor modifications to documentation. 2013-10-29 11:06:06 -07:00
Joseph E. Gonzalez 77626d1507 Adding collect neighbors and documenting GraphOps. 2013-10-29 11:05:42 -07:00
Joseph E. Gonzalez 942de98433 Making suggested changes. 2013-10-29 10:19:49 -07:00