ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36009][GRAPHX] Add missing GraphX classes to registerKryoClasses util method

Browse source 
### What changes were proposed in this pull request?
Add `ShippableVertexPartition` and `RoutingTablePartition` to the classes registered with Kryo in `GraphXUtils.registerKryoClasses`.

### Why are the changes needed?
`VertexRDDImpl` uses an `RDD[ShippableVertexPartition[VD]]` however, `GraphXUtils.registerKryoClasses` does not register `ShippableVertexPartition`. This means when running with `spark.kryo.registrationRequired` set to `true`, we get a "Class is not registered" exception. This is an issue as it prevents other unregistered classes from being discovered using `spark.kryo.registrationRequired` as the first unregistered class found halts the whole job. It also potentially decreases the serialised size of the RDD when using Kryo.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Tested manually by running with `spark.kryo.registrationRequired` set to `true` and verifying no "Class is not registered" exception was thrown.

Closes #32973 from matthewrj/bug/register-graphx-classes.

Lead-authored-by: Matthew Jones <matthew@vitaler.com>
Co-authored-by: Matthew Jones <mlety2@gmail.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
This commit is contained in:
Matthew Jones 2021-07-06 07:25:22 -05:00 committed by Sean Owen
parent d5d1222686
commit 2fff060b6d
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala
View file

@ -35,6 +35,8 @@ object GraphXUtils {
classOf[Edge[Object]],
classOf[(VertexId, Object)],
classOf[EdgePartition[Object, Object]],
classOf[ShippableVertexPartition[Object]],
classOf[RoutingTablePartition],
classOf[BitSet],
classOf[VertexIdToIndexMap],
classOf[VertexAttributeBlock[Object]],