2023-07-03 16:52:13 -04:00
# Big Todos for CIDR:
- [ ] Clone the main [Spark Optimizer Rules ](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L72 )
- [x] PushProjectionThroughUnion,
2023-07-09 13:54:54 -04:00
- [x] PushProjectionThroughLimit,
2023-07-03 16:52:13 -04:00
- [x] ReorderJoin,
2023-07-09 13:54:54 -04:00
- [x] EliminateOuterJoin,
2023-07-03 16:52:13 -04:00
- [ ] PushDownPredicates,
- [ ] PushDownLeftSemiAntiJoin,
- [ ] PushLeftSemiLeftAntiThroughJoin,
- [ ] LimitPushDown,
- [ ] LimitPushDownThroughWindow,
- [ ] ColumnPruning,
- [ ] GenerateOptimization,
- [ ] CollapseRepartition,
- [ ] CollapseProject,
- [ ] OptimizeWindowFunctions,
- [ ] CollapseWindow,
- [ ] EliminateOffsets,
- [ ] EliminateLimits,
- [ ] CombineUnions,
- [ ] OptimizeRepartition,
- [ ] TransposeWindow,
- [ ] NullPropagation,
- [ ] NullDownPropagation,
- [ ] ConstantPropagation,
- [ ] FoldablePropagation,
- [ ] OptimizeIn,
- [ ] OptimizeRand,
- [ ] ConstantFolding,
- [ ] EliminateAggregateFilter,
- [ ] ReorderAssociativeOperator,
- [ ] LikeSimplification,
- [ ] BooleanSimplification,
- [ ] SimplifyConditionals,
- [ ] PushFoldableIntoBranches,
- [ ] RemoveDispensableExpressions,
- [ ] SimplifyBinaryComparison,
- [ ] ReplaceNullWithFalseInPredicate,
- [ ] PruneFilters,
- [ ] SimplifyCasts,
- [ ] SimplifyCaseConversionExpressions,
- [ ] RewriteCorrelatedScalarSubquery,
- [ ] RewriteLateralSubquery,
- [ ] EliminateSerialization,
- [ ] RemoveRedundantAliases,
- [ ] RemoveRedundantAggregates,
- [ ] UnwrapCastInBinaryComparison,
- [ ] RemoveNoopOperators,
- [ ] OptimizeUpdateFields,
- [ ] SimplifyExtractValueOps,
- [ ] OptimizeCsvJsonExprs,
- [ ] CombineConcats,
- [ ] PushdownPredicatesAndPruneColumnsForCTEDef
- [ ] Codegen: Output a Spark-like optimizer based on the above rules
2023-07-05 20:34:36 -04:00
- [x] Logic generation
2023-07-09 13:54:54 -04:00
- [x] Compilation pipeline to streamline testing
2023-07-03 16:52:13 -04:00
- [ ] Test: Do these rules generate the same results as spark?
- [ ] Apply the rule merge optimization
- [ ] Generate a merged decision tree
- [ ] Codegen: Generate a rule for the joint BDD
- [ ] Test: Do these rules generate the same results as spark?
- [ ] Test: Can the generated rule be plugged back into spark?
- [ ] Apply the view maintenance optimization
- [ ] Recursive Pattern-Matching Logic
- [ ] Codegen: Generate a new optimizer
- [ ] Test: Do these rules generate the same results as spark?
