Ludovic Henry
b52d47a920
[SPARK-35295][ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0
...
### What changes were proposed in this pull request?
Bump to `dev.ludovic.netlib:2.0` which provides JNI-based wrappers for BLAS, ARPACK, and LAPACK. Theseare not taking dependencies on GPL or LGPL libraries, allowing to provide out-of-the-box support for hardware acceleration when a native library is present (this is still up to the end-user to install such library on their system, like OpenBLAS, Intel MKL, and libarpack2).
### Why are the changes needed?
Great performance improvement for ML-related workload on vanilla-distributions of Spark.
### Does this PR introduce _any_ user-facing change?
Users now take advantage of hardware acceleration as long as a native library is installed (like OpenBLAS, Intel MKL and libarpack2).
### How was this patch tested?
Spark test-suite + dev.ludovic.netlib testsuite.
#### JDK8:
```
[info] OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java8BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 220 226 6 454.9 2.2 1.0X
[info] java 221 228 5 451.9 2.2 1.0X
[info] native 209 215 5 478.7 2.1 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 121 125 3 823.3 1.2 1.0X
[info] java 121 125 3 824.3 1.2 1.0X
[info] native 101 105 3 988.4 1.0 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 212 219 6 470.9 2.1 1.0X
[info] java 208 212 4 481.0 2.1 1.0X
[info] native 209 215 5 478.5 2.1 1.0X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 114 119 3 878.9 1.1 1.0X
[info] java 99 105 3 1011.4 1.0 1.2X
[info] native 97 103 3 1026.7 1.0 1.2X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 111 2 925.9 1.1 1.0X
[info] java 71 73 2 1414.9 0.7 1.5X
[info] native 54 56 2 1847.0 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 2 1046.8 1.0 1.0X
[info] java 47 48 1 2129.8 0.5 2.0X
[info] native 29 30 1 3404.7 0.3 3.3X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 139 143 2 718.2 1.4 1.0X
[info] java 46 47 1 2171.2 0.5 3.0X
[info] native 44 46 2 2261.8 0.4 3.1X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 154 157 4 651.0 1.5 1.0X
[info] java 40 42 1 2469.3 0.4 3.8X
[info] native 26 27 1 3787.6 0.3 5.8X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 185 195 8 541.0 1.8 1.0X
[info] java 186 196 7 538.5 1.9 1.0X
[info] native 177 187 7 564.1 1.8 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 98 102 3 1016.2 1.0 1.0X
[info] java 98 102 3 1017.8 1.0 1.0X
[info] native 87 91 3 1143.2 0.9 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 68 70 1 1474.7 0.7 1.0X
[info] java 51 52 1 1973.0 0.5 1.3X
[info] native 30 32 1 3298.8 0.3 2.2X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 99 2 1037.9 1.0 1.0X
[info] java 50 51 1 1999.6 0.5 1.9X
[info] native 30 31 1 3368.1 0.3 3.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 59 61 1 1688.7 0.6 1.0X
[info] java 41 42 1 2461.9 0.4 1.5X
[info] native 15 16 1 6593.0 0.2 3.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 90 92 1 1116.2 0.9 1.0X
[info] java 39 40 1 2565.8 0.4 2.3X
[info] native 15 16 1 6594.2 0.2 5.9X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 192 202 7 520.5 1.9 1.0X
[info] java 203 214 7 491.9 2.0 0.9X
[info] native 176 187 7 568.8 1.8 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 59 61 1 846.1 1.2 1.0X
[info] java 38 39 1 1313.5 0.8 1.6X
[info] native 24 27 1 2047.8 0.5 2.4X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 97 101 3 515.4 1.9 1.0X
[info] java 97 101 2 515.1 1.9 1.0X
[info] native 88 91 3 569.1 1.8 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 169 174 3 295.4 3.4 1.0X
[info] java 169 174 3 295.4 3.4 1.0X
[info] native 160 165 4 312.2 3.2 1.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 561 577 13 1782.3 0.6 1.0X
[info] java 225 231 4 4446.2 0.2 2.5X
[info] native 31 32 3 32473.1 0.0 18.2X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 570 584 9 1754.8 0.6 1.0X
[info] java 224 230 4 4457.3 0.2 2.5X
[info] native 31 32 1 32493.4 0.0 18.5X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 855 866 6 1169.2 0.9 1.0X
[info] java 224 228 3 4466.9 0.2 3.8X
[info] native 31 32 1 32395.5 0.0 27.7X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1328 1344 8 752.8 1.3 1.0X
[info] java 224 230 4 4458.9 0.2 5.9X
[info] native 31 32 1 32201.8 0.0 42.8X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 534 541 5 1873.0 0.5 1.0X
[info] java 220 224 3 4542.8 0.2 2.4X
[info] native 15 16 1 66803.1 0.0 35.7X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 544 551 6 1839.6 0.5 1.0X
[info] java 220 224 4 4538.2 0.2 2.5X
[info] native 15 16 1 65589.9 0.0 35.7X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 833 845 21 1201.0 0.8 1.0X
[info] java 220 224 3 4548.7 0.2 3.8X
[info] native 15 16 1 66603.2 0.0 55.5X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 899 907 5 1112.9 0.9 1.0X
[info] java 221 224 2 4531.6 0.2 4.1X
[info] native 15 16 1 65944.9 0.0 59.3X
```
#### JDK11:
```
[info] OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java11BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 195 200 3 512.2 2.0 1.0X
[info] java 197 202 3 507.0 2.0 1.0X
[info] native 184 189 4 543.0 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 112 3 921.8 1.1 1.0X
[info] java 101 105 3 989.4 1.0 1.1X
[info] native 87 91 3 1147.1 0.9 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 187 191 3 535.1 1.9 1.0X
[info] java 182 188 3 548.8 1.8 1.0X
[info] native 178 182 3 562.2 1.8 1.1X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 110 114 3 909.3 1.1 1.0X
[info] java 86 93 4 1159.3 0.9 1.3X
[info] native 86 90 3 1162.4 0.9 1.3X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 106 108 2 943.6 1.1 1.0X
[info] java 70 71 2 1426.8 0.7 1.5X
[info] native 54 56 2 1835.4 0.5 1.9X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 1 1047.1 1.0 1.0X
[info] java 43 44 1 2331.9 0.4 2.2X
[info] native 29 30 1 3392.1 0.3 3.2X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 114 115 2 880.7 1.1 1.0X
[info] java 42 43 1 2398.1 0.4 2.7X
[info] native 45 46 1 2233.3 0.4 2.5X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 140 143 2 714.6 1.4 1.0X
[info] java 28 29 1 3531.0 0.3 4.9X
[info] native 26 27 1 3820.0 0.3 5.3X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 156 166 7 641.3 1.6 1.0X
[info] java 158 167 6 633.2 1.6 1.0X
[info] native 150 160 7 664.8 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 88 2 1181.7 0.8 1.0X
[info] java 85 88 2 1176.0 0.9 1.0X
[info] native 75 78 2 1333.2 0.8 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 58 59 1 1731.1 0.6 1.0X
[info] java 41 43 1 2415.5 0.4 1.4X
[info] native 30 31 1 3293.9 0.3 1.9X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 94 96 1 1063.4 0.9 1.0X
[info] java 41 42 1 2435.8 0.4 2.3X
[info] native 30 30 1 3379.8 0.3 3.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 44 45 1 2278.9 0.4 1.0X
[info] java 37 38 0 2686.8 0.4 1.2X
[info] native 15 16 1 6555.4 0.2 2.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 88 89 1 1142.1 0.9 1.0X
[info] java 33 34 1 3010.7 0.3 2.6X
[info] native 15 16 1 6553.9 0.2 5.7X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 164 172 4 609.4 1.6 1.0X
[info] java 163 172 5 612.6 1.6 1.0X
[info] native 150 159 4 667.0 1.5 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 49 50 1 1029.4 1.0 1.0X
[info] java 41 42 1 1209.4 0.8 1.2X
[info] native 25 27 1 2029.2 0.5 2.0X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 80 85 3 622.2 1.6 1.0X
[info] java 80 85 3 622.4 1.6 1.0X
[info] native 75 79 3 668.7 1.5 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 137 142 3 364.1 2.7 1.0X
[info] java 139 142 2 360.4 2.8 1.0X
[info] native 131 135 3 380.4 2.6 1.0X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 517 525 5 1935.5 0.5 1.0X
[info] java 213 216 3 4704.8 0.2 2.4X
[info] native 31 31 1 32705.6 0.0 16.9X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 589 601 6 1698.6 0.6 1.0X
[info] java 213 217 3 4693.3 0.2 2.8X
[info] native 31 32 1 32498.9 0.0 19.1X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 851 865 6 1175.3 0.9 1.0X
[info] java 212 216 3 4717.0 0.2 4.0X
[info] native 30 32 1 32903.0 0.0 28.0X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1301 1316 6 768.4 1.3 1.0X
[info] java 212 216 2 4717.4 0.2 6.1X
[info] native 31 32 1 32606.0 0.0 42.4X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 454 460 2 2203.0 0.5 1.0X
[info] java 208 212 3 4803.8 0.2 2.2X
[info] native 15 16 0 66586.0 0.0 30.2X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 529 536 4 1889.7 0.5 1.0X
[info] java 208 212 3 4798.6 0.2 2.5X
[info] native 15 16 1 66751.4 0.0 35.3X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 830 840 5 1205.1 0.8 1.0X
[info] java 208 211 2 4814.1 0.2 4.0X
[info] native 15 15 1 67676.4 0.0 56.2X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 894 907 7 1118.7 0.9 1.0X
[info] java 208 211 3 4809.6 0.2 4.3X
[info] native 15 16 1 66675.2 0.0 59.6X
```
#### JDK16:
```
[info] OpenJDK 64-Bit Server VM 16+36 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.VectorBLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 193 199 3 517.5 1.9 1.0X
[info] java 181 186 4 553.2 1.8 1.1X
[info] native 181 185 5 553.6 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 112 2 925.1 1.1 1.0X
[info] java 88 91 3 1138.6 0.9 1.2X
[info] native 87 91 3 1144.2 0.9 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 184 189 3 542.5 1.8 1.0X
[info] java 181 185 3 552.8 1.8 1.0X
[info] native 179 183 2 558.0 1.8 1.0X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 97 101 3 1031.6 1.0 1.0X
[info] java 86 90 2 1163.7 0.9 1.1X
[info] native 85 88 2 1182.9 0.8 1.1X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 107 109 2 932.4 1.1 1.0X
[info] java 54 56 2 1846.7 0.5 2.0X
[info] native 54 56 2 1846.7 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 1 1043.6 1.0 1.0X
[info] java 29 30 1 3439.3 0.3 3.3X
[info] native 29 30 1 3423.9 0.3 3.3X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 121 123 2 829.8 1.2 1.0X
[info] java 32 32 1 3171.3 0.3 3.8X
[info] native 45 46 1 2246.2 0.4 2.7X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 142 144 2 705.9 1.4 1.0X
[info] java 15 16 1 6585.8 0.2 9.3X
[info] native 26 27 1 3839.5 0.3 5.4X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 157 165 5 635.6 1.6 1.0X
[info] java 151 159 5 664.0 1.5 1.0X
[info] native 151 160 5 663.6 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 89 2 1172.3 0.9 1.0X
[info] java 75 79 3 1337.3 0.7 1.1X
[info] native 75 79 2 1335.5 0.7 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 58 59 1 1731.5 0.6 1.0X
[info] java 28 29 1 3544.2 0.3 2.0X
[info] native 30 31 1 3306.2 0.3 1.9X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 90 92 1 1108.3 0.9 1.0X
[info] java 28 28 1 3622.5 0.3 3.3X
[info] native 30 31 1 3381.3 0.3 3.1X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 44 45 1 2284.7 0.4 1.0X
[info] java 14 15 1 7034.0 0.1 3.1X
[info] native 15 16 1 6643.7 0.2 2.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 86 1 1177.4 0.8 1.0X
[info] java 15 15 1 6886.1 0.1 5.8X
[info] native 15 16 1 6560.1 0.2 5.6X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 164 173 6 608.1 1.6 1.0X
[info] java 148 157 5 675.2 1.5 1.1X
[info] native 152 160 5 659.9 1.5 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 61 63 1 815.4 1.2 1.0X
[info] java 16 17 1 3104.3 0.3 3.8X
[info] native 24 27 1 2071.9 0.5 2.5X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 81 85 2 616.4 1.6 1.0X
[info] java 81 85 2 614.7 1.6 1.0X
[info] native 75 78 2 669.5 1.5 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 138 141 3 362.7 2.8 1.0X
[info] java 137 140 2 365.3 2.7 1.0X
[info] native 131 134 2 382.9 2.6 1.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 525 544 8 1906.2 0.5 1.0X
[info] java 61 68 3 16358.1 0.1 8.6X
[info] native 31 32 1 32623.7 0.0 17.1X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 580 598 12 1724.5 0.6 1.0X
[info] java 61 68 4 16302.5 0.1 9.5X
[info] native 30 32 1 32962.8 0.0 19.1X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 829 838 4 1206.2 0.8 1.0X
[info] java 61 69 3 16339.7 0.1 13.5X
[info] native 30 31 1 33231.9 0.0 27.6X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1352 1363 5 739.6 1.4 1.0X
[info] java 61 69 3 16347.0 0.1 22.1X
[info] native 31 32 1 32740.3 0.0 44.3X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 482 493 7 2073.1 0.5 1.0X
[info] java 35 38 2 28315.3 0.0 13.7X
[info] native 15 15 1 67579.7 0.0 32.6X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 472 482 4 2119.0 0.5 1.0X
[info] java 36 38 2 28138.1 0.0 13.3X
[info] native 15 16 1 66616.5 0.0 31.4X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 823 830 5 1215.2 0.8 1.0X
[info] java 35 38 2 28681.4 0.0 23.6X
[info] native 15 15 1 67908.4 0.0 55.9X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 896 908 7 1115.8 0.9 1.0X
[info] java 35 38 2 28402.0 0.0 25.5X
[info] native 15 16 0 66691.2 0.0 59.8X
```
TODO:
- [x] update documentation in `docs/` and `docs/ml-linalg-guide.md` refering `com.github.fommil.netlib`
- [ ] merge https://github.com/luhenry/netlib/pull/1 with all feedback from this PR + remove references to snapshot repositories in `pom.xml` and `project/SparkBuild.scala`.
Closes #32415 from luhenry/master.
Authored-by: Ludovic Henry <git@ludovic.dev>
Signed-off-by: Sean Owen <srowen@gmail.com>
2021-05-12 08:59:36 -05:00
Ludovic Henry
5b77ebb57b
[SPARK-35150][ML] Accelerate fallback BLAS with dev.ludovic.netlib
...
### What changes were proposed in this pull request?
Following https://github.com/apache/spark/pull/30810 , I've continued looking for ways to accelerate the usage of BLAS in Spark. With this PR, I integrate work done in the [`dev.ludovic.netlib`](https://github.com/luhenry/netlib/ ) Maven package.
The `dev.ludovic.netlib` library wraps the original `com.github.fommil.netlib` library and focus on accelerating the linear algebra routines in use in Spark. When running the `org.apache.spark.ml.linalg.BLASBenchmark` benchmarking suite, I get the results at [1] on an Intel machine. Moreover, this library is thoroughly tested to return the exact same results as the reference implementation.
Under the hood, it reimplements the necessary algorithms in pure autovectorization-friendly Java 8, as well as takes advantage of the Vector API and Foreign Linker API introduced in JDK 16 when available.
A table summarising which version gets loaded in which case:
```
| | BLAS.nativeBLAS | BLAS.javaBLAS |
| --------------------- | -------------------------------------------------- | -------------------------------------------------- |
| with -Pnetlib-lgpl | 1. dev.ludovic.netlib.blas.NetlibNativeBLAS, a | 1. dev.ludovic.netlib.blas.VectorizedBLAS |
| | wrapper for com.github.fommil:all | (JDK16+, relies on the Vector API, requires |
| | 2. dev.ludovic.netlib.blas.ForeignBLAS (JDK16+, | `--add-modules=jdk.incubator.vector` on JDK16) |
| | relies on the Foreign Linker API, requires | 2. dev.ludovic.netlib.blas.Java11BLAS (JDK11+) |
| | `--add-modules=jdk.incubator.foreign | 3. dev.ludovic.netlib.blas.JavaBLAS |
| | -Dforeign.restricted=warn`) | 4. dev.ludovic.netlib.blas.NetlibF2jBLAS, a |
| | 3. fails to load, falls back to BLAS.javaBLAS in | wrapper for com.github.fommil:core |
| | org.apache.spark.ml.linalg.BLAS | |
| --------------------- | -------------------------------------------------- | -------------------------------------------------- |
| without -Pnetlib-lgpl | 1. dev.ludovic.netlib.blas.ForeignBLAS (JDK16+, | 1. dev.ludovic.netlib.blas.VectorizedBLAS |
| | relies on the Foreign Linker API, requires | (JDK16+, relies on the Vector API, requires |
| | `--add-modules=jdk.incubator.foreign | `--add-modules=jdk.incubator.vector` on JDK16) |
| | -Dforeign.restricted=warn`) | 2. dev.ludovic.netlib.blas.Java11BLAS (JDK11+) |
| | 2. fails to load, falls back to BLAS.javaBLAS in | 3. dev.ludovic.netlib.blas.JavaBLAS |
| | org.apache.spark.ml.linalg.BLAS | 4. dev.ludovic.netlib.blas.NetlibF2jBLAS, a |
| | | wrapper for com.github.fommil:core |
| --------------------- | -------------------------------------------------- | -------------------------------------------------- |
```
### Why are the changes needed?
Accelerates linear algebra operations when the pure-java fallback method is in use. Transparently falls back to native implementation (OpenBLAS, MKL) when available.
### Does this PR introduce _any_ user-facing change?
No, all changes are transparent to the user.
### How was this patch tested?
The `dev.ludovic.netlib` library has its own test suite [2]. It has also been validated by running the Spark test suite and benchmarking suite.
[1] Results for `org.apache.spark.ml.linalg.BLASBenchmark`:
#### JDK8:
```
[info] OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.NetlibF2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java8BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.Java8BLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 223 232 8 448.0 2.2 1.0X
[info] java 221 228 7 453.0 2.2 1.0X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 122 128 4 821.2 1.2 1.0X
[info] java 122 128 4 822.3 1.2 1.0X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 109 112 2 921.4 1.1 1.0X
[info] java 70 74 3 1423.5 0.7 1.5X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 96 98 2 1046.1 1.0 1.0X
[info] java 47 49 2 2121.7 0.5 2.0X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 184 195 8 544.3 1.8 1.0X
[info] java 185 196 7 539.5 1.9 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 99 104 4 1011.9 1.0 1.0X
[info] java 99 104 4 1010.4 1.0 1.0X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 947.2 1.1 1.0X
[info] java 0 0 0 1584.8 0.6 1.7X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 867.4 1.2 1.0X
[info] java 1 1 0 865.0 1.2 1.0X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 485.9 2.1 1.0X
[info] java 1 1 0 486.8 2.1 1.0X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 1843.0 0.5 1.0X
[info] java 0 0 0 2690.6 0.4 1.5X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 1214.7 0.8 1.0X
[info] java 0 0 0 2536.8 0.4 2.1X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 1895.9 0.5 1.0X
[info] java 0 0 0 2961.1 0.3 1.6X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 1223.4 0.8 1.0X
[info] java 0 0 0 3091.4 0.3 2.5X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 560 575 20 1787.1 0.6 1.0X
[info] java 226 232 5 4432.4 0.2 2.5X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 570 586 23 1755.2 0.6 1.0X
[info] java 227 232 4 4410.1 0.2 2.5X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 863 879 17 1158.4 0.9 1.0X
[info] java 227 231 3 4407.9 0.2 3.8X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1282 1305 23 780.0 1.3 1.0X
[info] java 227 232 4 4413.4 0.2 5.7X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 538 548 8 1858.6 0.5 1.0X
[info] java 221 226 3 4521.1 0.2 2.4X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 549 558 10 1819.9 0.5 1.0X
[info] java 222 229 7 4503.5 0.2 2.5X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 838 852 12 1193.0 0.8 1.0X
[info] java 222 229 5 4500.5 0.2 3.8X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 905 919 18 1104.8 0.9 1.0X
[info] java 221 228 5 4521.3 0.2 4.1X
```
#### JDK11:
```
[info] OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.NetlibF2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java11BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.Java11BLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 195 204 10 512.7 2.0 1.0X
[info] java 195 202 7 512.4 2.0 1.0X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 108 113 4 923.3 1.1 1.0X
[info] java 102 107 4 984.4 1.0 1.1X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 107 110 3 938.1 1.1 1.0X
[info] java 69 72 3 1447.1 0.7 1.5X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 96 98 2 1046.5 1.0 1.0X
[info] java 43 45 2 2317.1 0.4 2.2X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 155 168 8 644.2 1.6 1.0X
[info] java 158 169 8 632.8 1.6 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 85 90 4 1178.1 0.8 1.0X
[info] java 86 90 4 1167.7 0.9 1.0X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 0 0 0 1182.1 0.8 1.0X
[info] java 0 0 0 1432.1 0.7 1.2X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 898.7 1.1 1.0X
[info] java 1 1 0 891.5 1.1 1.0X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 495.4 2.0 1.0X
[info] java 1 1 0 495.7 2.0 1.0X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 0 0 0 2271.6 0.4 1.0X
[info] java 0 0 0 3648.1 0.3 1.6X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 1229.3 0.8 1.0X
[info] java 0 0 0 2711.3 0.4 2.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 0 0 0 2677.5 0.4 1.0X
[info] java 0 0 0 3288.2 0.3 1.2X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 1233.0 0.8 1.0X
[info] java 0 0 0 2766.3 0.4 2.2X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 520 536 16 1923.6 0.5 1.0X
[info] java 214 221 7 4669.5 0.2 2.4X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 593 612 17 1686.5 0.6 1.0X
[info] java 215 219 3 4643.3 0.2 2.8X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 853 870 16 1172.8 0.9 1.0X
[info] java 215 218 3 4659.7 0.2 4.0X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1350 1370 23 740.8 1.3 1.0X
[info] java 215 219 4 4656.6 0.2 6.3X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 460 468 6 2173.2 0.5 1.0X
[info] java 210 213 2 4752.7 0.2 2.2X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 535 544 8 1869.3 0.5 1.0X
[info] java 210 215 5 4761.8 0.2 2.5X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 843 853 11 1186.8 0.8 1.0X
[info] java 209 214 4 4793.4 0.2 4.0X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 891 904 15 1122.0 0.9 1.0X
[info] java 209 214 4 4777.2 0.2 4.3X
```
#### JDK16:
```
[info] OpenJDK 64-Bit Server VM 16+36 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.NetlibF2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.VectorizedBLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.VectorizedBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 194 199 7 515.7 1.9 1.0X
[info] java 181 186 3 551.1 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 109 115 4 915.0 1.1 1.0X
[info] java 88 92 3 1138.8 0.9 1.2X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 108 110 2 922.6 1.1 1.0X
[info] java 54 56 2 1839.2 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 96 97 2 1046.1 1.0 1.0X
[info] java 29 30 1 3393.4 0.3 3.2X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 156 165 5 643.0 1.6 1.0X
[info] java 150 159 5 667.1 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 85 91 6 1171.0 0.9 1.0X
[info] java 75 79 3 1340.6 0.7 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 917.0 1.1 1.0X
[info] java 0 0 0 8147.2 0.1 8.9X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 859.3 1.2 1.0X
[info] java 1 1 0 859.3 1.2 1.0X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 482.1 2.1 1.0X
[info] java 1 1 0 482.6 2.1 1.0X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 0 0 0 2214.2 0.5 1.0X
[info] java 0 0 0 7975.8 0.1 3.6X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 1231.4 0.8 1.0X
[info] java 0 0 0 8680.9 0.1 7.0X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 0 0 0 2684.3 0.4 1.0X
[info] java 0 0 0 18527.1 0.1 6.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1 1 0 1235.4 0.8 1.0X
[info] java 0 0 0 17347.9 0.1 14.0X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 530 552 18 1887.5 0.5 1.0X
[info] java 58 64 3 17143.9 0.1 9.1X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 598 620 17 1671.1 0.6 1.0X
[info] java 58 64 3 17196.6 0.1 10.3X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 834 847 14 1199.4 0.8 1.0X
[info] java 57 63 4 17486.9 0.1 14.6X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 1338 1366 22 747.3 1.3 1.0X
[info] java 58 63 3 17356.6 0.1 23.2X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 489 501 9 2045.5 0.5 1.0X
[info] java 36 38 2 27721.9 0.0 13.6X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 478 488 9 2094.0 0.5 1.0X
[info] java 36 38 2 27813.2 0.0 13.3X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 825 837 10 1211.6 0.8 1.0X
[info] java 35 38 2 28433.1 0.0 23.5X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 900 918 15 1111.6 0.9 1.0X
[info] java 36 38 2 28073.0 0.0 25.3X
```
[2] https://github.com/luhenry/netlib/tree/master/blas/src/test/java/dev/ludovic/netlib/blas
Closes #32253 from luhenry/master.
Authored-by: Ludovic Henry <git@ludovic.dev>
Signed-off-by: Sean Owen <srowen@gmail.com>
2021-04-27 14:00:59 -05:00
Ludovic Henry
9244066ca6
[SPARK-33882][ML] Add a vectorized BLAS implementation
...
### What changes were proposed in this pull request?
This patch introduces a VectorizedBLAS class which implements such hardware-accelerated BLAS operations. This feature is hidden behind the "vectorized" profile that you can enable by passing "-Pvectorized" to sbt or maven.
The Vector API has been introduced in JDK 16. Following discussion on the mailing list, this API is introduced transparently and needs to be enabled explicitely.
### Why are the changes needed?
Whenever a native BLAS implementation isn't available on the system, Spark automatically falls back onto a Java implementation. With the recent release of the Vector API in the OpenJDK [1], we can use hardware acceleration for such operations.
This change was also discussed on the mailing list. [2]
### Does this PR introduce _any_ user-facing change?
It introduces a build-time profile called `vectorized`. You can pass it to sbt and mvn with `-Pvectorized`. There is no change to the end-user of Spark and it should only impact Spark developpers. It is also disabled by default.
### How was this patch tested?
It passes `build/sbt mllib-local/test` with and without `-Pvectorized` with JDK 16. This patch also introduces benchmarks for BLAS.
The benchmark results are as follows:
```
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 37 37 0 271.5 3.7 1.0X
[info] vector 24 25 4 416.1 2.4 1.5X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 70 70 0 143.2 7.0 1.0X
[info] vector 35 35 2 288.7 3.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 50 51 1 199.8 5.0 1.0X
[info] vector 15 15 0 648.7 1.5 3.2X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 34 34 0 295.6 3.4 1.0X
[info] vector 19 19 0 531.2 1.9 1.8X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 25 25 1 399.0 2.5 1.0X
[info] vector 8 9 1 1177.3 0.8 3.0X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 27 27 0 0.0 26651.5 1.0X
[info] vector 21 21 0 0.0 20646.3 1.3X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 36 36 0 0.0 35501.4 1.0X
[info] vector 22 22 0 0.0 21930.3 1.6X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 20 20 0 0.0 20283.3 1.0X
[info] vector 9 9 0 0.1 8657.7 2.3X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 30 30 0 0.0 29845.8 1.0X
[info] vector 10 10 1 0.1 9695.4 3.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 182 182 0 0.5 1820.0 1.0X
[info] vector 160 160 1 0.6 1597.6 1.1X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 211 211 1 0.5 2106.2 1.0X
[info] vector 156 157 0 0.6 1564.4 1.3X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] f2j 276 276 0 0.4 2757.8 1.0X
[info] vector 137 137 0 0.7 1365.1 2.0X
```
/cc srowen xkrogen
[1] https://openjdk.java.net/jeps/338
[2] https://mail-archives.apache.org/mod_mbox/spark-dev/202012.mbox/%3cDM5PR2101MB11106162BB3AF32AD29C6C79B0C69DM5PR2101MB1110.namprd21.prod.outlook.com%3e
Closes #30810 from luhenry/master.
Lead-authored-by: Ludovic Henry <luhenry@microsoft.com>
Co-authored-by: Ludovic Henry <git@ludovic.dev>
Signed-off-by: Sean Owen <srowen@gmail.com>
2021-04-14 11:36:58 -05:00
Dongjoon Hyun
de9818f043
[SPARK-33662][BUILD] Setting version to 3.2.0-SNAPSHOT
...
### What changes were proposed in this pull request?
This PR aims to update `master` branch version to 3.2.0-SNAPSHOT.
### Why are the changes needed?
Start to prepare Apache Spark 3.2.0.
### Does this PR introduce _any_ user-facing change?
N/A.
### How was this patch tested?
Pass the CIs.
Closes #30606 from dongjoon-hyun/SPARK-3.2.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
2020-12-04 14:10:42 -08:00
gatorsmile
28b8713036
[SPARK-30950][BUILD] Setting version to 3.1.0-SNAPSHOT
...
### What changes were proposed in this pull request?
This patch is to bump the master branch version to 3.1.0-SNAPSHOT.
### Why are the changes needed?
N/A
### Does this PR introduce any user-facing change?
N/A
### How was this patch tested?
N/A
Closes #27698 from gatorsmile/updateVersion.
Authored-by: gatorsmile <gatorsmile@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-02-25 19:44:31 -08:00
Yuming Wang
696288f623
[INFRA] Reverts commit 56dcd79 and c216ef1
...
### What changes were proposed in this pull request?
1. Revert "Preparing development version 3.0.1-SNAPSHOT": 56dcd79c216ef1https://github.com/apache/spark/compare/5de5e46..wangyum:revert-master
Closes #26915 from wangyum/revert-master.
Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Yuming Wang <wgyumg@gmail.com>
2019-12-16 19:57:44 -07:00
Yuming Wang
56dcd79992
Preparing development version 3.0.1-SNAPSHOT
2019-12-17 01:57:27 +00:00
Yuming Wang
c216ef1d03
Preparing Spark release v3.0.0-preview2-rc2
2019-12-17 01:57:21 +00:00
Xingbo Jiang
8207c835b4
Revert "Prepare Spark release v3.0.0-preview-rc2"
...
This reverts commit 007c873ae3
2019-10-30 17:45:44 -07:00
Xingbo Jiang
007c873ae3
Prepare Spark release v3.0.0-preview-rc2
...
### What changes were proposed in this pull request?
To push the built jars to maven release repository, we need to remove the 'SNAPSHOT' tag from the version name.
Made the following changes in this PR:
* Update all the `3.0.0-SNAPSHOT` version name to `3.0.0-preview`
* Update the sparkR version number check logic to allow jvm version like `3.0.0-preview`
**Please note those changes were generated by the release script in the past, but this time since we manually add tags on master branch, we need to manually apply those changes too.**
We shall revert the changes after 3.0.0-preview release passed.
### Why are the changes needed?
To make the maven release repository to accept the built jars.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
N/A
2019-10-30 17:42:59 -07:00
Xingbo Jiang
b33a58c0c6
Revert "Prepare Spark release v3.0.0-preview-rc1"
...
This reverts commit 5eddbb5f1d
2019-10-28 22:32:34 -07:00
Xingbo Jiang
5eddbb5f1d
Prepare Spark release v3.0.0-preview-rc1
...
### What changes were proposed in this pull request?
To push the built jars to maven release repository, we need to remove the 'SNAPSHOT' tag from the version name.
Made the following changes in this PR:
* Update all the `3.0.0-SNAPSHOT` version name to `3.0.0-preview`
* Update the PySpark version from `3.0.0.dev0` to `3.0.0`
**Please note those changes were generated by the release script in the past, but this time since we manually add tags on master branch, we need to manually apply those changes too.**
We shall revert the changes after 3.0.0-preview release passed.
### Why are the changes needed?
To make the maven release repository to accept the built jars.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
N/A
Closes #26243 from jiangxb1987/3.0.0-preview-prepare.
Lead-authored-by: Xingbo Jiang <xingbo.jiang@databricks.com>
Co-authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: Xingbo Jiang <xingbo.jiang@databricks.com>
2019-10-28 22:31:29 -07:00
DB Tsai
ad853c5678
[SPARK-25956] Make Scala 2.12 as default Scala version in Spark 3.0
...
## What changes were proposed in this pull request?
This PR makes Spark's default Scala version as 2.12, and Scala 2.11 will be the alternative version. This implies that Scala 2.12 will be used by our CI builds including pull request builds.
We'll update the Jenkins to include a new compile-only jobs for Scala 2.11 to ensure the code can be still compiled with Scala 2.11.
## How was this patch tested?
existing tests
Closes #22967 from dbtsai/scala2.12.
Authored-by: DB Tsai <d_tsai@apple.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
2018-11-14 16:22:23 -08:00
gatorsmile
9bf397c0e4
[SPARK-25592] Setting version to 3.0.0-SNAPSHOT
...
## What changes were proposed in this pull request?
This patch is to bump the master branch version to 3.0.0-SNAPSHOT.
## How was this patch tested?
N/A
Closes #22606 from gatorsmile/bump3.0.
Authored-by: gatorsmile <gatorsmile@gmail.com>
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
2018-10-02 08:48:24 -07:00
gatorsmile
bb2f069cf2
[SPARK-25436] Bump master branch version to 2.5.0-SNAPSHOT
...
## What changes were proposed in this pull request?
In the dev list, we can still discuss whether the next version is 2.5.0 or 3.0.0. Let us first bump the master branch version to `2.5.0-SNAPSHOT`.
## How was this patch tested?
N/A
Closes #22426 from gatorsmile/bumpVersionMaster.
Authored-by: gatorsmile <gatorsmile@gmail.com>
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
2018-09-15 16:24:02 -07:00
gatorsmile
651f76153f
[SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT
...
## What changes were proposed in this pull request?
This patch bumps the master branch version to `2.4.0-SNAPSHOT`.
## How was this patch tested?
N/A
Author: gatorsmile <gatorsmile@gmail.com>
Closes #20222 from gatorsmile/bump24.
2018-01-13 00:37:59 +08:00
Josh Rosen
f44c8a843c
[SPARK-20453] Bump master branch version to 2.3.0-SNAPSHOT
...
This patch bumps the master branch version to `2.3.0-SNAPSHOT`.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #17753 from JoshRosen/SPARK-20453.
2017-04-24 21:48:04 -07:00
Ryan Williams
afd9bc1d8a
[SPARK-17807][CORE] split test-tags into test-JAR
...
Remove spark-tag's compile-scope dependency (and, indirectly, spark-core's compile-scope transitive-dependency) on scalatest by splitting test-oriented tags into spark-tags' test JAR.
Alternative to #16303 .
Author: Ryan Williams <ryan.blake.williams@gmail.com>
Closes #16311 from ryan-williams/tt.
2016-12-21 16:37:20 -08:00
Reynold Xin
c7c7265950
[SPARK-18695] Bump master branch version to 2.2.0-SNAPSHOT
...
## What changes were proposed in this pull request?
This patch bumps master branch version to 2.2.0-SNAPSHOT.
## How was this patch tested?
N/A
Author: Reynold Xin <rxin@databricks.com>
Closes #16126 from rxin/SPARK-18695.
2016-12-02 21:09:37 -08:00
Xin Ren
21a6dd2aef
[SPARK-16535][BUILD] In pom.xml, remove groupId which is redundant definition and inherited from the parent
...
https://issues.apache.org/jira/browse/SPARK-16535
## What changes were proposed in this pull request?
When I scan through the pom.xml of sub projects, I found this warning as below and attached screenshot
```
Definition of groupId is redundant, because it's inherited from the parent
```
I've tried to remove some of the lines with groupId definition, and the build on my local machine is still ok.
```
<groupId>org.apache.spark</groupId>
```
As I just find now `<maven.version>3.3.9</maven.version>` is being used in Spark 2.x, and Maven-3 supports versionless parent elements: Maven 3 will remove the need to specify the parent version in sub modules. THIS is great (in Maven 3.1).
ref: http://stackoverflow.com/questions/3157240/maven-3-worth-it/3166762#3166762
## How was this patch tested?
I've tested by re-building the project, and build succeeded.
Author: Xin Ren <iamshrek@126.com>
Closes #14189 from keypointt/SPARK-16535.
2016-07-19 11:59:46 +01:00
Reynold Xin
ffcb6e055a
[SPARK-16477] Bump master version to 2.1.0-SNAPSHOT
...
## What changes were proposed in this pull request?
After SPARK-16476 (committed earlier today as #14128 ), we can finally bump the version number.
## How was this patch tested?
N/A
Author: Reynold Xin <rxin@databricks.com>
Closes #14130 from rxin/SPARK-16477.
2016-07-11 09:42:56 -07:00
Sean Owen
122302cbf5
[SPARK-15290][BUILD] Move annotations, like @Since / @DeveloperApi, into spark-tags
...
## What changes were proposed in this pull request?
(See https://github.com/apache/spark/pull/12416 where most of this was already reviewed and committed; this is just the module structure and move part. This change does not move the annotations into test scope, which was the apparently problem last time.)
Rename `spark-test-tags` -> `spark-tags`; move common annotations like `Since` to `spark-tags`
## How was this patch tested?
Jenkins tests.
Author: Sean Owen <sowen@cloudera.com>
Closes #13074 from srowen/SPARK-15290.
2016-05-17 09:55:53 +01:00
Xiangrui Meng
0847fe4eb3
[SPARK-14653][ML] Remove json4s from mllib-local
...
## What changes were proposed in this pull request?
This PR moves Vector.toJson/fromJson to ml.linalg.VectorEncoder under mllib/ to keep mllib-local's dependency minimal. The json encoding is used by Params. So we still need this feature in SPARK-14615, where we will switch to ml.linalg in spark.ml APIs.
## How was this patch tested?
Copied existing unit tests over.
cc; dbtsai
Author: Xiangrui Meng <meng@databricks.com>
Closes #12802 from mengxr/SPARK-14653.
2016-04-30 06:30:39 -07:00
Yin Huai
9c7c42bc6a
Revert "[SPARK-14613][ML] Add @Since into the matrix and vector classes in spark-mllib-local"
...
This reverts commit dae538a4d7
2016-04-28 19:57:41 -07:00
Pravin Gadakh
dae538a4d7
[SPARK-14613][ML] Add @Since into the matrix and vector classes in spark-mllib-local
...
## What changes were proposed in this pull request?
This PR adds `since` tag into the matrix and vector classes in spark-mllib-local.
## How was this patch tested?
Scala-style checks passed.
Author: Pravin Gadakh <prgadakh@in.ibm.com>
Closes #12416 from pravingadakh/SPARK-14613.
2016-04-28 15:59:18 -07:00
DB Tsai
96534aa47c
[SPARK-14549][ML] Copy the Vector and Matrix classes from mllib to ml in mllib-local
...
## What changes were proposed in this pull request?
This task will copy the Vector and Matrix classes from mllib to ml package in mllib-local jar. The UDTs and `since` annotation in ml vector and matrix will be removed from now. UDTs will be achieved by #SPARK-14487, and `since` will be replaced by /* since 1.2.0 */
The BLAS implementation will be copied, and some of the test utilities will be copies as well.
Summary of changes:
1. In mllib-local/src/main/scala/org/apache/spark/**ml**/linalg/BLAS.scala
- Copied from mllib/src/main/scala/org/apache/spark/**mllib**/linalg/BLAS.scala
- logDebug("gemm: alpha is equal to 0 and beta is equal to 1. Returning C.") is removed in ml version.
2. In mllib-local/src/main/scala/org/apache/spark/**ml**/linalg/Matrices.scala
- Copied from mllib/src/main/scala/org/apache/spark/**mllib**/linalg/Matrices.scala
- `Since` was removed, and we'll use standard `/* Since /*` Java doc. Will be in another PR.
- `UDT` related code was removed, and will use `SPARK-13944` https://github.com/apache/spark/pull/12259 to replace the annotation.
3. In mllib-local/src/main/scala/org/apache/spark/**ml**/linalg/Vectors.scala
- Copied from mllib/src/main/scala/org/apache/spark/**mllib**/linalg/Vectors.scala
- `Since` was removed.
- `UDT` related code was removed.
- In `def parseNumeric`, it was throwing `throw new SparkException(s"Cannot parse $other.")`, and now it's throwing `throw new IllegalArgumentException(s"Cannot parse $other.")`
4. In mllib/src/main/scala/org/apache/spark/**mllib**/linalg/Vectors.scala
- For consistency with ML version of vector, `def parseNumeric` is now throwing `throw new IllegalArgumentException(s"Cannot parse $other.")`
5. mllib/src/main/scala/org/apache/spark/**mllib**/util/NumericParser.scala is moved to mllib-local/src/main/scala/org/apache/spark/**ml**/util/NumericParser.scala
- All the `throw new SparkException` were replaced by `throw new IllegalArgumentException`
## How was this patch tested?
unit tests
Author: DB Tsai <dbt@netflix.com>
Closes #12317 from dbtsai/dbtsai-ml-vector.
2016-04-15 01:17:03 -07:00
Sean Owen
9fa43a33b9
[SPARK-14612][ML] Consolidate the version of dependencies in mllib and mllib-local into one place
...
## What changes were proposed in this pull request?
Move json4s, breeze dependency declaration into parent
## How was this patch tested?
Should be no functional change, but Jenkins tests will test that.
Author: Sean Owen <sowen@cloudera.com>
Closes #12390 from srowen/SPARK-14612.
2016-04-14 10:48:17 -07:00
DB Tsai
efaf7d1820
[SPARK-14462][ML][MLLIB] Add the mllib-local build to maven pom
...
## What changes were proposed in this pull request?
In order to separate the linear algebra, and vector matrix classes into a standalone jar, we need to setup the build first. This PR will create a new jar called mllib-local with minimal dependencies.
The previous PR was failing the build because of `spark-core:test` dependency, and that was reverted. In this PR, `FunSuite` with `// scalastyle:ignore funsuite` in mllib-local test was used, similar to sketch.
Thanks.
## How was this patch tested?
Unit tests
mengxr tedyu holdenk
Author: DB Tsai <dbt@netflix.com>
Closes #12298 from dbtsai/dbtsai-mllib-local-build-fix.
2016-04-11 09:35:47 -07:00
Xiangrui Meng
415446cc9b
Revert "[SPARK-14462][ML][MLLIB] add the mllib-local build to maven pom"
...
This reverts commit 1598d11bb0
2016-04-09 14:03:03 -07:00
DB Tsai
1598d11bb0
[SPARK-14462][ML][MLLIB] add the mllib-local build to maven pom
...
## What changes were proposed in this pull request?
In order to separate the linear algebra, and vector matrix classes into a standalone jar, we need to setup the build first. This PR will create a new jar called mllib-local with minimal dependencies. The test scope will still depend on spark-core and spark-core-test in order to use the common utilities, but the runtime will avoid any platform dependency. Couple platform independent classes will be moved to this package to demonstrate how this work.
## How was this patch tested?
Unit tests
Author: DB Tsai <dbt@netflix.com>
Closes #12241 from dbtsai/dbtsai-mllib-local-build.
2016-04-09 09:21:12 -07:00
Ludovic Henry
Ludovic Henry
Dongjoon Hyun
gatorsmile
Yuming Wang
Yuming Wang
Xingbo Jiang
DB Tsai
Josh Rosen
Ryan Williams
Reynold Xin
Xin Ren
Sean Owen
Xiangrui Meng
Yin Huai
Pravin Gadakh
DB Tsai
