[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
HikariCP/2.5.1//HikariCP-2.5.1.jar
|
|
|
|
|
JLargeArrays/1.5//JLargeArrays-1.5.jar
|
|
|
|
|
JTransforms/3.1//JTransforms-3.1.jar
|
2020-07-25 13:58:25 -04:00
|
|
|
RoaringBitmap/0.9.0//RoaringBitmap-0.9.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
ST4/4.0.4//ST4-4.0.4.jar
|
|
|
|
|
activation/1.1.1//activation-1.1.1.jar
|
2020-12-10 22:15:01 -05:00
|
|
|
aircompressor/0.16//aircompressor-0.16.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
algebra_2.12/2.0.0-M2//algebra_2.12-2.0.0-M2.jar
|
2020-12-10 22:15:01 -05:00
|
|
|
annotations/17.0.0//annotations-17.0.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar
|
2021-05-20 04:24:40 -04:00
|
|
|
antlr4-runtime/4.8//antlr4-runtime-4.8.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
aopalliance-repackaged/2.6.1//aopalliance-repackaged-2.6.1.jar
|
[SPARK-35295][ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0
### What changes were proposed in this pull request?
Bump to `dev.ludovic.netlib:2.0` which provides JNI-based wrappers for BLAS, ARPACK, and LAPACK. Theseare not taking dependencies on GPL or LGPL libraries, allowing to provide out-of-the-box support for hardware acceleration when a native library is present (this is still up to the end-user to install such library on their system, like OpenBLAS, Intel MKL, and libarpack2).
### Why are the changes needed?
Great performance improvement for ML-related workload on vanilla-distributions of Spark.
### Does this PR introduce _any_ user-facing change?
Users now take advantage of hardware acceleration as long as a native library is installed (like OpenBLAS, Intel MKL and libarpack2).
### How was this patch tested?
Spark test-suite + dev.ludovic.netlib testsuite.
#### JDK8:
```
[info] OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java8BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 220 226 6 454.9 2.2 1.0X
[info] java 221 228 5 451.9 2.2 1.0X
[info] native 209 215 5 478.7 2.1 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 121 125 3 823.3 1.2 1.0X
[info] java 121 125 3 824.3 1.2 1.0X
[info] native 101 105 3 988.4 1.0 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 212 219 6 470.9 2.1 1.0X
[info] java 208 212 4 481.0 2.1 1.0X
[info] native 209 215 5 478.5 2.1 1.0X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 114 119 3 878.9 1.1 1.0X
[info] java 99 105 3 1011.4 1.0 1.2X
[info] native 97 103 3 1026.7 1.0 1.2X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 111 2 925.9 1.1 1.0X
[info] java 71 73 2 1414.9 0.7 1.5X
[info] native 54 56 2 1847.0 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 2 1046.8 1.0 1.0X
[info] java 47 48 1 2129.8 0.5 2.0X
[info] native 29 30 1 3404.7 0.3 3.3X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 139 143 2 718.2 1.4 1.0X
[info] java 46 47 1 2171.2 0.5 3.0X
[info] native 44 46 2 2261.8 0.4 3.1X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 154 157 4 651.0 1.5 1.0X
[info] java 40 42 1 2469.3 0.4 3.8X
[info] native 26 27 1 3787.6 0.3 5.8X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 185 195 8 541.0 1.8 1.0X
[info] java 186 196 7 538.5 1.9 1.0X
[info] native 177 187 7 564.1 1.8 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 98 102 3 1016.2 1.0 1.0X
[info] java 98 102 3 1017.8 1.0 1.0X
[info] native 87 91 3 1143.2 0.9 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 68 70 1 1474.7 0.7 1.0X
[info] java 51 52 1 1973.0 0.5 1.3X
[info] native 30 32 1 3298.8 0.3 2.2X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 99 2 1037.9 1.0 1.0X
[info] java 50 51 1 1999.6 0.5 1.9X
[info] native 30 31 1 3368.1 0.3 3.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 59 61 1 1688.7 0.6 1.0X
[info] java 41 42 1 2461.9 0.4 1.5X
[info] native 15 16 1 6593.0 0.2 3.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 90 92 1 1116.2 0.9 1.0X
[info] java 39 40 1 2565.8 0.4 2.3X
[info] native 15 16 1 6594.2 0.2 5.9X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 192 202 7 520.5 1.9 1.0X
[info] java 203 214 7 491.9 2.0 0.9X
[info] native 176 187 7 568.8 1.8 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 59 61 1 846.1 1.2 1.0X
[info] java 38 39 1 1313.5 0.8 1.6X
[info] native 24 27 1 2047.8 0.5 2.4X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 97 101 3 515.4 1.9 1.0X
[info] java 97 101 2 515.1 1.9 1.0X
[info] native 88 91 3 569.1 1.8 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 169 174 3 295.4 3.4 1.0X
[info] java 169 174 3 295.4 3.4 1.0X
[info] native 160 165 4 312.2 3.2 1.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 561 577 13 1782.3 0.6 1.0X
[info] java 225 231 4 4446.2 0.2 2.5X
[info] native 31 32 3 32473.1 0.0 18.2X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 570 584 9 1754.8 0.6 1.0X
[info] java 224 230 4 4457.3 0.2 2.5X
[info] native 31 32 1 32493.4 0.0 18.5X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 855 866 6 1169.2 0.9 1.0X
[info] java 224 228 3 4466.9 0.2 3.8X
[info] native 31 32 1 32395.5 0.0 27.7X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1328 1344 8 752.8 1.3 1.0X
[info] java 224 230 4 4458.9 0.2 5.9X
[info] native 31 32 1 32201.8 0.0 42.8X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 534 541 5 1873.0 0.5 1.0X
[info] java 220 224 3 4542.8 0.2 2.4X
[info] native 15 16 1 66803.1 0.0 35.7X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 544 551 6 1839.6 0.5 1.0X
[info] java 220 224 4 4538.2 0.2 2.5X
[info] native 15 16 1 65589.9 0.0 35.7X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 833 845 21 1201.0 0.8 1.0X
[info] java 220 224 3 4548.7 0.2 3.8X
[info] native 15 16 1 66603.2 0.0 55.5X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 899 907 5 1112.9 0.9 1.0X
[info] java 221 224 2 4531.6 0.2 4.1X
[info] native 15 16 1 65944.9 0.0 59.3X
```
#### JDK11:
```
[info] OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java11BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 195 200 3 512.2 2.0 1.0X
[info] java 197 202 3 507.0 2.0 1.0X
[info] native 184 189 4 543.0 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 112 3 921.8 1.1 1.0X
[info] java 101 105 3 989.4 1.0 1.1X
[info] native 87 91 3 1147.1 0.9 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 187 191 3 535.1 1.9 1.0X
[info] java 182 188 3 548.8 1.8 1.0X
[info] native 178 182 3 562.2 1.8 1.1X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 110 114 3 909.3 1.1 1.0X
[info] java 86 93 4 1159.3 0.9 1.3X
[info] native 86 90 3 1162.4 0.9 1.3X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 106 108 2 943.6 1.1 1.0X
[info] java 70 71 2 1426.8 0.7 1.5X
[info] native 54 56 2 1835.4 0.5 1.9X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 1 1047.1 1.0 1.0X
[info] java 43 44 1 2331.9 0.4 2.2X
[info] native 29 30 1 3392.1 0.3 3.2X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 114 115 2 880.7 1.1 1.0X
[info] java 42 43 1 2398.1 0.4 2.7X
[info] native 45 46 1 2233.3 0.4 2.5X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 140 143 2 714.6 1.4 1.0X
[info] java 28 29 1 3531.0 0.3 4.9X
[info] native 26 27 1 3820.0 0.3 5.3X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 156 166 7 641.3 1.6 1.0X
[info] java 158 167 6 633.2 1.6 1.0X
[info] native 150 160 7 664.8 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 88 2 1181.7 0.8 1.0X
[info] java 85 88 2 1176.0 0.9 1.0X
[info] native 75 78 2 1333.2 0.8 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 58 59 1 1731.1 0.6 1.0X
[info] java 41 43 1 2415.5 0.4 1.4X
[info] native 30 31 1 3293.9 0.3 1.9X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 94 96 1 1063.4 0.9 1.0X
[info] java 41 42 1 2435.8 0.4 2.3X
[info] native 30 30 1 3379.8 0.3 3.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 44 45 1 2278.9 0.4 1.0X
[info] java 37 38 0 2686.8 0.4 1.2X
[info] native 15 16 1 6555.4 0.2 2.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 88 89 1 1142.1 0.9 1.0X
[info] java 33 34 1 3010.7 0.3 2.6X
[info] native 15 16 1 6553.9 0.2 5.7X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 164 172 4 609.4 1.6 1.0X
[info] java 163 172 5 612.6 1.6 1.0X
[info] native 150 159 4 667.0 1.5 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 49 50 1 1029.4 1.0 1.0X
[info] java 41 42 1 1209.4 0.8 1.2X
[info] native 25 27 1 2029.2 0.5 2.0X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 80 85 3 622.2 1.6 1.0X
[info] java 80 85 3 622.4 1.6 1.0X
[info] native 75 79 3 668.7 1.5 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 137 142 3 364.1 2.7 1.0X
[info] java 139 142 2 360.4 2.8 1.0X
[info] native 131 135 3 380.4 2.6 1.0X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 517 525 5 1935.5 0.5 1.0X
[info] java 213 216 3 4704.8 0.2 2.4X
[info] native 31 31 1 32705.6 0.0 16.9X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 589 601 6 1698.6 0.6 1.0X
[info] java 213 217 3 4693.3 0.2 2.8X
[info] native 31 32 1 32498.9 0.0 19.1X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 851 865 6 1175.3 0.9 1.0X
[info] java 212 216 3 4717.0 0.2 4.0X
[info] native 30 32 1 32903.0 0.0 28.0X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1301 1316 6 768.4 1.3 1.0X
[info] java 212 216 2 4717.4 0.2 6.1X
[info] native 31 32 1 32606.0 0.0 42.4X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 454 460 2 2203.0 0.5 1.0X
[info] java 208 212 3 4803.8 0.2 2.2X
[info] native 15 16 0 66586.0 0.0 30.2X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 529 536 4 1889.7 0.5 1.0X
[info] java 208 212 3 4798.6 0.2 2.5X
[info] native 15 16 1 66751.4 0.0 35.3X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 830 840 5 1205.1 0.8 1.0X
[info] java 208 211 2 4814.1 0.2 4.0X
[info] native 15 15 1 67676.4 0.0 56.2X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 894 907 7 1118.7 0.9 1.0X
[info] java 208 211 3 4809.6 0.2 4.3X
[info] native 15 16 1 66675.2 0.0 59.6X
```
#### JDK16:
```
[info] OpenJDK 64-Bit Server VM 16+36 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.VectorBLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 193 199 3 517.5 1.9 1.0X
[info] java 181 186 4 553.2 1.8 1.1X
[info] native 181 185 5 553.6 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 112 2 925.1 1.1 1.0X
[info] java 88 91 3 1138.6 0.9 1.2X
[info] native 87 91 3 1144.2 0.9 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 184 189 3 542.5 1.8 1.0X
[info] java 181 185 3 552.8 1.8 1.0X
[info] native 179 183 2 558.0 1.8 1.0X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 97 101 3 1031.6 1.0 1.0X
[info] java 86 90 2 1163.7 0.9 1.1X
[info] native 85 88 2 1182.9 0.8 1.1X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 107 109 2 932.4 1.1 1.0X
[info] java 54 56 2 1846.7 0.5 2.0X
[info] native 54 56 2 1846.7 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 1 1043.6 1.0 1.0X
[info] java 29 30 1 3439.3 0.3 3.3X
[info] native 29 30 1 3423.9 0.3 3.3X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 121 123 2 829.8 1.2 1.0X
[info] java 32 32 1 3171.3 0.3 3.8X
[info] native 45 46 1 2246.2 0.4 2.7X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 142 144 2 705.9 1.4 1.0X
[info] java 15 16 1 6585.8 0.2 9.3X
[info] native 26 27 1 3839.5 0.3 5.4X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 157 165 5 635.6 1.6 1.0X
[info] java 151 159 5 664.0 1.5 1.0X
[info] native 151 160 5 663.6 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 89 2 1172.3 0.9 1.0X
[info] java 75 79 3 1337.3 0.7 1.1X
[info] native 75 79 2 1335.5 0.7 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 58 59 1 1731.5 0.6 1.0X
[info] java 28 29 1 3544.2 0.3 2.0X
[info] native 30 31 1 3306.2 0.3 1.9X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 90 92 1 1108.3 0.9 1.0X
[info] java 28 28 1 3622.5 0.3 3.3X
[info] native 30 31 1 3381.3 0.3 3.1X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 44 45 1 2284.7 0.4 1.0X
[info] java 14 15 1 7034.0 0.1 3.1X
[info] native 15 16 1 6643.7 0.2 2.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 86 1 1177.4 0.8 1.0X
[info] java 15 15 1 6886.1 0.1 5.8X
[info] native 15 16 1 6560.1 0.2 5.6X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 164 173 6 608.1 1.6 1.0X
[info] java 148 157 5 675.2 1.5 1.1X
[info] native 152 160 5 659.9 1.5 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 61 63 1 815.4 1.2 1.0X
[info] java 16 17 1 3104.3 0.3 3.8X
[info] native 24 27 1 2071.9 0.5 2.5X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 81 85 2 616.4 1.6 1.0X
[info] java 81 85 2 614.7 1.6 1.0X
[info] native 75 78 2 669.5 1.5 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 138 141 3 362.7 2.8 1.0X
[info] java 137 140 2 365.3 2.7 1.0X
[info] native 131 134 2 382.9 2.6 1.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 525 544 8 1906.2 0.5 1.0X
[info] java 61 68 3 16358.1 0.1 8.6X
[info] native 31 32 1 32623.7 0.0 17.1X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 580 598 12 1724.5 0.6 1.0X
[info] java 61 68 4 16302.5 0.1 9.5X
[info] native 30 32 1 32962.8 0.0 19.1X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 829 838 4 1206.2 0.8 1.0X
[info] java 61 69 3 16339.7 0.1 13.5X
[info] native 30 31 1 33231.9 0.0 27.6X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1352 1363 5 739.6 1.4 1.0X
[info] java 61 69 3 16347.0 0.1 22.1X
[info] native 31 32 1 32740.3 0.0 44.3X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 482 493 7 2073.1 0.5 1.0X
[info] java 35 38 2 28315.3 0.0 13.7X
[info] native 15 15 1 67579.7 0.0 32.6X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 472 482 4 2119.0 0.5 1.0X
[info] java 36 38 2 28138.1 0.0 13.3X
[info] native 15 16 1 66616.5 0.0 31.4X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 823 830 5 1215.2 0.8 1.0X
[info] java 35 38 2 28681.4 0.0 23.6X
[info] native 15 15 1 67908.4 0.0 55.9X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 896 908 7 1115.8 0.9 1.0X
[info] java 35 38 2 28402.0 0.0 25.5X
[info] native 15 16 0 66691.2 0.0 59.8X
```
TODO:
- [x] update documentation in `docs/` and `docs/ml-linalg-guide.md` refering `com.github.fommil.netlib`
- [ ] merge https://github.com/luhenry/netlib/pull/1 with all feedback from this PR + remove references to snapshot repositories in `pom.xml` and `project/SparkBuild.scala`.
Closes #32415 from luhenry/master.
Authored-by: Ludovic Henry <git@ludovic.dev>
Signed-off-by: Sean Owen <srowen@gmail.com>
2021-05-12 09:59:36 -04:00
|
|
|
arpack/2.2.0//arpack-2.2.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
arpack_combined_all/0.1//arpack_combined_all-0.1.jar
|
2020-11-09 22:07:16 -05:00
|
|
|
arrow-format/2.0.0//arrow-format-2.0.0.jar
|
|
|
|
|
arrow-memory-core/2.0.0//arrow-memory-core-2.0.0.jar
|
|
|
|
|
arrow-memory-netty/2.0.0//arrow-memory-netty-2.0.0.jar
|
|
|
|
|
arrow-vector/2.0.0//arrow-vector-2.0.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
audience-annotations/0.5.0//audience-annotations-0.5.0.jar
|
|
|
|
|
automaton/1.11-8//automaton-1.11-8.jar
|
2021-03-22 07:30:14 -04:00
|
|
|
avro-ipc/1.10.2//avro-ipc-1.10.2.jar
|
|
|
|
|
avro-mapred/1.10.2//avro-mapred-1.10.2.jar
|
|
|
|
|
avro/1.10.2//avro-1.10.2.jar
|
[SPARK-35295][ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0
### What changes were proposed in this pull request?
Bump to `dev.ludovic.netlib:2.0` which provides JNI-based wrappers for BLAS, ARPACK, and LAPACK. Theseare not taking dependencies on GPL or LGPL libraries, allowing to provide out-of-the-box support for hardware acceleration when a native library is present (this is still up to the end-user to install such library on their system, like OpenBLAS, Intel MKL, and libarpack2).
### Why are the changes needed?
Great performance improvement for ML-related workload on vanilla-distributions of Spark.
### Does this PR introduce _any_ user-facing change?
Users now take advantage of hardware acceleration as long as a native library is installed (like OpenBLAS, Intel MKL and libarpack2).
### How was this patch tested?
Spark test-suite + dev.ludovic.netlib testsuite.
#### JDK8:
```
[info] OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java8BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 220 226 6 454.9 2.2 1.0X
[info] java 221 228 5 451.9 2.2 1.0X
[info] native 209 215 5 478.7 2.1 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 121 125 3 823.3 1.2 1.0X
[info] java 121 125 3 824.3 1.2 1.0X
[info] native 101 105 3 988.4 1.0 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 212 219 6 470.9 2.1 1.0X
[info] java 208 212 4 481.0 2.1 1.0X
[info] native 209 215 5 478.5 2.1 1.0X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 114 119 3 878.9 1.1 1.0X
[info] java 99 105 3 1011.4 1.0 1.2X
[info] native 97 103 3 1026.7 1.0 1.2X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 111 2 925.9 1.1 1.0X
[info] java 71 73 2 1414.9 0.7 1.5X
[info] native 54 56 2 1847.0 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 2 1046.8 1.0 1.0X
[info] java 47 48 1 2129.8 0.5 2.0X
[info] native 29 30 1 3404.7 0.3 3.3X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 139 143 2 718.2 1.4 1.0X
[info] java 46 47 1 2171.2 0.5 3.0X
[info] native 44 46 2 2261.8 0.4 3.1X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 154 157 4 651.0 1.5 1.0X
[info] java 40 42 1 2469.3 0.4 3.8X
[info] native 26 27 1 3787.6 0.3 5.8X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 185 195 8 541.0 1.8 1.0X
[info] java 186 196 7 538.5 1.9 1.0X
[info] native 177 187 7 564.1 1.8 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 98 102 3 1016.2 1.0 1.0X
[info] java 98 102 3 1017.8 1.0 1.0X
[info] native 87 91 3 1143.2 0.9 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 68 70 1 1474.7 0.7 1.0X
[info] java 51 52 1 1973.0 0.5 1.3X
[info] native 30 32 1 3298.8 0.3 2.2X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 99 2 1037.9 1.0 1.0X
[info] java 50 51 1 1999.6 0.5 1.9X
[info] native 30 31 1 3368.1 0.3 3.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 59 61 1 1688.7 0.6 1.0X
[info] java 41 42 1 2461.9 0.4 1.5X
[info] native 15 16 1 6593.0 0.2 3.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 90 92 1 1116.2 0.9 1.0X
[info] java 39 40 1 2565.8 0.4 2.3X
[info] native 15 16 1 6594.2 0.2 5.9X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 192 202 7 520.5 1.9 1.0X
[info] java 203 214 7 491.9 2.0 0.9X
[info] native 176 187 7 568.8 1.8 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 59 61 1 846.1 1.2 1.0X
[info] java 38 39 1 1313.5 0.8 1.6X
[info] native 24 27 1 2047.8 0.5 2.4X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 97 101 3 515.4 1.9 1.0X
[info] java 97 101 2 515.1 1.9 1.0X
[info] native 88 91 3 569.1 1.8 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 169 174 3 295.4 3.4 1.0X
[info] java 169 174 3 295.4 3.4 1.0X
[info] native 160 165 4 312.2 3.2 1.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 561 577 13 1782.3 0.6 1.0X
[info] java 225 231 4 4446.2 0.2 2.5X
[info] native 31 32 3 32473.1 0.0 18.2X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 570 584 9 1754.8 0.6 1.0X
[info] java 224 230 4 4457.3 0.2 2.5X
[info] native 31 32 1 32493.4 0.0 18.5X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 855 866 6 1169.2 0.9 1.0X
[info] java 224 228 3 4466.9 0.2 3.8X
[info] native 31 32 1 32395.5 0.0 27.7X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1328 1344 8 752.8 1.3 1.0X
[info] java 224 230 4 4458.9 0.2 5.9X
[info] native 31 32 1 32201.8 0.0 42.8X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 534 541 5 1873.0 0.5 1.0X
[info] java 220 224 3 4542.8 0.2 2.4X
[info] native 15 16 1 66803.1 0.0 35.7X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 544 551 6 1839.6 0.5 1.0X
[info] java 220 224 4 4538.2 0.2 2.5X
[info] native 15 16 1 65589.9 0.0 35.7X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 833 845 21 1201.0 0.8 1.0X
[info] java 220 224 3 4548.7 0.2 3.8X
[info] native 15 16 1 66603.2 0.0 55.5X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 899 907 5 1112.9 0.9 1.0X
[info] java 221 224 2 4531.6 0.2 4.1X
[info] native 15 16 1 65944.9 0.0 59.3X
```
#### JDK11:
```
[info] OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java11BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 195 200 3 512.2 2.0 1.0X
[info] java 197 202 3 507.0 2.0 1.0X
[info] native 184 189 4 543.0 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 112 3 921.8 1.1 1.0X
[info] java 101 105 3 989.4 1.0 1.1X
[info] native 87 91 3 1147.1 0.9 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 187 191 3 535.1 1.9 1.0X
[info] java 182 188 3 548.8 1.8 1.0X
[info] native 178 182 3 562.2 1.8 1.1X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 110 114 3 909.3 1.1 1.0X
[info] java 86 93 4 1159.3 0.9 1.3X
[info] native 86 90 3 1162.4 0.9 1.3X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 106 108 2 943.6 1.1 1.0X
[info] java 70 71 2 1426.8 0.7 1.5X
[info] native 54 56 2 1835.4 0.5 1.9X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 1 1047.1 1.0 1.0X
[info] java 43 44 1 2331.9 0.4 2.2X
[info] native 29 30 1 3392.1 0.3 3.2X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 114 115 2 880.7 1.1 1.0X
[info] java 42 43 1 2398.1 0.4 2.7X
[info] native 45 46 1 2233.3 0.4 2.5X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 140 143 2 714.6 1.4 1.0X
[info] java 28 29 1 3531.0 0.3 4.9X
[info] native 26 27 1 3820.0 0.3 5.3X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 156 166 7 641.3 1.6 1.0X
[info] java 158 167 6 633.2 1.6 1.0X
[info] native 150 160 7 664.8 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 88 2 1181.7 0.8 1.0X
[info] java 85 88 2 1176.0 0.9 1.0X
[info] native 75 78 2 1333.2 0.8 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 58 59 1 1731.1 0.6 1.0X
[info] java 41 43 1 2415.5 0.4 1.4X
[info] native 30 31 1 3293.9 0.3 1.9X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 94 96 1 1063.4 0.9 1.0X
[info] java 41 42 1 2435.8 0.4 2.3X
[info] native 30 30 1 3379.8 0.3 3.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 44 45 1 2278.9 0.4 1.0X
[info] java 37 38 0 2686.8 0.4 1.2X
[info] native 15 16 1 6555.4 0.2 2.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 88 89 1 1142.1 0.9 1.0X
[info] java 33 34 1 3010.7 0.3 2.6X
[info] native 15 16 1 6553.9 0.2 5.7X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 164 172 4 609.4 1.6 1.0X
[info] java 163 172 5 612.6 1.6 1.0X
[info] native 150 159 4 667.0 1.5 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 49 50 1 1029.4 1.0 1.0X
[info] java 41 42 1 1209.4 0.8 1.2X
[info] native 25 27 1 2029.2 0.5 2.0X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 80 85 3 622.2 1.6 1.0X
[info] java 80 85 3 622.4 1.6 1.0X
[info] native 75 79 3 668.7 1.5 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 137 142 3 364.1 2.7 1.0X
[info] java 139 142 2 360.4 2.8 1.0X
[info] native 131 135 3 380.4 2.6 1.0X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 517 525 5 1935.5 0.5 1.0X
[info] java 213 216 3 4704.8 0.2 2.4X
[info] native 31 31 1 32705.6 0.0 16.9X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 589 601 6 1698.6 0.6 1.0X
[info] java 213 217 3 4693.3 0.2 2.8X
[info] native 31 32 1 32498.9 0.0 19.1X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 851 865 6 1175.3 0.9 1.0X
[info] java 212 216 3 4717.0 0.2 4.0X
[info] native 30 32 1 32903.0 0.0 28.0X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1301 1316 6 768.4 1.3 1.0X
[info] java 212 216 2 4717.4 0.2 6.1X
[info] native 31 32 1 32606.0 0.0 42.4X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 454 460 2 2203.0 0.5 1.0X
[info] java 208 212 3 4803.8 0.2 2.2X
[info] native 15 16 0 66586.0 0.0 30.2X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 529 536 4 1889.7 0.5 1.0X
[info] java 208 212 3 4798.6 0.2 2.5X
[info] native 15 16 1 66751.4 0.0 35.3X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 830 840 5 1205.1 0.8 1.0X
[info] java 208 211 2 4814.1 0.2 4.0X
[info] native 15 15 1 67676.4 0.0 56.2X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 894 907 7 1118.7 0.9 1.0X
[info] java 208 211 3 4809.6 0.2 4.3X
[info] native 15 16 1 66675.2 0.0 59.6X
```
#### JDK16:
```
[info] OpenJDK 64-Bit Server VM 16+36 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.VectorBLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 193 199 3 517.5 1.9 1.0X
[info] java 181 186 4 553.2 1.8 1.1X
[info] native 181 185 5 553.6 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 112 2 925.1 1.1 1.0X
[info] java 88 91 3 1138.6 0.9 1.2X
[info] native 87 91 3 1144.2 0.9 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 184 189 3 542.5 1.8 1.0X
[info] java 181 185 3 552.8 1.8 1.0X
[info] native 179 183 2 558.0 1.8 1.0X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 97 101 3 1031.6 1.0 1.0X
[info] java 86 90 2 1163.7 0.9 1.1X
[info] native 85 88 2 1182.9 0.8 1.1X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 107 109 2 932.4 1.1 1.0X
[info] java 54 56 2 1846.7 0.5 2.0X
[info] native 54 56 2 1846.7 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 1 1043.6 1.0 1.0X
[info] java 29 30 1 3439.3 0.3 3.3X
[info] native 29 30 1 3423.9 0.3 3.3X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 121 123 2 829.8 1.2 1.0X
[info] java 32 32 1 3171.3 0.3 3.8X
[info] native 45 46 1 2246.2 0.4 2.7X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 142 144 2 705.9 1.4 1.0X
[info] java 15 16 1 6585.8 0.2 9.3X
[info] native 26 27 1 3839.5 0.3 5.4X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 157 165 5 635.6 1.6 1.0X
[info] java 151 159 5 664.0 1.5 1.0X
[info] native 151 160 5 663.6 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 89 2 1172.3 0.9 1.0X
[info] java 75 79 3 1337.3 0.7 1.1X
[info] native 75 79 2 1335.5 0.7 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 58 59 1 1731.5 0.6 1.0X
[info] java 28 29 1 3544.2 0.3 2.0X
[info] native 30 31 1 3306.2 0.3 1.9X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 90 92 1 1108.3 0.9 1.0X
[info] java 28 28 1 3622.5 0.3 3.3X
[info] native 30 31 1 3381.3 0.3 3.1X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 44 45 1 2284.7 0.4 1.0X
[info] java 14 15 1 7034.0 0.1 3.1X
[info] native 15 16 1 6643.7 0.2 2.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 86 1 1177.4 0.8 1.0X
[info] java 15 15 1 6886.1 0.1 5.8X
[info] native 15 16 1 6560.1 0.2 5.6X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 164 173 6 608.1 1.6 1.0X
[info] java 148 157 5 675.2 1.5 1.1X
[info] native 152 160 5 659.9 1.5 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 61 63 1 815.4 1.2 1.0X
[info] java 16 17 1 3104.3 0.3 3.8X
[info] native 24 27 1 2071.9 0.5 2.5X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 81 85 2 616.4 1.6 1.0X
[info] java 81 85 2 614.7 1.6 1.0X
[info] native 75 78 2 669.5 1.5 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 138 141 3 362.7 2.8 1.0X
[info] java 137 140 2 365.3 2.7 1.0X
[info] native 131 134 2 382.9 2.6 1.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 525 544 8 1906.2 0.5 1.0X
[info] java 61 68 3 16358.1 0.1 8.6X
[info] native 31 32 1 32623.7 0.0 17.1X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 580 598 12 1724.5 0.6 1.0X
[info] java 61 68 4 16302.5 0.1 9.5X
[info] native 30 32 1 32962.8 0.0 19.1X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 829 838 4 1206.2 0.8 1.0X
[info] java 61 69 3 16339.7 0.1 13.5X
[info] native 30 31 1 33231.9 0.0 27.6X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1352 1363 5 739.6 1.4 1.0X
[info] java 61 69 3 16347.0 0.1 22.1X
[info] native 31 32 1 32740.3 0.0 44.3X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 482 493 7 2073.1 0.5 1.0X
[info] java 35 38 2 28315.3 0.0 13.7X
[info] native 15 15 1 67579.7 0.0 32.6X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 472 482 4 2119.0 0.5 1.0X
[info] java 36 38 2 28138.1 0.0 13.3X
[info] native 15 16 1 66616.5 0.0 31.4X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 823 830 5 1215.2 0.8 1.0X
[info] java 35 38 2 28681.4 0.0 23.6X
[info] native 15 15 1 67908.4 0.0 55.9X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 896 908 7 1115.8 0.9 1.0X
[info] java 35 38 2 28402.0 0.0 25.5X
[info] native 15 16 0 66691.2 0.0 59.8X
```
TODO:
- [x] update documentation in `docs/` and `docs/ml-linalg-guide.md` refering `com.github.fommil.netlib`
- [ ] merge https://github.com/luhenry/netlib/pull/1 with all feedback from this PR + remove references to snapshot repositories in `pom.xml` and `project/SparkBuild.scala`.
Closes #32415 from luhenry/master.
Authored-by: Ludovic Henry <git@ludovic.dev>
Signed-off-by: Sean Owen <srowen@gmail.com>
2021-05-12 09:59:36 -04:00
|
|
|
blas/2.2.0//blas-2.2.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar
|
|
|
|
|
breeze-macros_2.12/1.0//breeze-macros_2.12-1.0.jar
|
|
|
|
|
breeze_2.12/1.0//breeze_2.12-1.0.jar
|
|
|
|
|
cats-kernel_2.12/2.0.0-M4//cats-kernel_2.12-2.0.0-M4.jar
|
2020-01-19 21:39:38 -05:00
|
|
|
chill-java/0.9.5//chill-java-0.9.5.jar
|
|
|
|
|
chill_2.12/0.9.5//chill_2.12-0.9.5.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
commons-cli/1.2//commons-cli-1.2.jar
|
2020-12-13 17:36:54 -05:00
|
|
|
commons-codec/1.15//commons-codec-1.15.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
commons-collections/3.2.2//commons-collections-3.2.2.jar
|
2021-05-12 09:57:57 -04:00
|
|
|
commons-compiler/3.1.4//commons-compiler-3.1.4.jar
|
2020-11-09 21:08:55 -05:00
|
|
|
commons-compress/1.20//commons-compress-1.20.jar
|
2020-11-09 17:33:27 -05:00
|
|
|
commons-crypto/1.1.0//commons-crypto-1.1.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
commons-dbcp/1.4//commons-dbcp-1.4.jar
|
|
|
|
|
commons-httpclient/3.1//commons-httpclient-3.1.jar
|
2021-02-08 00:53:42 -05:00
|
|
|
commons-io/2.8.0//commons-io-2.8.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
commons-lang/2.6//commons-lang-2.6.jar
|
2021-04-29 12:27:28 -04:00
|
|
|
commons-lang3/3.12.0//commons-lang3-3.12.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
commons-logging/1.1.3//commons-logging-1.1.3.jar
|
|
|
|
|
commons-math3/3.4.1//commons-math3-3.4.1.jar
|
|
|
|
|
commons-net/3.1//commons-net-3.1.jar
|
|
|
|
|
commons-pool/1.5.4//commons-pool-1.5.4.jar
|
|
|
|
|
commons-text/1.6//commons-text-1.6.jar
|
|
|
|
|
compress-lzf/1.0.3//compress-lzf-1.0.3.jar
|
|
|
|
|
core/1.1.2//core-1.1.2.jar
|
|
|
|
|
curator-client/2.13.0//curator-client-2.13.0.jar
|
|
|
|
|
curator-framework/2.13.0//curator-framework-2.13.0.jar
|
|
|
|
|
curator-recipes/2.13.0//curator-recipes-2.13.0.jar
|
|
|
|
|
datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar
|
|
|
|
|
datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
|
|
|
|
|
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
|
2021-01-06 00:50:16 -05:00
|
|
|
derby/10.14.2.0//derby-10.14.2.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
|
|
|
|
|
flatbuffers-java/1.9.0//flatbuffers-java-1.9.0.jar
|
|
|
|
|
generex/1.0.2//generex-1.0.2.jar
|
|
|
|
|
gson/2.2.4//gson-2.2.4.jar
|
|
|
|
|
guava/14.0.1//guava-14.0.1.jar
|
[SPARK-33212][BUILD] Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile
### What changes were proposed in this pull request?
This:
1. switches Spark to use shaded Hadoop clients, namely hadoop-client-api and hadoop-client-runtime, for Hadoop 3.x.
2. upgrade built-in version for Hadoop 3.x to Hadoop 3.2.2
Note that for Hadoop 2.7, we'll still use the same modules such as hadoop-client.
In order to still keep default Hadoop profile to be hadoop-3.2, this defines the following Maven properties:
```
hadoop-client-api.artifact
hadoop-client-runtime.artifact
hadoop-client-minicluster.artifact
```
which default to:
```
hadoop-client-api
hadoop-client-runtime
hadoop-client-minicluster
```
but all switch to `hadoop-client` when the Hadoop profile is hadoop-2.7. A side affect from this is we'll import the same dependency multiple times. For this I have to disable Maven enforcer `banDuplicatePomDependencyVersions`.
Besides above, there are the following changes:
- explicitly add a few dependencies which are imported via transitive dependencies from Hadoop jars, but are removed from the shaded client jars.
- removed the use of `ProxyUriUtils.getPath` from `ApplicationMaster` which is a server-side/private API.
- modified `IsolatedClientLoader` to exclude `hadoop-auth` jars when Hadoop version is 3.x. This change should only matter when we're not sharing Hadoop classes with Spark (which is _mostly_ used in tests).
### Why are the changes needed?
Hadoop 3.2.2 is released with new features and bug fixes, so it's good for the Spark community to adopt it. However, latest Hadoop versions starting from Hadoop 3.2.1 have upgraded to use Guava 27+. In order to resolve Guava conflicts, this takes the approach by switching to shaded client jars provided by Hadoop. This also has the benefits of avoid pulling other 3rd party dependencies from Hadoop side so as to avoid more potential future conflicts.
### Does this PR introduce _any_ user-facing change?
When people use Spark with `hadoop-provided` option, they should make sure class path contains `hadoop-client-api` and `hadoop-client-runtime` jars. In addition, they may need to make sure these jars appear before other Hadoop jars in the order. Otherwise, classes may be loaded from the other non-shaded Hadoop jars and cause potential conflicts.
### How was this patch tested?
Relying on existing tests.
Closes #30701 from sunchao/test-hadoop-3.2.2.
Authored-by: Chao Sun <sunchao@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2021-01-15 17:06:50 -05:00
|
|
|
hadoop-client-api/3.2.2//hadoop-client-api-3.2.2.jar
|
|
|
|
|
hadoop-client-runtime/3.2.2//hadoop-client-runtime-3.2.2.jar
|
2021-02-28 19:37:49 -05:00
|
|
|
hadoop-yarn-server-web-proxy/3.2.2//hadoop-yarn-server-web-proxy-3.2.2.jar
|
2021-01-18 00:54:35 -05:00
|
|
|
hive-beeline/2.3.8//hive-beeline-2.3.8.jar
|
|
|
|
|
hive-cli/2.3.8//hive-cli-2.3.8.jar
|
|
|
|
|
hive-common/2.3.8//hive-common-2.3.8.jar
|
|
|
|
|
hive-exec/2.3.8/core/hive-exec-2.3.8-core.jar
|
|
|
|
|
hive-jdbc/2.3.8//hive-jdbc-2.3.8.jar
|
|
|
|
|
hive-llap-common/2.3.8//hive-llap-common-2.3.8.jar
|
|
|
|
|
hive-metastore/2.3.8//hive-metastore-2.3.8.jar
|
|
|
|
|
hive-serde/2.3.8//hive-serde-2.3.8.jar
|
2020-11-25 15:37:59 -05:00
|
|
|
hive-service-rpc/3.1.2//hive-service-rpc-3.1.2.jar
|
2021-01-18 00:54:35 -05:00
|
|
|
hive-shims-0.23/2.3.8//hive-shims-0.23-2.3.8.jar
|
|
|
|
|
hive-shims-common/2.3.8//hive-shims-common-2.3.8.jar
|
|
|
|
|
hive-shims-scheduler/2.3.8//hive-shims-scheduler-2.3.8.jar
|
|
|
|
|
hive-shims/2.3.8//hive-shims-2.3.8.jar
|
2020-10-01 15:41:40 -04:00
|
|
|
hive-storage-api/2.7.2//hive-storage-api-2.7.2.jar
|
2021-01-18 00:54:35 -05:00
|
|
|
hive-vector-code-gen/2.3.8//hive-vector-code-gen-2.3.8.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
hk2-api/2.6.1//hk2-api-2.6.1.jar
|
|
|
|
|
hk2-locator/2.6.1//hk2-locator-2.6.1.jar
|
|
|
|
|
hk2-utils/2.6.1//hk2-utils-2.6.1.jar
|
|
|
|
|
htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
|
2020-12-07 02:02:36 -05:00
|
|
|
httpclient/4.5.13//httpclient-4.5.13.jar
|
2021-05-23 11:16:50 -04:00
|
|
|
httpcore/4.4.14//httpcore-4.4.14.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
|
|
|
|
|
ivy/2.4.0//ivy-2.4.0.jar
|
2021-05-30 21:28:43 -04:00
|
|
|
jackson-annotations/2.12.3//jackson-annotations-2.12.3.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
|
2021-05-30 21:28:43 -04:00
|
|
|
jackson-core/2.12.3//jackson-core-2.12.3.jar
|
|
|
|
|
jackson-databind/2.12.3//jackson-databind-2.12.3.jar
|
|
|
|
|
jackson-dataformat-yaml/2.12.3//jackson-dataformat-yaml-2.12.3.jar
|
2020-11-03 01:23:26 -05:00
|
|
|
jackson-datatype-jsr310/2.11.2//jackson-datatype-jsr310-2.11.2.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
|
2021-05-30 21:28:43 -04:00
|
|
|
jackson-module-scala_2.12/2.12.3//jackson-module-scala_2.12-2.12.3.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
|
|
|
|
|
jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
|
2020-12-14 00:14:38 -05:00
|
|
|
jakarta.servlet-api/4.0.3//jakarta.servlet-api-4.0.3.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar
|
|
|
|
|
jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar
|
|
|
|
|
jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
|
2021-05-12 09:57:57 -04:00
|
|
|
janino/3.1.4//janino-3.1.4.jar
|
2020-01-25 18:41:55 -05:00
|
|
|
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
|
|
|
|
|
javolution/5.5.1//javolution-5.5.1.jar
|
|
|
|
|
jaxb-api/2.2.11//jaxb-api-2.2.11.jar
|
|
|
|
|
jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
|
2020-05-04 11:14:12 -04:00
|
|
|
jcl-over-slf4j/1.7.30//jcl-over-slf4j-1.7.30.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
jdo-api/3.0.1//jdo-api-3.0.1.jar
|
2021-05-06 12:08:10 -04:00
|
|
|
jersey-client/2.34//jersey-client-2.34.jar
|
|
|
|
|
jersey-common/2.34//jersey-common-2.34.jar
|
|
|
|
|
jersey-container-servlet-core/2.34//jersey-container-servlet-core-2.34.jar
|
|
|
|
|
jersey-container-servlet/2.34//jersey-container-servlet-2.34.jar
|
|
|
|
|
jersey-hk2/2.34//jersey-hk2-2.34.jar
|
|
|
|
|
jersey-server/2.34//jersey-server-2.34.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
jline/2.14.6//jline-2.14.6.jar
|
2021-05-25 14:29:03 -04:00
|
|
|
joda-time/2.10.10//joda-time-2.10.10.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
jodd-core/3.5.2//jodd-core-3.5.2.jar
|
|
|
|
|
jpam/1.1//jpam-1.1.jar
|
|
|
|
|
json/1.8//json-1.8.jar
|
2021-05-26 04:10:14 -04:00
|
|
|
json4s-ast_2.12/3.7.0-M11//json4s-ast_2.12-3.7.0-M11.jar
|
|
|
|
|
json4s-core_2.12/3.7.0-M11//json4s-core_2.12-3.7.0-M11.jar
|
|
|
|
|
json4s-jackson_2.12/3.7.0-M11//json4s-jackson_2.12-3.7.0-M11.jar
|
|
|
|
|
json4s-scalap_2.12/3.7.0-M11//json4s-scalap_2.12-3.7.0-M11.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
jsr305/3.0.0//jsr305-3.0.0.jar
|
|
|
|
|
jta/1.1//jta-1.1.jar
|
2020-05-04 11:14:12 -04:00
|
|
|
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
|
2021-06-07 01:27:08 -04:00
|
|
|
kubernetes-client/5.4.1//kubernetes-client-5.4.1.jar
|
|
|
|
|
kubernetes-model-admissionregistration/5.4.1//kubernetes-model-admissionregistration-5.4.1.jar
|
|
|
|
|
kubernetes-model-apiextensions/5.4.1//kubernetes-model-apiextensions-5.4.1.jar
|
|
|
|
|
kubernetes-model-apps/5.4.1//kubernetes-model-apps-5.4.1.jar
|
|
|
|
|
kubernetes-model-autoscaling/5.4.1//kubernetes-model-autoscaling-5.4.1.jar
|
|
|
|
|
kubernetes-model-batch/5.4.1//kubernetes-model-batch-5.4.1.jar
|
|
|
|
|
kubernetes-model-certificates/5.4.1//kubernetes-model-certificates-5.4.1.jar
|
|
|
|
|
kubernetes-model-common/5.4.1//kubernetes-model-common-5.4.1.jar
|
|
|
|
|
kubernetes-model-coordination/5.4.1//kubernetes-model-coordination-5.4.1.jar
|
|
|
|
|
kubernetes-model-core/5.4.1//kubernetes-model-core-5.4.1.jar
|
|
|
|
|
kubernetes-model-discovery/5.4.1//kubernetes-model-discovery-5.4.1.jar
|
|
|
|
|
kubernetes-model-events/5.4.1//kubernetes-model-events-5.4.1.jar
|
|
|
|
|
kubernetes-model-extensions/5.4.1//kubernetes-model-extensions-5.4.1.jar
|
|
|
|
|
kubernetes-model-flowcontrol/5.4.1//kubernetes-model-flowcontrol-5.4.1.jar
|
|
|
|
|
kubernetes-model-metrics/5.4.1//kubernetes-model-metrics-5.4.1.jar
|
|
|
|
|
kubernetes-model-networking/5.4.1//kubernetes-model-networking-5.4.1.jar
|
|
|
|
|
kubernetes-model-node/5.4.1//kubernetes-model-node-5.4.1.jar
|
|
|
|
|
kubernetes-model-policy/5.4.1//kubernetes-model-policy-5.4.1.jar
|
|
|
|
|
kubernetes-model-rbac/5.4.1//kubernetes-model-rbac-5.4.1.jar
|
|
|
|
|
kubernetes-model-scheduling/5.4.1//kubernetes-model-scheduling-5.4.1.jar
|
|
|
|
|
kubernetes-model-storageclass/5.4.1//kubernetes-model-storageclass-5.4.1.jar
|
[SPARK-35295][ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0
### What changes were proposed in this pull request?
Bump to `dev.ludovic.netlib:2.0` which provides JNI-based wrappers for BLAS, ARPACK, and LAPACK. Theseare not taking dependencies on GPL or LGPL libraries, allowing to provide out-of-the-box support for hardware acceleration when a native library is present (this is still up to the end-user to install such library on their system, like OpenBLAS, Intel MKL, and libarpack2).
### Why are the changes needed?
Great performance improvement for ML-related workload on vanilla-distributions of Spark.
### Does this PR introduce _any_ user-facing change?
Users now take advantage of hardware acceleration as long as a native library is installed (like OpenBLAS, Intel MKL and libarpack2).
### How was this patch tested?
Spark test-suite + dev.ludovic.netlib testsuite.
#### JDK8:
```
[info] OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java8BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 220 226 6 454.9 2.2 1.0X
[info] java 221 228 5 451.9 2.2 1.0X
[info] native 209 215 5 478.7 2.1 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 121 125 3 823.3 1.2 1.0X
[info] java 121 125 3 824.3 1.2 1.0X
[info] native 101 105 3 988.4 1.0 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 212 219 6 470.9 2.1 1.0X
[info] java 208 212 4 481.0 2.1 1.0X
[info] native 209 215 5 478.5 2.1 1.0X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 114 119 3 878.9 1.1 1.0X
[info] java 99 105 3 1011.4 1.0 1.2X
[info] native 97 103 3 1026.7 1.0 1.2X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 111 2 925.9 1.1 1.0X
[info] java 71 73 2 1414.9 0.7 1.5X
[info] native 54 56 2 1847.0 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 2 1046.8 1.0 1.0X
[info] java 47 48 1 2129.8 0.5 2.0X
[info] native 29 30 1 3404.7 0.3 3.3X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 139 143 2 718.2 1.4 1.0X
[info] java 46 47 1 2171.2 0.5 3.0X
[info] native 44 46 2 2261.8 0.4 3.1X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 154 157 4 651.0 1.5 1.0X
[info] java 40 42 1 2469.3 0.4 3.8X
[info] native 26 27 1 3787.6 0.3 5.8X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 185 195 8 541.0 1.8 1.0X
[info] java 186 196 7 538.5 1.9 1.0X
[info] native 177 187 7 564.1 1.8 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 98 102 3 1016.2 1.0 1.0X
[info] java 98 102 3 1017.8 1.0 1.0X
[info] native 87 91 3 1143.2 0.9 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 68 70 1 1474.7 0.7 1.0X
[info] java 51 52 1 1973.0 0.5 1.3X
[info] native 30 32 1 3298.8 0.3 2.2X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 99 2 1037.9 1.0 1.0X
[info] java 50 51 1 1999.6 0.5 1.9X
[info] native 30 31 1 3368.1 0.3 3.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 59 61 1 1688.7 0.6 1.0X
[info] java 41 42 1 2461.9 0.4 1.5X
[info] native 15 16 1 6593.0 0.2 3.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 90 92 1 1116.2 0.9 1.0X
[info] java 39 40 1 2565.8 0.4 2.3X
[info] native 15 16 1 6594.2 0.2 5.9X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 192 202 7 520.5 1.9 1.0X
[info] java 203 214 7 491.9 2.0 0.9X
[info] native 176 187 7 568.8 1.8 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 59 61 1 846.1 1.2 1.0X
[info] java 38 39 1 1313.5 0.8 1.6X
[info] native 24 27 1 2047.8 0.5 2.4X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 97 101 3 515.4 1.9 1.0X
[info] java 97 101 2 515.1 1.9 1.0X
[info] native 88 91 3 569.1 1.8 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 169 174 3 295.4 3.4 1.0X
[info] java 169 174 3 295.4 3.4 1.0X
[info] native 160 165 4 312.2 3.2 1.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 561 577 13 1782.3 0.6 1.0X
[info] java 225 231 4 4446.2 0.2 2.5X
[info] native 31 32 3 32473.1 0.0 18.2X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 570 584 9 1754.8 0.6 1.0X
[info] java 224 230 4 4457.3 0.2 2.5X
[info] native 31 32 1 32493.4 0.0 18.5X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 855 866 6 1169.2 0.9 1.0X
[info] java 224 228 3 4466.9 0.2 3.8X
[info] native 31 32 1 32395.5 0.0 27.7X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1328 1344 8 752.8 1.3 1.0X
[info] java 224 230 4 4458.9 0.2 5.9X
[info] native 31 32 1 32201.8 0.0 42.8X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 534 541 5 1873.0 0.5 1.0X
[info] java 220 224 3 4542.8 0.2 2.4X
[info] native 15 16 1 66803.1 0.0 35.7X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 544 551 6 1839.6 0.5 1.0X
[info] java 220 224 4 4538.2 0.2 2.5X
[info] native 15 16 1 65589.9 0.0 35.7X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 833 845 21 1201.0 0.8 1.0X
[info] java 220 224 3 4548.7 0.2 3.8X
[info] native 15 16 1 66603.2 0.0 55.5X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 899 907 5 1112.9 0.9 1.0X
[info] java 221 224 2 4531.6 0.2 4.1X
[info] native 15 16 1 65944.9 0.0 59.3X
```
#### JDK11:
```
[info] OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.Java11BLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 195 200 3 512.2 2.0 1.0X
[info] java 197 202 3 507.0 2.0 1.0X
[info] native 184 189 4 543.0 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 112 3 921.8 1.1 1.0X
[info] java 101 105 3 989.4 1.0 1.1X
[info] native 87 91 3 1147.1 0.9 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 187 191 3 535.1 1.9 1.0X
[info] java 182 188 3 548.8 1.8 1.0X
[info] native 178 182 3 562.2 1.8 1.1X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 110 114 3 909.3 1.1 1.0X
[info] java 86 93 4 1159.3 0.9 1.3X
[info] native 86 90 3 1162.4 0.9 1.3X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 106 108 2 943.6 1.1 1.0X
[info] java 70 71 2 1426.8 0.7 1.5X
[info] native 54 56 2 1835.4 0.5 1.9X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 1 1047.1 1.0 1.0X
[info] java 43 44 1 2331.9 0.4 2.2X
[info] native 29 30 1 3392.1 0.3 3.2X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 114 115 2 880.7 1.1 1.0X
[info] java 42 43 1 2398.1 0.4 2.7X
[info] native 45 46 1 2233.3 0.4 2.5X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 140 143 2 714.6 1.4 1.0X
[info] java 28 29 1 3531.0 0.3 4.9X
[info] native 26 27 1 3820.0 0.3 5.3X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 156 166 7 641.3 1.6 1.0X
[info] java 158 167 6 633.2 1.6 1.0X
[info] native 150 160 7 664.8 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 88 2 1181.7 0.8 1.0X
[info] java 85 88 2 1176.0 0.9 1.0X
[info] native 75 78 2 1333.2 0.8 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 58 59 1 1731.1 0.6 1.0X
[info] java 41 43 1 2415.5 0.4 1.4X
[info] native 30 31 1 3293.9 0.3 1.9X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 94 96 1 1063.4 0.9 1.0X
[info] java 41 42 1 2435.8 0.4 2.3X
[info] native 30 30 1 3379.8 0.3 3.2X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 44 45 1 2278.9 0.4 1.0X
[info] java 37 38 0 2686.8 0.4 1.2X
[info] native 15 16 1 6555.4 0.2 2.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 88 89 1 1142.1 0.9 1.0X
[info] java 33 34 1 3010.7 0.3 2.6X
[info] native 15 16 1 6553.9 0.2 5.7X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 164 172 4 609.4 1.6 1.0X
[info] java 163 172 5 612.6 1.6 1.0X
[info] native 150 159 4 667.0 1.5 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 49 50 1 1029.4 1.0 1.0X
[info] java 41 42 1 1209.4 0.8 1.2X
[info] native 25 27 1 2029.2 0.5 2.0X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 80 85 3 622.2 1.6 1.0X
[info] java 80 85 3 622.4 1.6 1.0X
[info] native 75 79 3 668.7 1.5 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 137 142 3 364.1 2.7 1.0X
[info] java 139 142 2 360.4 2.8 1.0X
[info] native 131 135 3 380.4 2.6 1.0X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 517 525 5 1935.5 0.5 1.0X
[info] java 213 216 3 4704.8 0.2 2.4X
[info] native 31 31 1 32705.6 0.0 16.9X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 589 601 6 1698.6 0.6 1.0X
[info] java 213 217 3 4693.3 0.2 2.8X
[info] native 31 32 1 32498.9 0.0 19.1X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 851 865 6 1175.3 0.9 1.0X
[info] java 212 216 3 4717.0 0.2 4.0X
[info] native 30 32 1 32903.0 0.0 28.0X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1301 1316 6 768.4 1.3 1.0X
[info] java 212 216 2 4717.4 0.2 6.1X
[info] native 31 32 1 32606.0 0.0 42.4X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 454 460 2 2203.0 0.5 1.0X
[info] java 208 212 3 4803.8 0.2 2.2X
[info] native 15 16 0 66586.0 0.0 30.2X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 529 536 4 1889.7 0.5 1.0X
[info] java 208 212 3 4798.6 0.2 2.5X
[info] native 15 16 1 66751.4 0.0 35.3X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 830 840 5 1205.1 0.8 1.0X
[info] java 208 211 2 4814.1 0.2 4.0X
[info] native 15 15 1 67676.4 0.0 56.2X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 894 907 7 1118.7 0.9 1.0X
[info] java 208 211 3 4809.6 0.2 4.3X
[info] native 15 16 1 66675.2 0.0 59.6X
```
#### JDK16:
```
[info] OpenJDK 64-Bit Server VM 16+36 on Linux 5.8.0-50-generic
[info] Intel(R) Xeon(R) E-2276G CPU 3.80GHz
[info]
[info] f2jBLAS = dev.ludovic.netlib.blas.F2jBLAS
[info] javaBLAS = dev.ludovic.netlib.blas.VectorBLAS
[info] nativeBLAS = dev.ludovic.netlib.blas.JNIBLAS
[info]
[info] daxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 193 199 3 517.5 1.9 1.0X
[info] java 181 186 4 553.2 1.8 1.1X
[info] native 181 185 5 553.6 1.8 1.1X
[info]
[info] saxpy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 108 112 2 925.1 1.1 1.0X
[info] java 88 91 3 1138.6 0.9 1.2X
[info] native 87 91 3 1144.2 0.9 1.2X
[info]
[info] dcopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 184 189 3 542.5 1.8 1.0X
[info] java 181 185 3 552.8 1.8 1.0X
[info] native 179 183 2 558.0 1.8 1.0X
[info]
[info] scopy: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 97 101 3 1031.6 1.0 1.0X
[info] java 86 90 2 1163.7 0.9 1.1X
[info] native 85 88 2 1182.9 0.8 1.1X
[info]
[info] ddot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 107 109 2 932.4 1.1 1.0X
[info] java 54 56 2 1846.7 0.5 2.0X
[info] native 54 56 2 1846.7 0.5 2.0X
[info]
[info] sdot: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 96 97 1 1043.6 1.0 1.0X
[info] java 29 30 1 3439.3 0.3 3.3X
[info] native 29 30 1 3423.9 0.3 3.3X
[info]
[info] dnrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 121 123 2 829.8 1.2 1.0X
[info] java 32 32 1 3171.3 0.3 3.8X
[info] native 45 46 1 2246.2 0.4 2.7X
[info]
[info] snrm2: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 142 144 2 705.9 1.4 1.0X
[info] java 15 16 1 6585.8 0.2 9.3X
[info] native 26 27 1 3839.5 0.3 5.4X
[info]
[info] dscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 157 165 5 635.6 1.6 1.0X
[info] java 151 159 5 664.0 1.5 1.0X
[info] native 151 160 5 663.6 1.5 1.0X
[info]
[info] sscal: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 89 2 1172.3 0.9 1.0X
[info] java 75 79 3 1337.3 0.7 1.1X
[info] native 75 79 2 1335.5 0.7 1.1X
[info]
[info] dgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 58 59 1 1731.5 0.6 1.0X
[info] java 28 29 1 3544.2 0.3 2.0X
[info] native 30 31 1 3306.2 0.3 1.9X
[info]
[info] dgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 90 92 1 1108.3 0.9 1.0X
[info] java 28 28 1 3622.5 0.3 3.3X
[info] native 30 31 1 3381.3 0.3 3.1X
[info]
[info] sgemv[N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 44 45 1 2284.7 0.4 1.0X
[info] java 14 15 1 7034.0 0.1 3.1X
[info] native 15 16 1 6643.7 0.2 2.9X
[info]
[info] sgemv[T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 85 86 1 1177.4 0.8 1.0X
[info] java 15 15 1 6886.1 0.1 5.8X
[info] native 15 16 1 6560.1 0.2 5.6X
[info]
[info] dger: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 164 173 6 608.1 1.6 1.0X
[info] java 148 157 5 675.2 1.5 1.1X
[info] native 152 160 5 659.9 1.5 1.1X
[info]
[info] dspmv[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 61 63 1 815.4 1.2 1.0X
[info] java 16 17 1 3104.3 0.3 3.8X
[info] native 24 27 1 2071.9 0.5 2.5X
[info]
[info] dspr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 81 85 2 616.4 1.6 1.0X
[info] java 81 85 2 614.7 1.6 1.0X
[info] native 75 78 2 669.5 1.5 1.1X
[info]
[info] dsyr[U]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 138 141 3 362.7 2.8 1.0X
[info] java 137 140 2 365.3 2.7 1.0X
[info] native 131 134 2 382.9 2.6 1.1X
[info]
[info] dgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 525 544 8 1906.2 0.5 1.0X
[info] java 61 68 3 16358.1 0.1 8.6X
[info] native 31 32 1 32623.7 0.0 17.1X
[info]
[info] dgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 580 598 12 1724.5 0.6 1.0X
[info] java 61 68 4 16302.5 0.1 9.5X
[info] native 30 32 1 32962.8 0.0 19.1X
[info]
[info] dgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 829 838 4 1206.2 0.8 1.0X
[info] java 61 69 3 16339.7 0.1 13.5X
[info] native 30 31 1 33231.9 0.0 27.6X
[info]
[info] dgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 1352 1363 5 739.6 1.4 1.0X
[info] java 61 69 3 16347.0 0.1 22.1X
[info] native 31 32 1 32740.3 0.0 44.3X
[info]
[info] sgemm[N,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 482 493 7 2073.1 0.5 1.0X
[info] java 35 38 2 28315.3 0.0 13.7X
[info] native 15 15 1 67579.7 0.0 32.6X
[info]
[info] sgemm[N,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 472 482 4 2119.0 0.5 1.0X
[info] java 36 38 2 28138.1 0.0 13.3X
[info] native 15 16 1 66616.5 0.0 31.4X
[info]
[info] sgemm[T,N]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 823 830 5 1215.2 0.8 1.0X
[info] java 35 38 2 28681.4 0.0 23.6X
[info] native 15 15 1 67908.4 0.0 55.9X
[info]
[info] sgemm[T,T]: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] -----------------------------------------------------------------------------------------------
[info] f2j 896 908 7 1115.8 0.9 1.0X
[info] java 35 38 2 28402.0 0.0 25.5X
[info] native 15 16 0 66691.2 0.0 59.8X
```
TODO:
- [x] update documentation in `docs/` and `docs/ml-linalg-guide.md` refering `com.github.fommil.netlib`
- [ ] merge https://github.com/luhenry/netlib/pull/1 with all feedback from this PR + remove references to snapshot repositories in `pom.xml` and `project/SparkBuild.scala`.
Closes #32415 from luhenry/master.
Authored-by: Ludovic Henry <git@ludovic.dev>
Signed-off-by: Sean Owen <srowen@gmail.com>
2021-05-12 09:59:36 -04:00
|
|
|
lapack/2.2.0//lapack-2.2.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
|
|
|
|
|
libfb303/0.9.3//libfb303-0.9.3.jar
|
|
|
|
|
libthrift/0.12.0//libthrift-0.12.0.jar
|
|
|
|
|
log4j/1.2.17//log4j-1.2.17.jar
|
2020-09-30 22:00:18 -04:00
|
|
|
logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
|
2020-01-19 22:05:30 -05:00
|
|
|
lz4-java/1.7.1//lz4-java-1.7.1.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
machinist_2.12/0.6.8//machinist_2.12-0.6.8.jar
|
|
|
|
|
macro-compat_2.12/1.1.1//macro-compat_2.12-1.1.1.jar
|
|
|
|
|
mesos/1.4.0/shaded-protobuf/mesos-1.4.0-shaded-protobuf.jar
|
2021-05-22 01:53:32 -04:00
|
|
|
metrics-core/4.2.0//metrics-core-4.2.0.jar
|
|
|
|
|
metrics-graphite/4.2.0//metrics-graphite-4.2.0.jar
|
|
|
|
|
metrics-jmx/4.2.0//metrics-jmx-4.2.0.jar
|
|
|
|
|
metrics-json/4.2.0//metrics-json-4.2.0.jar
|
|
|
|
|
metrics-jvm/4.2.0//metrics-jvm-4.2.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
minlog/1.3.0//minlog-1.3.0.jar
|
2021-04-20 19:28:43 -04:00
|
|
|
netty-all/4.1.63.Final//netty-all-4.1.63.Final.jar
|
2020-07-23 19:20:17 -04:00
|
|
|
objenesis/2.6//objenesis-2.6.jar
|
2020-09-30 22:00:18 -04:00
|
|
|
okhttp/3.12.12//okhttp-3.12.12.jar
|
2020-07-23 19:20:17 -04:00
|
|
|
okio/1.14.0//okio-1.14.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
opencsv/2.3//opencsv-2.3.jar
|
2021-05-22 13:35:40 -04:00
|
|
|
orc-core/1.6.8//orc-core-1.6.8.jar
|
|
|
|
|
orc-mapreduce/1.6.8//orc-mapreduce-1.6.8.jar
|
|
|
|
|
orc-shims/1.6.8//orc-shims-1.6.8.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
oro/2.0.8//oro-2.0.8.jar
|
|
|
|
|
osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
|
|
|
|
|
paranamer/2.8//paranamer-2.8.jar
|
2021-03-27 10:56:29 -04:00
|
|
|
parquet-column/1.12.0//parquet-column-1.12.0.jar
|
|
|
|
|
parquet-common/1.12.0//parquet-common-1.12.0.jar
|
|
|
|
|
parquet-encoding/1.12.0//parquet-encoding-1.12.0.jar
|
|
|
|
|
parquet-format-structures/1.12.0//parquet-format-structures-1.12.0.jar
|
|
|
|
|
parquet-hadoop/1.12.0//parquet-hadoop-1.12.0.jar
|
|
|
|
|
parquet-jackson/1.12.0//parquet-jackson-1.12.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
|
2021-03-11 10:51:41 -05:00
|
|
|
py4j/0.10.9.2//py4j-0.10.9.2.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
pyrolite/4.30//pyrolite-4.30.jar
|
|
|
|
|
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
|
2021-05-30 19:08:13 -04:00
|
|
|
scala-compiler/2.12.14//scala-compiler-2.12.14.jar
|
|
|
|
|
scala-library/2.12.14//scala-library-2.12.14.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
scala-parser-combinators_2.12/1.1.2//scala-parser-combinators_2.12-1.1.2.jar
|
2021-05-30 19:08:13 -04:00
|
|
|
scala-reflect/2.12.14//scala-reflect-2.12.14.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
scala-xml_2.12/1.2.0//scala-xml_2.12-1.2.0.jar
|
|
|
|
|
shapeless_2.12/2.3.3//shapeless_2.12-2.3.3.jar
|
2020-07-25 13:58:25 -04:00
|
|
|
shims/0.9.0//shims-0.9.0.jar
|
2020-05-04 11:14:12 -04:00
|
|
|
slf4j-api/1.7.30//slf4j-api-1.7.30.jar
|
|
|
|
|
slf4j-log4j12/1.7.30//slf4j-log4j12-1.7.30.jar
|
2021-03-21 03:36:38 -04:00
|
|
|
snakeyaml/1.27//snakeyaml-1.27.jar
|
2021-04-30 00:26:16 -04:00
|
|
|
snappy-java/1.1.8.4//snappy-java-1.1.8.4.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
spire-macros_2.12/0.17.0-M1//spire-macros_2.12-0.17.0-M1.jar
|
|
|
|
|
spire-platform_2.12/0.17.0-M1//spire-platform_2.12-0.17.0-M1.jar
|
|
|
|
|
spire-util_2.12/0.17.0-M1//spire-util_2.12-0.17.0-M1.jar
|
|
|
|
|
spire_2.12/0.17.0-M1//spire_2.12-0.17.0-M1.jar
|
|
|
|
|
stax-api/1.0.1//stax-api-1.0.1.jar
|
|
|
|
|
stream/2.9.6//stream-2.9.6.jar
|
|
|
|
|
super-csv/2.2.0//super-csv-2.2.0.jar
|
2020-01-31 20:41:27 -05:00
|
|
|
threeten-extra/1.5.0//threeten-extra-1.5.0.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
transaction-api/1.1//transaction-api-1.1.jar
|
2021-01-19 21:40:37 -05:00
|
|
|
univocity-parsers/2.9.1//univocity-parsers-2.9.1.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
velocity/1.5//velocity-1.5.jar
|
2021-05-22 13:33:15 -04:00
|
|
|
xbean-asm7-shaded/4.16//xbean-asm7-shaded-4.16.jar
|
2021-01-20 18:42:27 -05:00
|
|
|
xz/1.8//xz-1.8.jar
|
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or downstream tools.
However, current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally. Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.careers@gmail.com>
Co-authored-by: mengCareers <meng.careers@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-01-15 23:19:44 -05:00
|
|
|
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
|
2021-01-16 00:12:41 -05:00
|
|
|
zookeeper-jute/3.6.2//zookeeper-jute-3.6.2.jar
|
|
|
|
|
zookeeper/3.6.2//zookeeper-3.6.2.jar
|
2021-03-09 01:40:49 -05:00
|
|
|
zstd-jni/1.4.9-1//zstd-jni-1.4.9-1.jar
|
