ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-3226][MLLIB] doc update for native libraries

Browse source 
to mention `-Pnetlib-lgpl` option. atalwalkar

Author: Xiangrui Meng <meng@databricks.com>

Closes #2128 from mengxr/mllib-native and squashes the following commits:

4cbba57 [Xiangrui Meng] update mllib dependencies
This commit is contained in:
Xiangrui Meng 2014-08-26 15:12:27 -07:00
parent 6b5584ef1c
commit adbd5c1636
1 changed files with 15 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
docs/mllib-guide.md
View file

@ -35,18 +35,23 @@ and the migration guide below will explain all changes between releases.
# Dependencies
MLlib uses the linear algebra package [Breeze](http://www.scalanlp.org/), which depends on
[netlib-java](https://github.com/fommil/netlib-java), and
[jblas](https://github.com/mikiobraun/jblas).
MLlib uses the linear algebra package [Breeze](http://www.scalanlp.org/),
which depends on [netlib-java](https://github.com/fommil/netlib-java),
and [jblas](https://github.com/mikiobraun/jblas).
`netlib-java` and `jblas` depend on native Fortran routines.
You need to install the
[gfortran runtime library](https://github.com/mikiobraun/jblas/wiki/Missing-Libraries) if it is not
already present on your nodes. MLlib will throw a linking error if it cannot detect these libraries
automatically. Due to license issues, we do not include `netlib-java`'s native libraries in MLlib's
dependency set. If no native library is available at runtime, you will see a warning message. To
use native libraries from `netlib-java`, please include artifact
`com.github.fommil.netlib:all:1.1.2` as a dependency of your project or build your own (see
[instructions](https://github.com/fommil/netlib-java/blob/master/README.md#machine-optimised-system-libraries)).
[gfortran runtime library](https://github.com/mikiobraun/jblas/wiki/Missing-Libraries)
if it is not already present on your nodes.
MLlib will throw a linking error if it cannot detect these libraries automatically.
Due to license issues, we do not include `netlib-java`'s native libraries in MLlib's
dependency set under default settings.
If no native library is available at runtime, you will see a warning message.
To use native libraries from `netlib-java`, please build Spark with `-Pnetlib-lgpl` or
include `com.github.fommil.netlib:all:1.1.2` as a dependency of your project.
If you want to use optimized BLAS/LAPACK libraries such as
[OpenBLAS](http://www.openblas.net/), please link its shared libraries to
`/usr/lib/libblas.so.3` and `/usr/lib/liblapack.so.3`, respectively.
BLAS/LAPACK libraries on worker nodes should be built without multithreading.
To use MLlib in Python, you will need [NumPy](http://www.numpy.org) version 1.4 or newer.