ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[docs] [SPARK-4820] Spark build encounters "File name too long" on some encrypted filesystems

Browse source 
Added a note instructing users how to build Spark in an encrypted file system.

Author: Theodore Vasiloudis <tvas@sics.se>

Closes #5041 from thvasilo/patch-2 and squashes the following commits:

09d890b [Theodore Vasiloudis] Workaroung for buiding in an encrypted filesystem
This commit is contained in:
Theodore Vasiloudis 2015-03-17 11:25:01 +00:00 committed by Sean Owen
parent 005d1c5f29
commit e26db9be47
1 changed files with 12 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
docs/building-spark.md
View file

@ -23,6 +23,18 @@ build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
Other build examples can be found below.
**Note:** When building on an encrypted filesystem (if your home directory is encrypted, for example), then the Spark build might fail with a "Filename too long" error. As a workaround, add the following in the configuration args of the `scala-maven-plugin` in the project `pom.xml`:
<arg>-Xmax-classfile-name</arg>
<arg>128</arg>
and in `project/SparkBuild.scala` add:
scalacOptions in Compile ++= Seq("-Xmax-classfile-name", "128"),
to the `sharedSettings` val. See also [this PR](https://github.com/apache/spark/pull/2883/files) if you are unsure of where to add these lines.
# Setting up Maven's Memory Usage
You'll need to configure Maven to use more memory than usual by setting `MAVEN_OPTS`. We recommend the following settings: