ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/project/SparkBuild.scala

477 lines
22 KiB
Scala
Raw Normal View History

Add Apache license headers and LICENSE and NOTICE files
2013-07-16 20:21:33 -04:00
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Add support for building against hadoop2-yarn : adding new maven profile for it
2013-04-07 08:17:38 -04:00
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
import sbt._
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 03:17:13 -04:00
import sbt.Classpaths.publishTask
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
import Keys._
Really upgrade to SBT 0.11.1 (through build.properties and plugin changes)
2011-11-09 00:45:29 -05:00
import sbtassembly.Plugin._
import AssemblyKeys._
fixed review comments
2014-01-03 04:07:42 -05:00
import scala.util.Properties
Merge pull request #557 from ScrapCodes/style. Closes #557. SPARK-1058, Fix Style Errors and Add Scala Style to Spark Build. Author: Patrick Wendell <pwendell@gmail.com> Author: Prashant Sharma <scrapcodes@gmail.com> == Merge branch commits == commit 1a8bd1c059b842cb95cc246aaea74a79fec684f4 Author: Prashant Sharma <scrapcodes@gmail.com> Date: Sun Feb 9 17:39:07 2014 +0530 scala style fixes commit f91709887a8e0b608c5c2b282db19b8a44d53a43 Author: Patrick Wendell <pwendell@gmail.com> Date: Fri Jan 24 11:22:53 2014 -0800 Adding scalastyle snapshot
2014-02-09 13:09:19 -05:00
import org.scalastyle.sbt.ScalastylePlugin.{Settings => ScalaStyleSettings}
Merge the old sbt-launch-lib.bash with the new sbt-launcher jar downloading logic. This allows developers to pass options (such as -D) to sbt. I also modified the SparkBuild to ensure spark specific properties are propagated to forked test JVMs. Author: Michael Armbrust <michael@databricks.com> Closes #14 from marmbrus/sbtScripts and squashes the following commits: c008b18 [Michael Armbrust] Merge the old sbt-launch-lib.bash with the new sbt-launcher jar downloading logic.
2014-03-02 03:35:23 -05:00
import scala.collection.JavaConversions._
Uncomment Maven publishing stuff and set version to 0.6.0
2012-10-13 18:55:39 -04:00
// For Sonatype publishing
Comment out PGP stuff for publish-local to work
2012-10-14 20:36:20 -04:00
//import com.jsuereth.pgp.sbtplugin.PgpKeys._
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
object SparkBuild extends Build {
Remove references to unsupported Hadoop versions
2013-08-21 20:12:03 -04:00
// Hadoop version to build against. For example, "1.0.4" for Apache releases, or
// "2.0.0-mr1-cdh4.2.0" for Cloudera Hadoop. Note that these variables can be set
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
// through the environment variables SPARK_HADOOP_VERSION and SPARK_YARN.
Downgraded default build hadoop version to 1.0.4.
2013-08-21 14:38:24 -04:00
val DEFAULT_HADOOP_VERSION = "1.0.4"
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
Add optional Hadoop 2.2 settings in sbt build. If the Hadoop used is version 2.2 or derived from it, then Spark will be compiled against protobuf-2.5 and a protobuf-2.5 version of Akka 2.0.5.
2013-11-23 20:08:01 -05:00
// Whether the Hadoop version to build against is 2.2.x, or a variant of it. This can be set
Use published "org.spark-project.akka-*" in sbt build for Hadoop-2.2 dependencies. This also includes: -Change `isNewYarn` to `isNewHadoop`, since the protobuf-2.5 dependency is from Hadoop-2.2 itself. -Regexp bugix Credits to @alig for this patch.
2013-12-01 19:50:24 -05:00
// through the SPARK_IS_NEW_HADOOP environment variable.
val DEFAULT_IS_NEW_HADOOP = false
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
val DEFAULT_YARN = false
Add support for building against hadoop2-yarn : adding new maven profile for it
2013-04-07 08:17:38 -04:00
fewer words
2013-08-09 16:54:41 -04:00
// HBase version; set as appropriate.
Pull HBASE_VERSION in the head of sbt build
2013-08-09 15:45:18 -04:00
val HBASE_VERSION = "0.94.6"
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
Fix target JVM version in scala build
2013-09-08 17:37:45 -04:00
// Target JVM version
scala 2.10 requires Java 1.6, using Scala 2.10.3, resolved maven-scala-plugin warning
2013-10-05 15:41:09 -04:00
val SCALAC_JVM_VERSION = "jvm-1.6"
val JAVAC_JVM_VERSION = "1.6"
Fix target JVM version in scala build
2013-09-08 17:37:45 -04:00
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
lazy val root = Project("root", file("."), settings = rootSettings) aggregate(allProjects: _*)
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
lazy val core = Project("core", file("core"), settings = coreSettings)
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
lazy val repl = Project("repl", file("repl"), settings = replSettings)
graph -> graphx
2014-01-09 17:31:33 -05:00
.dependsOn(core, graphx, bagel, mllib)
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
lazy val tools = Project("tools", file("tools"), settings = toolsSettings) dependsOn(core) dependsOn(streaming)
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
lazy val bagel = Project("bagel", file("bagel"), settings = bagelSettings) dependsOn(core)
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
graph -> graphx
2014-01-09 17:31:33 -05:00
lazy val graphx = Project("graphx", file("graphx"), settings = graphxSettings) dependsOn(core)
Maven and sbt build changes for SparkGraph.
2013-02-19 15:43:13 -05:00
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
lazy val streaming = Project("streaming", file("streaming"), settings = streamingSettings) dependsOn(core)
Added the Spark Streaing code, ported to Akka 2
2012-07-28 23:03:26 -04:00
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
lazy val mllib = Project("mllib", file("mllib"), settings = mllibSettings) dependsOn(core)
Import RidgeRegression example Conflicts: run
2013-07-05 14:13:41 -04:00
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
lazy val assemblyProj = Project("assembly", file("assembly"), settings = assemblyProjSettings)
SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues... This patch removes Ganglia integration from the default build. It allows users willing to link against LGPL code to use Ganglia by adding build flags or linking against a new Spark artifact called spark-ganglia-lgpl. This brings Spark in line with the Apache policy on LGPL code enumerated here: https://www.apache.org/legal/3party.html#options-optional Author: Patrick Wendell <pwendell@gmail.com> Closes #108 from pwendell/ganglia and squashes the following commits: 326712a [Patrick Wendell] Responding to review feedback 5f28ee4 [Patrick Wendell] SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues.
2014-03-11 14:16:59 -04:00
.dependsOn(core, graphx, bagel, mllib, repl, streaming) dependsOn(maybeYarn: _*) dependsOn(maybeGanglia: _*)
Import RidgeRegression example Conflicts: run
2013-07-05 14:13:41 -04:00
Rename SBT target to assemble-deps.
2013-10-16 20:05:46 -04:00
lazy val assembleDeps = TaskKey[Unit]("assemble-deps", "Build assembly of dependencies and packages Spark projects")
Add new SBT target for dependency assembly
2013-10-09 07:24:34 -04:00
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 03:17:13 -04:00
// A configuration to set an alternative publishLocalConfiguration
lazy val MavenCompile = config("m2r") extend(Compile)
publish-local should go to maven + ivy by default
2012-10-01 18:34:47 -04:00
lazy val publishLocalBoth = TaskKey[Unit]("publish-local", "publish local for m2 and ivy")
fixed review comments
2014-01-03 04:07:42 -05:00
val sparkHome = System.getProperty("user.dir")
Made java options to be applied during tests so that they become self explanatory.
2014-01-06 05:33:31 -05:00
Allow make-distribution.sh to specify Hadoop version used
2013-08-14 20:34:34 -04:00
// Allows build configuration to be set through environment variables
fixed review comments
2014-01-03 04:07:42 -05:00
lazy val hadoopVersion = Properties.envOrElse("SPARK_HADOOP_VERSION", DEFAULT_HADOOP_VERSION)
lazy val isNewHadoop = Properties.envOrNone("SPARK_IS_NEW_HADOOP") match {
Add optional Hadoop 2.2 settings in sbt build. If the Hadoop used is version 2.2 or derived from it, then Spark will be compiled against protobuf-2.5 and a protobuf-2.5 version of Akka 2.0.5.
2013-11-23 20:08:01 -05:00
case None => {
Use published "org.spark-project.akka-*" in sbt build for Hadoop-2.2 dependencies. This also includes: -Change `isNewYarn` to `isNewHadoop`, since the protobuf-2.5 dependency is from Hadoop-2.2 itself. -Regexp bugix Credits to @alig for this patch.
2013-12-01 19:50:24 -05:00
val isNewHadoopVersion = "2.[2-9]+".r.findFirstIn(hadoopVersion).isDefined
(isNewHadoopVersion|| DEFAULT_IS_NEW_HADOOP)
Add optional Hadoop 2.2 settings in sbt build. If the Hadoop used is version 2.2 or derived from it, then Spark will be compiled against protobuf-2.5 and a protobuf-2.5 version of Akka 2.0.5.
2013-11-23 20:08:01 -05:00
}
case Some(v) => v.toBoolean
}
Disabled yarn 2.2 and added a message in the sbt build
2013-12-12 05:50:10 -05:00
fixed review comments
2014-01-03 04:07:42 -05:00
lazy val isYarnEnabled = Properties.envOrNone("SPARK_YARN") match {
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
case None => DEFAULT_YARN
Allow make-distribution.sh to specify Hadoop version used
2013-08-14 20:34:34 -04:00
case Some(v) => v.toBoolean
}
SPARK-1071: Tidy logging strategy and use of log4j Prompted by a recent thread on the mailing list, I tried and failed to see if Spark can be made independent of log4j. There are a few cases where control of the underlying logging is pretty useful, and to do that, you have to bind to a specific logger. Instead I propose some tidying that leaves Spark's use of log4j, but gets rid of warnings and should still enable downstream users to switch. The idea is to pipe everything (except log4j) through SLF4J, and have Spark use SLF4J directly when logging, and where Spark needs to output info (REPL and tests), bind from SLF4J to log4j. This leaves the same behavior in Spark. It means that downstream users who want to use something except log4j should: - Exclude dependencies on log4j, slf4j-log4j12 from Spark - Include dependency on log4j-over-slf4j - Include dependency on another logger X, and another slf4j-X - Recreate any log config that Spark does, that is needed, in the other logger's config That sounds about right. Here are the key changes: - Include the jcl-over-slf4j shim everywhere by depending on it in core. - Exclude dependencies on commons-logging from third-party libraries. - Include the jul-to-slf4j shim everywhere by depending on it in core. - Exclude slf4j-* dependencies from third-party libraries to prevent collision or warnings - Added missing slf4j-log4j12 binding to GraphX, Bagel module tests And minor/incidental changes: - Update to SLF4J 1.7.5, which happily matches Hadoop 2’s version and is a recommended update over 1.7.2 - (Remove a duplicate HBase dependency declaration in SparkBuild.scala) - (Remove a duplicate mockito dependency declaration that was causing warnings and bugging me) Author: Sean Owen <sowen@cloudera.com> Closes #570 from srowen/SPARK-1071 and squashes the following commits: 52eac9f [Sean Owen] Add slf4j-over-log4j12 dependency to core (non-test) and remove it from things that depend on core. 77a7fa9 [Sean Owen] SPARK-1071: Tidy logging strategy and use of log4j
2014-02-23 14:40:55 -05:00
lazy val hadoopClient = if (hadoopVersion.startsWith("0.20.") || hadoopVersion == "1.0.0") "hadoop-core" else "hadoop-client"
SPARK-1121 Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set Author: Prashant Sharma <prashant.s@imaginea.com> Closes #6 from ScrapCodes/SPARK-1121/avro-dep-fix and squashes the following commits: 9b29e34 [Prashant Sharma] Review feedback on PR 46ed2ad [Prashant Sharma] SPARK-1121-Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
2014-02-27 02:40:49 -05:00
val maybeAvro = if (hadoopVersion.startsWith("0.23.") && isYarnEnabled) Seq("org.apache.avro" % "avro" % "1.7.4") else Seq()
[java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs Author: Prashant Sharma <prashant.s@imaginea.com> Author: Patrick Wendell <pwendell@gmail.com> Closes #17 from ScrapCodes/java8-lambdas and squashes the following commits: 95850e6 [Patrick Wendell] Some doc improvements and build changes to the Java 8 patch. 85a954e [Prashant Sharma] Nit. import orderings. 673f7ac [Prashant Sharma] Added support for -java-home as well 80a13e8 [Prashant Sharma] Used fake class tag syntax 26eb3f6 [Prashant Sharma] Patrick's comments on PR. 35d8d79 [Prashant Sharma] Specified java 8 building in the docs 31d4cd6 [Prashant Sharma] Maven build to support -Pjava8-tests flag. 4ab87d3 [Prashant Sharma] Review feedback on the pr c33dc2c [Prashant Sharma] SPARK-964, Java 8 API Support.
2014-03-04 01:31:30 -05:00
SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues... This patch removes Ganglia integration from the default build. It allows users willing to link against LGPL code to use Ganglia by adding build flags or linking against a new Spark artifact called spark-ganglia-lgpl. This brings Spark in line with the Apache policy on LGPL code enumerated here: https://www.apache.org/legal/3party.html#options-optional Author: Patrick Wendell <pwendell@gmail.com> Closes #108 from pwendell/ganglia and squashes the following commits: 326712a [Patrick Wendell] Responding to review feedback 5f28ee4 [Patrick Wendell] SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues.
2014-03-11 14:16:59 -04:00
// Include Ganglia integration if the user has enabled Ganglia
// This is isolated from the normal build due to LGPL-licensed code in the library
lazy val isGangliaEnabled = Properties.envOrNone("SPARK_GANGLIA_LGPL").isDefined
lazy val gangliaProj = Project("spark-ganglia-lgpl", file("extras/spark-ganglia-lgpl"), settings = gangliaSettings).dependsOn(core)
val maybeGanglia: Seq[ClasspathDependency] = if (isGangliaEnabled) Seq(gangliaProj) else Seq()
val maybeGangliaRef: Seq[ProjectReference] = if (isGangliaEnabled) Seq(gangliaProj) else Seq()
// Include the Java 8 project if the JVM version is 8+
[java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs Author: Prashant Sharma <prashant.s@imaginea.com> Author: Patrick Wendell <pwendell@gmail.com> Closes #17 from ScrapCodes/java8-lambdas and squashes the following commits: 95850e6 [Patrick Wendell] Some doc improvements and build changes to the Java 8 patch. 85a954e [Prashant Sharma] Nit. import orderings. 673f7ac [Prashant Sharma] Added support for -java-home as well 80a13e8 [Prashant Sharma] Used fake class tag syntax 26eb3f6 [Prashant Sharma] Patrick's comments on PR. 35d8d79 [Prashant Sharma] Specified java 8 building in the docs 31d4cd6 [Prashant Sharma] Maven build to support -Pjava8-tests flag. 4ab87d3 [Prashant Sharma] Review feedback on the pr c33dc2c [Prashant Sharma] SPARK-964, Java 8 API Support.
2014-03-04 01:31:30 -05:00
lazy val javaVersion = System.getProperty("java.specification.version")
lazy val isJava8Enabled = javaVersion.toDouble >= "1.8".toDouble
val maybeJava8Tests = if (isJava8Enabled) Seq[ProjectReference](java8Tests) else Seq[ProjectReference]()
lazy val java8Tests = Project("java8-tests", file("extras/java8-tests"), settings = java8TestsSettings).
dependsOn(core) dependsOn(streaming % "compile->compile;test->test")
SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues... This patch removes Ganglia integration from the default build. It allows users willing to link against LGPL code to use Ganglia by adding build flags or linking against a new Spark artifact called spark-ganglia-lgpl. This brings Spark in line with the Apache policy on LGPL code enumerated here: https://www.apache.org/legal/3party.html#options-optional Author: Patrick Wendell <pwendell@gmail.com> Closes #108 from pwendell/ganglia and squashes the following commits: 326712a [Patrick Wendell] Responding to review feedback 5f28ee4 [Patrick Wendell] SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues.
2014-03-11 14:16:59 -04:00
// Include the YARN project if the user has enabled YARN
Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2
2014-01-01 22:23:33 -05:00
lazy val yarnAlpha = Project("yarn-alpha", file("yarn/alpha"), settings = yarnAlphaSettings) dependsOn(core)
lazy val yarn = Project("yarn", file("yarn/stable"), settings = yarnSettings) dependsOn(core)
Disabled yarn 2.2 and added a message in the sbt build
2013-12-12 05:50:10 -05:00
SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues... This patch removes Ganglia integration from the default build. It allows users willing to link against LGPL code to use Ganglia by adding build flags or linking against a new Spark artifact called spark-ganglia-lgpl. This brings Spark in line with the Apache policy on LGPL code enumerated here: https://www.apache.org/legal/3party.html#options-optional Author: Patrick Wendell <pwendell@gmail.com> Closes #108 from pwendell/ganglia and squashes the following commits: 326712a [Patrick Wendell] Responding to review feedback 5f28ee4 [Patrick Wendell] SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues.
2014-03-11 14:16:59 -04:00
lazy val maybeYarn: Seq[ClasspathDependency] = if (isYarnEnabled) Seq(if (isNewHadoop) yarn else yarnAlpha) else Seq()
lazy val maybeYarnRef: Seq[ProjectReference] = if (isYarnEnabled) Seq(if (isNewHadoop) yarn else yarnAlpha) else Seq()
Allow make-distribution.sh to specify Hadoop version used
2013-08-14 20:34:34 -04:00
Upgrade Kafka dependecy to 0.8.0 release version
2014-01-09 19:53:59 -05:00
lazy val externalTwitter = Project("external-twitter", file("external/twitter"), settings = twitterSettings)
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
.dependsOn(streaming % "compile->compile;test->test")
lazy val externalKafka = Project("external-kafka", file("external/kafka"), settings = kafkaSettings)
.dependsOn(streaming % "compile->compile;test->test")
lazy val externalFlume = Project("external-flume", file("external/flume"), settings = flumeSettings)
.dependsOn(streaming % "compile->compile;test->test")
Upgrade Kafka dependecy to 0.8.0 release version
2014-01-09 19:53:59 -05:00
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
lazy val externalZeromq = Project("external-zeromq", file("external/zeromq"), settings = zeromqSettings)
.dependsOn(streaming % "compile->compile;test->test")
Upgrade Kafka dependecy to 0.8.0 release version
2014-01-09 19:53:59 -05:00
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
lazy val externalMqtt = Project("external-mqtt", file("external/mqtt"), settings = mqttSettings)
.dependsOn(streaming % "compile->compile;test->test")
lazy val allExternal = Seq[ClasspathDependency](externalTwitter, externalKafka, externalFlume, externalZeromq, externalMqtt)
lazy val allExternalRefs = Seq[ProjectReference](externalTwitter, externalKafka, externalFlume, externalZeromq, externalMqtt)
Upgrade Kafka dependecy to 0.8.0 release version
2014-01-09 19:53:59 -05:00
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
lazy val examples = Project("examples", file("examples"), settings = examplesSettings)
graph -> graphx
2014-01-09 17:31:33 -05:00
.dependsOn(core, mllib, graphx, bagel, streaming, externalTwitter) dependsOn(allExternal: _*)
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
[java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs Author: Prashant Sharma <prashant.s@imaginea.com> Author: Patrick Wendell <pwendell@gmail.com> Closes #17 from ScrapCodes/java8-lambdas and squashes the following commits: 95850e6 [Patrick Wendell] Some doc improvements and build changes to the Java 8 patch. 85a954e [Prashant Sharma] Nit. import orderings. 673f7ac [Prashant Sharma] Added support for -java-home as well 80a13e8 [Prashant Sharma] Used fake class tag syntax 26eb3f6 [Prashant Sharma] Patrick's comments on PR. 35d8d79 [Prashant Sharma] Specified java 8 building in the docs 31d4cd6 [Prashant Sharma] Maven build to support -Pjava8-tests flag. 4ab87d3 [Prashant Sharma] Review feedback on the pr c33dc2c [Prashant Sharma] SPARK-964, Java 8 API Support.
2014-03-04 01:31:30 -05:00
// Everything except assembly, tools, java8Tests and examples belong to packageProjects
SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues... This patch removes Ganglia integration from the default build. It allows users willing to link against LGPL code to use Ganglia by adding build flags or linking against a new Spark artifact called spark-ganglia-lgpl. This brings Spark in line with the Apache policy on LGPL code enumerated here: https://www.apache.org/legal/3party.html#options-optional Author: Patrick Wendell <pwendell@gmail.com> Closes #108 from pwendell/ganglia and squashes the following commits: 326712a [Patrick Wendell] Responding to review feedback 5f28ee4 [Patrick Wendell] SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues.
2014-03-11 14:16:59 -04:00
lazy val packageProjects = Seq[ProjectReference](core, repl, bagel, streaming, mllib, graphx) ++ maybeYarnRef ++ maybeGangliaRef
Add new SBT target for dependency assembly
2013-10-09 07:24:34 -04:00
[java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs Author: Prashant Sharma <prashant.s@imaginea.com> Author: Patrick Wendell <pwendell@gmail.com> Closes #17 from ScrapCodes/java8-lambdas and squashes the following commits: 95850e6 [Patrick Wendell] Some doc improvements and build changes to the Java 8 patch. 85a954e [Prashant Sharma] Nit. import orderings. 673f7ac [Prashant Sharma] Added support for -java-home as well 80a13e8 [Prashant Sharma] Used fake class tag syntax 26eb3f6 [Prashant Sharma] Patrick's comments on PR. 35d8d79 [Prashant Sharma] Specified java 8 building in the docs 31d4cd6 [Prashant Sharma] Maven build to support -Pjava8-tests flag. 4ab87d3 [Prashant Sharma] Review feedback on the pr c33dc2c [Prashant Sharma] SPARK-964, Java 8 API Support.
2014-03-04 01:31:30 -05:00
lazy val allProjects = packageProjects ++ allExternalRefs ++
Seq[ProjectReference](examples, tools, assemblyProj) ++ maybeJava8Tests
Allow make-distribution.sh to specify Hadoop version used
2013-08-14 20:34:34 -04:00
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
def sharedSettings = Defaults.defaultSettings ++ Seq(
Merged with master
2013-09-06 08:23:01 -04:00
organization := "org.apache.spark",
Remove remaining references to incubation This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier. Author: Patrick Wendell <pwendell@gmail.com> Closes #51 from pwendell/tlp and squashes the following commits: d553b1b [Patrick Wendell] Remove remaining references to incubation
2014-03-02 04:00:16 -05:00
version := "1.0.0-SNAPSHOT",
scala 2.10 requires Java 1.6, using Scala 2.10.3, resolved maven-scala-plugin warning
2013-10-05 15:41:09 -04:00
scalaVersion := "2.10.3",
Removed scala -optimize flag.
2013-09-26 16:58:10 -04:00
scalacOptions := Seq("-Xmax-classfile-name", "120", "-unchecked", "-deprecation",
Fix target JVM version in scala build
2013-09-08 17:37:45 -04:00
"-target:" + SCALAC_JVM_VERSION),
javacOptions := Seq("-target", JAVAC_JVM_VERSION, "-source", JAVAC_JVM_VERSION),
Only include JAR files in lib in the unmanaged classpath.
2011-08-30 01:58:53 -04:00
unmanagedJars in Compile <<= baseDirectory map { base => (base / "lib" ** "*.jar").classpath },
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
retrieveManaged := true,
[java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs Author: Prashant Sharma <prashant.s@imaginea.com> Author: Patrick Wendell <pwendell@gmail.com> Closes #17 from ScrapCodes/java8-lambdas and squashes the following commits: 95850e6 [Patrick Wendell] Some doc improvements and build changes to the Java 8 patch. 85a954e [Prashant Sharma] Nit. import orderings. 673f7ac [Prashant Sharma] Added support for -java-home as well 80a13e8 [Prashant Sharma] Used fake class tag syntax 26eb3f6 [Prashant Sharma] Patrick's comments on PR. 35d8d79 [Prashant Sharma] Specified java 8 building in the docs 31d4cd6 [Prashant Sharma] Maven build to support -Pjava8-tests flag. 4ab87d3 [Prashant Sharma] Review feedback on the pr c33dc2c [Prashant Sharma] SPARK-964, Java 8 API Support.
2014-03-04 01:31:30 -05:00
javaHome := Properties.envOrNone("JAVA_HOME").map(file),
SPARK-1121 Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set Author: Prashant Sharma <prashant.s@imaginea.com> Closes #6 from ScrapCodes/SPARK-1121/avro-dep-fix and squashes the following commits: 9b29e34 [Prashant Sharma] Review feedback on PR 46ed2ad [Prashant Sharma] SPARK-1121-Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
2014-02-27 02:40:49 -05:00
// This is to add convenience of enabling sbt -Dsbt.offline=true for making the build offline.
offline := "true".equalsIgnoreCase(sys.props("sbt.offline")),
Retrieve jars to a flat directory so * can be used for the classpath.
2013-01-08 15:44:33 -05:00
retrievePattern := "[type]s/[artifact](-[revision])(-[classifier]).[ext]",
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
transitiveClassifiers in Scope.GlobalScope := Seq("sources"),
Add and configure junit_xml_listener as a replacement for XmlTestReport.
2011-07-20 20:04:29 -04:00
testListeners <<= target.map(t => Seq(new eu.henkelmann.sbt.JUnitXmlTestsListener(t.getAbsolutePath))),
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
Turn on forking in test JVMs to reduce the pressure on perm gen and code cache sizes due to having 2 instances of the Scala compiler and a bunch of classloaders.
2013-04-19 01:25:21 -04:00
// Fork new JVMs for tests and set Java options for those
fork := true,
Made java options to be applied during tests so that they become self explanatory.
2014-01-06 05:33:31 -05:00
javaOptions in Test += "-Dspark.home=" + sparkHome,
javaOptions in Test += "-Dspark.testing=1",
Merge the old sbt-launch-lib.bash with the new sbt-launcher jar downloading logic. This allows developers to pass options (such as -D) to sbt. I also modified the SparkBuild to ensure spark specific properties are propagated to forked test JVMs. Author: Michael Armbrust <michael@databricks.com> Closes #14 from marmbrus/sbtScripts and squashes the following commits: c008b18 [Michael Armbrust] Merge the old sbt-launch-lib.bash with the new sbt-launcher jar downloading logic.
2014-03-02 03:35:23 -05:00
javaOptions in Test ++= System.getProperties.filter(_._1 startsWith "spark").map { case (k,v) => s"-D$k=$v" }.toSeq,
Provide more memory for tests
2013-08-24 02:25:49 -04:00
javaOptions += "-Xmx3g",
Show full stack trace and time taken in unit tests.
2013-12-24 00:20:20 -05:00
// Show full stack trace and duration in test cases.
testOptions in Test += Tests.Argument("-oDF"),
Adjusted visibility of various components.
2014-01-13 22:58:53 -05:00
// Remove certain packages from Scaladoc
scalacOptions in (Compile,doc) := Seq("-skip-packages", Seq(
"akka",
"org.apache.spark.network",
"org.apache.spark.deploy",
"org.apache.spark.util.collection"
).mkString(":")),
Show full stack trace and time taken in unit tests.
2013-12-24 00:20:20 -05:00
Fix tests from different projects running in parallel in SBT 0.12
2013-04-11 22:29:37 -04:00
// Only allow one test at a time, even across projects, since they run in the same JVM
concurrentRestrictions in Global += Tags.limit(Tags.Test, 1),
ask ivy/sbt to check local maven repo under ~/.m2
2013-10-01 18:46:51 -04:00
// also check the local Maven repository ~/.m2
resolvers ++= Seq(Resolver.file("Local Maven Repo", file(Path.userHome + "/.m2/repository"))),
Added pom.xml for external projects and removed unnecessary dependencies and repositoris from other poms and sbt.
2013-12-31 03:28:57 -05:00
// For Sonatype publishing
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
resolvers ++= Seq("sonatype-snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
"sonatype-staging" at "https://oss.sonatype.org/service/local/staging/deploy/maven2/"),
publishMavenStyle := true,
Comment out PGP stuff for publish-local to work
2012-10-14 20:36:20 -04:00
//useGpg in Global := true,
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
pomExtra := (
Add Apache parent POM
2013-09-02 21:34:03 -04:00
<parent>
<groupId>org.apache</groupId>
<artifactId>apache</artifactId>
<version>13</version>
</parent>
Remove remaining references to incubation This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier. Author: Patrick Wendell <pwendell@gmail.com> Closes #51 from pwendell/tlp and squashes the following commits: d553b1b [Patrick Wendell] Remove remaining references to incubation
2014-03-02 04:00:16 -05:00
<url>http://spark.apache.org/</url>
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
<licenses>
Changing tabs to spaces
2012-10-10 21:54:22 -04:00
<license>
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
<name>Apache 2.0 License</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.html</url>
Changing tabs to spaces
2012-10-10 21:54:22 -04:00
<distribution>repo</distribution>
</license>
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
</licenses>
<scm>
Remove remaining references to incubation This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier. Author: Patrick Wendell <pwendell@gmail.com> Closes #51 from pwendell/tlp and squashes the following commits: d553b1b [Patrick Wendell] Remove remaining references to incubation
2014-03-02 04:00:16 -05:00
<connection>scm:git:git@github.com:apache/spark.git</connection>
<url>scm:git:git@github.com:apache/spark.git</url>
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
</scm>
<developers>
Changing tabs to spaces
2012-10-10 21:54:22 -04:00
<developer>
<id>matei</id>
<name>Matei Zaharia</name>
<email>matei.zaharia@gmail.com</email>
<url>http://www.cs.berkeley.edu/~matei</url>
Fix some URLs
2013-09-01 01:45:53 -04:00
<organization>Apache Software Foundation</organization>
Remove remaining references to incubation This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier. Author: Patrick Wendell <pwendell@gmail.com> Closes #51 from pwendell/tlp and squashes the following commits: d553b1b [Patrick Wendell] Remove remaining references to incubation
2014-03-02 04:00:16 -05:00
<organizationUrl>http://spark.apache.org</organizationUrl>
Changing tabs to spaces
2012-10-10 21:54:22 -04:00
</developer>
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
</developers>
Initial work to rename package to org.apache.spark
2013-08-31 22:27:07 -04:00
<issueManagement>
<system>JIRA</system>
<url>https://spark-project.atlassian.net/browse/SPARK</url>
</issueManagement>
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
),
Uncomment Maven publishing stuff and set version to 0.6.0
2012-10-13 18:55:39 -04:00
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
/*
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
publishTo <<= version { (v: String) =>
val nexus = "https://oss.sonatype.org/"
Uncomment Maven publishing stuff and set version to 0.6.0
2012-10-13 18:55:39 -04:00
if (v.trim.endsWith("SNAPSHOT"))
Some("sonatype-snapshots" at nexus + "content/repositories/snapshots")
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
else
Changing tabs to spaces
2012-10-10 21:54:22 -04:00
Some("sonatype-staging" at nexus + "service/local/staging/deploy/maven2")
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
},
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
*/
Merge pull request #31 from sundeepn/branch-0.8 Resolving package conflicts with hadoop 0.23.9 Hadoop 0.23.9 is having a package conflict with easymock's dependencies. (cherry picked from commit 023e3fdf008b3194a36985a07923df9aaf64e520) Signed-off-by: Reynold Xin <rxin@apache.org>
2013-10-07 13:47:45 -04:00
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
libraryDependencies ++= Seq(
Update io.netty from 4.0.13 Final to 4.0.17.Final This update contains a lot of bug fixes and some new perf improvements. It is also binary compatible with the current 4.0.13.Final For more information: http://netty.io/news/2014/02/25/4-0-17-Final.html Author: Binh Nguyen <ngbinh@gmail.com> Author: Binh Nguyen <ngbinh@gmail.com> Closes #41 from ngbinh/master and squashes the following commits: a9498f4 [Binh Nguyen] update io.netty to 4.0.17.Final
2014-03-02 03:48:50 -05:00
"io.netty" % "netty-all" % "4.0.17.Final",
SPARK-1236 - Upgrade Jetty to 9.1.3.v20140225. Author: Reynold Xin <rxin@apache.org> Closes #113 from rxin/jetty9 and squashes the following commits: 867a2ce [Reynold Xin] Updated Jetty version to 9.1.3.v20140225 in Maven build file. d7c97ca [Reynold Xin] Return the correctly bound port. d14706f [Reynold Xin] Upgrade Jetty to 9.1.3.v20140225.
2014-03-13 15:16:04 -04:00
"org.eclipse.jetty" % "jetty-server" % "9.1.3.v20140225",
"org.eclipse.jetty" % "jetty-util" % "9.1.3.v20140225",
"org.eclipse.jetty" % "jetty-plus" % "9.1.3.v20140225",
"org.eclipse.jetty" % "jetty-security" % "9.1.3.v20140225",
[SPARK-959] Explicitly depend on org.eclipse.jetty.orbit jar Without this, in some cases, Ivy attempts to download the wrong file and fails, stopping the whole build. See bug for more details. (This is probably also the beginning of the slow death of our recently prettified dependencies. Form follow function.)
2013-11-18 01:37:46 -05:00
/** Workaround for SPARK-959. Dependency used by org.eclipse.jetty. Fixed in ivy 2.3.0. */
"org.eclipse.jetty.orbit" % "javax.servlet" % "2.5.0.v201103041518" artifacts Artifact("javax.servlet", "jar", "jar"),
Fixed a few test issues due to Akka 2.1, as well as SBT memory. Unfortunately, in Akka 2.1, ActorSystem.awaitTermination hangs for remote actors, and Akka also leaves a non-daemon Netty thread even when run in daemon mode. Thus I had to comment out some of the calls to awaitTermination, and we still have one failing test.
2013-06-08 04:09:24 -04:00
"org.scalatest" %% "scalatest" % "1.9.1" % "test",
"org.scalacheck" %% "scalacheck" % "1.10.0" % "test",
Increase JUnit test verbosity under SBT. Upgrade junit-interface plugin from 0.9 to 0.10. I noticed that the JavaAPISuite tests didn't appear to display any output locally or under Jenkins, making it difficult to know whether they were running. This change increases the verbosity to more closely match the ScalaTest tests.
2014-01-25 19:32:44 -05:00
"com.novocode" % "junit-interface" % "0.10" % "test",
Adding commons io as dependency
2013-07-15 02:33:48 -04:00
"org.easymock" % "easymock" % "3.1" % "test",
Merge branch 'master' into scala-2.10
2013-11-13 23:44:19 -05:00
"org.mockito" % "mockito-all" % "1.8.5" % "test",
Adding commons io as dependency
2013-07-15 02:33:48 -04:00
"commons-io" % "commons-io" % "2.4" % "test"
Fixed a few test issues due to Akka 2.1, as well as SBT memory. Unfortunately, in Akka 2.1, ActorSystem.awaitTermination hangs for remote actors, and Akka also leaves a non-daemon Netty thread even when run in daemon mode. Thus I had to comment out some of the calls to awaitTermination, and we still have one failing test.
2013-06-08 04:09:24 -04:00
),
Increase JUnit test verbosity under SBT. Upgrade junit-interface plugin from 0.9 to 0.10. I noticed that the JavaAPISuite tests didn't appear to display any output locally or under Jenkins, making it difficult to know whether they were running. This change increases the verbosity to more closely match the ScalaTest tests.
2014-01-25 19:32:44 -05:00
testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
Updating to latest akka 2.2.3, which fixes our only failing Driver Suite
2013-10-24 06:41:40 -04:00
parallelExecution := true,
Add workaround for bug in SBT (issue #206).
2011-09-26 19:04:59 -04:00
/* Workaround for issue #206 (fixed after SBT 0.11.0) */
watchTransitiveSources <<= Defaults.inDependencies[Task[Seq[File]]](watchSources.task,
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 03:17:13 -04:00
const(std.TaskExtra.constant(Nil)), aggregate = true, includeRoot = true) apply { _.join.map(_.flatten) },
otherResolvers := Seq(Resolver.file("dotM2", file(Path.userHome + "/.m2/repository"))),
publishLocalConfiguration in MavenCompile <<= (packagedArtifacts, deliverLocal, ivyLoggingLevel) map {
(arts, _, level) => new PublishConfiguration(None, "dotM2", arts, Seq(), level)
},
publishMavenStyle in MavenCompile := true,
publishLocal in MavenCompile <<= publishTask(publishLocalConfiguration in MavenCompile, deliverLocal),
publishLocalBoth <<= Seq(publishLocal in MavenCompile, publishLocal).dependOn
Merge pull request #557 from ScrapCodes/style. Closes #557. SPARK-1058, Fix Style Errors and Add Scala Style to Spark Build. Author: Patrick Wendell <pwendell@gmail.com> Author: Prashant Sharma <scrapcodes@gmail.com> == Merge branch commits == commit 1a8bd1c059b842cb95cc246aaea74a79fec684f4 Author: Prashant Sharma <scrapcodes@gmail.com> Date: Sun Feb 9 17:39:07 2014 +0530 scala style fixes commit f91709887a8e0b608c5c2b282db19b8a44d53a43 Author: Patrick Wendell <pwendell@gmail.com> Date: Fri Jan 24 11:22:53 2014 -0800 Adding scalastyle snapshot
2014-02-09 13:09:19 -05:00
) ++ net.virtualvoid.sbt.graph.Plugin.graphSettings ++ ScalaStyleSettings
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
SPARK-1071: Tidy logging strategy and use of log4j Prompted by a recent thread on the mailing list, I tried and failed to see if Spark can be made independent of log4j. There are a few cases where control of the underlying logging is pretty useful, and to do that, you have to bind to a specific logger. Instead I propose some tidying that leaves Spark's use of log4j, but gets rid of warnings and should still enable downstream users to switch. The idea is to pipe everything (except log4j) through SLF4J, and have Spark use SLF4J directly when logging, and where Spark needs to output info (REPL and tests), bind from SLF4J to log4j. This leaves the same behavior in Spark. It means that downstream users who want to use something except log4j should: - Exclude dependencies on log4j, slf4j-log4j12 from Spark - Include dependency on log4j-over-slf4j - Include dependency on another logger X, and another slf4j-X - Recreate any log config that Spark does, that is needed, in the other logger's config That sounds about right. Here are the key changes: - Include the jcl-over-slf4j shim everywhere by depending on it in core. - Exclude dependencies on commons-logging from third-party libraries. - Include the jul-to-slf4j shim everywhere by depending on it in core. - Exclude slf4j-* dependencies from third-party libraries to prevent collision or warnings - Added missing slf4j-log4j12 binding to GraphX, Bagel module tests And minor/incidental changes: - Update to SLF4J 1.7.5, which happily matches Hadoop 2’s version and is a recommended update over 1.7.2 - (Remove a duplicate HBase dependency declaration in SparkBuild.scala) - (Remove a duplicate mockito dependency declaration that was causing warnings and bugging me) Author: Sean Owen <sowen@cloudera.com> Closes #570 from srowen/SPARK-1071 and squashes the following commits: 52eac9f [Sean Owen] Add slf4j-over-log4j12 dependency to core (non-test) and remove it from things that depend on core. 77a7fa9 [Sean Owen] SPARK-1071: Tidy logging strategy and use of log4j
2014-02-23 14:40:55 -05:00
val slf4jVersion = "1.7.5"
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
Fixed a few test issues due to Akka 2.1, as well as SBT memory. Unfortunately, in Akka 2.1, ActorSystem.awaitTermination hangs for remote actors, and Akka also leaves a non-daemon Netty thread even when run in daemon mode. Thus I had to comment out some of the calls to awaitTermination, and we still have one failing test.
2013-06-08 04:09:24 -04:00
val excludeNetty = ExclusionRule(organization = "org.jboss.netty")
SPARK-782 Clean up for ASM dependency. This makes two changes. 1) Spark uses the shaded version of asm that is (conveniently) published with Kryo. 2) Existing exclude rules around asm are updated to reflect the new groupId of `org.ow2.asm`. This made all of the old rules not work with newer Hadoop versions that pull in new asm versions. Author: Patrick Wendell <pwendell@gmail.com> Closes #100 from pwendell/asm and squashes the following commits: 9235f3f [Patrick Wendell] SPARK-782 Clean up for ASM dependency.
2014-03-09 16:17:07 -04:00
val excludeAsm = ExclusionRule(organization = "org.ow2.asm")
val excludeOldAsm = ExclusionRule(organization = "asm")
SPARK-1071: Tidy logging strategy and use of log4j Prompted by a recent thread on the mailing list, I tried and failed to see if Spark can be made independent of log4j. There are a few cases where control of the underlying logging is pretty useful, and to do that, you have to bind to a specific logger. Instead I propose some tidying that leaves Spark's use of log4j, but gets rid of warnings and should still enable downstream users to switch. The idea is to pipe everything (except log4j) through SLF4J, and have Spark use SLF4J directly when logging, and where Spark needs to output info (REPL and tests), bind from SLF4J to log4j. This leaves the same behavior in Spark. It means that downstream users who want to use something except log4j should: - Exclude dependencies on log4j, slf4j-log4j12 from Spark - Include dependency on log4j-over-slf4j - Include dependency on another logger X, and another slf4j-X - Recreate any log config that Spark does, that is needed, in the other logger's config That sounds about right. Here are the key changes: - Include the jcl-over-slf4j shim everywhere by depending on it in core. - Exclude dependencies on commons-logging from third-party libraries. - Include the jul-to-slf4j shim everywhere by depending on it in core. - Exclude slf4j-* dependencies from third-party libraries to prevent collision or warnings - Added missing slf4j-log4j12 binding to GraphX, Bagel module tests And minor/incidental changes: - Update to SLF4J 1.7.5, which happily matches Hadoop 2’s version and is a recommended update over 1.7.2 - (Remove a duplicate HBase dependency declaration in SparkBuild.scala) - (Remove a duplicate mockito dependency declaration that was causing warnings and bugging me) Author: Sean Owen <sowen@cloudera.com> Closes #570 from srowen/SPARK-1071 and squashes the following commits: 52eac9f [Sean Owen] Add slf4j-over-log4j12 dependency to core (non-test) and remove it from things that depend on core. 77a7fa9 [Sean Owen] SPARK-1071: Tidy logging strategy and use of log4j
2014-02-23 14:40:55 -05:00
val excludeCommonsLogging = ExclusionRule(organization = "commons-logging")
val excludeSLF4J = ExclusionRule(organization = "org.slf4j")
SPARK-1084.2 (resubmitted) (Ported from https://github.com/apache/incubator-spark/pull/650 ) This adds one more change though, to fix the scala version warning introduced by json4s recently. Author: Sean Owen <sowen@cloudera.com> Closes #32 from srowen/SPARK-1084.2 and squashes the following commits: 9240abd [Sean Owen] Avoid scala version conflict in scalap induced by json4s dependency 1561cec [Sean Owen] Remove "exclude *" dependencies that are causing Maven warnings, and that are apparently unneeded anyway
2014-03-02 17:27:53 -05:00
val excludeScalap = ExclusionRule(organization = "org.scala-lang", artifact = "scalap")
Fixed a few test issues due to Akka 2.1, as well as SBT memory. Unfortunately, in Akka 2.1, ActorSystem.awaitTermination hangs for remote actors, and Akka also leaves a non-daemon Netty thread even when run in daemon mode. Thus I had to comment out some of the calls to awaitTermination, and we still have one failing test.
2013-06-08 04:09:24 -04:00
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 17:41:07 -04:00
def coreSettings = sharedSettings ++ Seq(
name := "spark-core",
Add dependency on Akka and Netty
2011-12-15 07:21:14 -05:00
resolvers ++= Seq(
fixed maven build for scala 2.10
2013-09-26 01:18:24 -04:00
"JBoss Repository" at "http://repository.jboss.org/nexus/content/repositories/releases/",
Akka 2.2 migration
2013-09-21 22:50:12 -04:00
"Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/"
Add dependency on Akka and Netty
2011-12-15 07:21:14 -05:00
),
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 17:41:07 -04:00
libraryDependencies ++= Seq(
SPARK-1121 Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set Author: Prashant Sharma <prashant.s@imaginea.com> Closes #6 from ScrapCodes/SPARK-1121/avro-dep-fix and squashes the following commits: 9b29e34 [Prashant Sharma] Review feedback on PR 46ed2ad [Prashant Sharma] SPARK-1121-Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
2014-02-27 02:40:49 -05:00
"com.google.guava" % "guava" % "14.0.1",
"com.google.code.findbugs" % "jsr305" % "1.3.9",
"log4j" % "log4j" % "1.2.17",
"org.slf4j" % "slf4j-api" % slf4jVersion,
"org.slf4j" % "slf4j-log4j12" % slf4jVersion,
"org.slf4j" % "jul-to-slf4j" % slf4jVersion,
"org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
"commons-daemon" % "commons-daemon" % "1.0.10", // workaround for bug HADOOP-9407
"com.ning" % "compress-lzf" % "1.0.0",
"org.xerial.snappy" % "snappy-java" % "1.0.5",
"org.spark-project.akka" %% "akka-remote" % "2.2.3-shaded-protobuf" excludeAll(excludeNetty),
"org.spark-project.akka" %% "akka-slf4j" % "2.2.3-shaded-protobuf" excludeAll(excludeNetty),
"org.spark-project.akka" %% "akka-testkit" % "2.2.3-shaded-protobuf" % "test",
SPARK-1084.2 (resubmitted) (Ported from https://github.com/apache/incubator-spark/pull/650 ) This adds one more change though, to fix the scala version warning introduced by json4s recently. Author: Sean Owen <sowen@cloudera.com> Closes #32 from srowen/SPARK-1084.2 and squashes the following commits: 9240abd [Sean Owen] Avoid scala version conflict in scalap induced by json4s dependency 1561cec [Sean Owen] Remove "exclude *" dependencies that are causing Maven warnings, and that are apparently unneeded anyway
2014-03-02 17:27:53 -05:00
"org.json4s" %% "json4s-jackson" % "3.2.6" excludeAll(excludeScalap),
SPARK-1121 Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set Author: Prashant Sharma <prashant.s@imaginea.com> Closes #6 from ScrapCodes/SPARK-1121/avro-dep-fix and squashes the following commits: 9b29e34 [Prashant Sharma] Review feedback on PR 46ed2ad [Prashant Sharma] SPARK-1121-Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
2014-02-27 02:40:49 -05:00
"it.unimi.dsi" % "fastutil" % "6.4.4",
"colt" % "colt" % "1.2.0",
"org.apache.mesos" % "mesos" % "0.13.0",
SPARK-1189: Add Security to Spark - Akka, Http, ConnectionManager, UI use servlets resubmit pull request. was https://github.com/apache/incubator-spark/pull/332. Author: Thomas Graves <tgraves@apache.org> Closes #33 from tgravescs/security-branch-0.9-with-client-rebase and squashes the following commits: dfe3918 [Thomas Graves] Fix merge conflict since startUserClass now using runAsUser 05eebed [Thomas Graves] Fix dependency lost in upmerge d1040ec [Thomas Graves] Fix up various imports 05ff5e0 [Thomas Graves] Fix up imports after upmerging to master ac046b3 [Thomas Graves] Merge remote-tracking branch 'upstream/master' into security-branch-0.9-with-client-rebase 13733e1 [Thomas Graves] Pass securityManager and SparkConf around where we can. Switch to use sparkConf for reading config whereever possible. Added ConnectionManagerSuite unit tests. 4a57acc [Thomas Graves] Change UI createHandler routines to createServlet since they now return servlets 2f77147 [Thomas Graves] Rework from comments 50dd9f2 [Thomas Graves] fix header in SecurityManager ecbfb65 [Thomas Graves] Fix spacing and formatting b514bec [Thomas Graves] Fix reference to config ed3d1c1 [Thomas Graves] Add security.md 6f7ddf3 [Thomas Graves] Convert SaslClient and SaslServer to scala, change spark.authenticate.ui to spark.ui.acls.enable, and fix up various other things from review comments 2d9e23e [Thomas Graves] Merge remote-tracking branch 'upstream/master' into security-branch-0.9-with-client-rebase_rework 5721c5a [Thomas Graves] update AkkaUtilsSuite test for the actorSelection changes, fix typos based on comments, and remove extra lines I missed in rebase from AkkaUtils f351763 [Thomas Graves] Add Security to Spark - Akka, Http, ConnectionManager, UI to use servlets
2014-03-06 19:27:50 -05:00
"commons-net" % "commons-net" % "2.2",
SPARK-1121 Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set Author: Prashant Sharma <prashant.s@imaginea.com> Closes #6 from ScrapCodes/SPARK-1121/avro-dep-fix and squashes the following commits: 9b29e34 [Prashant Sharma] Review feedback on PR 46ed2ad [Prashant Sharma] SPARK-1121-Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
2014-02-27 02:40:49 -05:00
"net.java.dev.jets3t" % "jets3t" % "0.7.1" excludeAll(excludeCommonsLogging),
"org.apache.derby" % "derby" % "10.4.2.0" % "test",
SPARK-782 Clean up for ASM dependency. This makes two changes. 1) Spark uses the shaded version of asm that is (conveniently) published with Kryo. 2) Existing exclude rules around asm are updated to reflect the new groupId of `org.ow2.asm`. This made all of the old rules not work with newer Hadoop versions that pull in new asm versions. Author: Patrick Wendell <pwendell@gmail.com> Closes #100 from pwendell/asm and squashes the following commits: 9235f3f [Patrick Wendell] SPARK-782 Clean up for ASM dependency.
2014-03-09 16:17:07 -04:00
"org.apache.hadoop" % hadoopClient % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm),
SPARK-1121 Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set Author: Prashant Sharma <prashant.s@imaginea.com> Closes #6 from ScrapCodes/SPARK-1121/avro-dep-fix and squashes the following commits: 9b29e34 [Prashant Sharma] Review feedback on PR 46ed2ad [Prashant Sharma] SPARK-1121-Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
2014-02-27 02:40:49 -05:00
"org.apache.curator" % "curator-recipes" % "2.4.0" excludeAll(excludeNetty),
"com.codahale.metrics" % "metrics-core" % "3.0.0",
"com.codahale.metrics" % "metrics-jvm" % "3.0.0",
"com.codahale.metrics" % "metrics-json" % "3.0.0",
"com.codahale.metrics" % "metrics-graphite" % "3.0.0",
SPARK-782 Clean up for ASM dependency. This makes two changes. 1) Spark uses the shaded version of asm that is (conveniently) published with Kryo. 2) Existing exclude rules around asm are updated to reflect the new groupId of `org.ow2.asm`. This made all of the old rules not work with newer Hadoop versions that pull in new asm versions. Author: Patrick Wendell <pwendell@gmail.com> Closes #100 from pwendell/asm and squashes the following commits: 9235f3f [Patrick Wendell] SPARK-782 Clean up for ASM dependency.
2014-03-09 16:17:07 -04:00
"com.twitter" %% "chill" % "0.3.1" excludeAll(excludeAsm),
"com.twitter" % "chill-java" % "0.3.1" excludeAll(excludeAsm),
SPARK-1121 Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set Author: Prashant Sharma <prashant.s@imaginea.com> Closes #6 from ScrapCodes/SPARK-1121/avro-dep-fix and squashes the following commits: 9b29e34 [Prashant Sharma] Review feedback on PR 46ed2ad [Prashant Sharma] SPARK-1121-Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
2014-02-27 02:40:49 -05:00
"com.clearspring.analytics" % "stream" % "2.5.1"
),
libraryDependencies ++= maybeAvro
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
)
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 17:41:07 -04:00
Adding code for publishing to Sonatype. By default - I'm leaving this commented out. This is because there is a bug in the PGP signing plugin which causes it to active even duing a publish-local. So we'll just uncomment when we decide to publish.
2012-10-05 23:59:31 -04:00
def rootSettings = sharedSettings ++ Seq(
publish := {}
)
porting of repl to scala-2.10
2013-03-17 01:17:17 -04:00
def replSettings = sharedSettings ++ Seq(
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 17:41:07 -04:00
name := "spark-repl",
Sync with master and some build fixes
2013-09-26 01:57:16 -04:00
libraryDependencies <+= scalaVersion(v => "org.scala-lang" % "scala-compiler" % v ),
libraryDependencies <+= scalaVersion(v => "org.scala-lang" % "jline" % v ),
libraryDependencies <+= scalaVersion(v => "org.scala-lang" % "scala-reflect" % v )
Fixed repl suite
2013-09-14 13:24:10 -04:00
)
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 17:41:07 -04:00
def examplesSettings = sharedSettings ++ Seq(
Add Maven build file for streaming, and fix some issues in SBT file As part of this, changed our Scala 2.9.2 Kafka library to be available as a local Maven repository, following the example in (http://blog.dub.podval.org/2010/01/maven-in-project-repository.html)
2013-01-20 22:22:24 -05:00
name := "spark-examples",
Add hBase example
2013-05-09 19:33:38 -04:00
libraryDependencies ++= Seq(
Merge branch 'master' into master-merge Conflicts: core/pom.xml core/src/main/scala/spark/MapOutputTracker.scala core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/RDDCheckpointData.scala core/src/main/scala/spark/SparkContext.scala core/src/main/scala/spark/Utils.scala core/src/main/scala/spark/api/python/PythonRDD.scala core/src/main/scala/spark/deploy/client/Client.scala core/src/main/scala/spark/deploy/master/MasterWebUI.scala core/src/main/scala/spark/deploy/worker/Worker.scala core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala core/src/main/scala/spark/rdd/BlockRDD.scala core/src/main/scala/spark/rdd/ZippedRDD.scala core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala core/src/main/scala/spark/storage/BlockManager.scala core/src/main/scala/spark/storage/BlockManagerMaster.scala core/src/main/scala/spark/storage/BlockManagerMasterActor.scala core/src/main/scala/spark/storage/BlockManagerUI.scala core/src/main/scala/spark/util/AkkaUtils.scala core/src/test/scala/spark/SizeEstimatorSuite.scala pom.xml project/SparkBuild.scala repl/src/main/scala/spark/repl/SparkILoop.scala repl/src/test/scala/spark/repl/ReplSuite.scala streaming/src/main/scala/spark/streaming/StreamingContext.scala streaming/src/main/scala/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/util/MasterFailureTest.scala
2013-07-03 02:13:26 -04:00
"com.twitter" %% "algebird-core" % "0.1.11",
SPARK-782 Clean up for ASM dependency. This makes two changes. 1) Spark uses the shaded version of asm that is (conveniently) published with Kryo. 2) Existing exclude rules around asm are updated to reflect the new groupId of `org.ow2.asm`. This made all of the old rules not work with newer Hadoop versions that pull in new asm versions. Author: Patrick Wendell <pwendell@gmail.com> Closes #100 from pwendell/asm and squashes the following commits: 9235f3f [Patrick Wendell] SPARK-782 Clean up for ASM dependency.
2014-03-09 16:17:07 -04:00
"org.apache.hbase" % "hbase" % HBASE_VERSION excludeAll(excludeNetty, excludeAsm, excludeOldAsm, excludeCommonsLogging),
Sync with master and some build fixes
2013-09-26 01:57:16 -04:00
"org.apache.cassandra" % "cassandra-all" % "1.2.6"
Merge remote-tracking branch 'milliondreams/casdemo' Conflicts: project/SparkBuild.scala
2013-06-18 08:55:31 -04:00
exclude("com.google.guava", "guava")
exclude("com.googlecode.concurrentlinkedhashmap", "concurrentlinkedhashmap-lru")
exclude("com.ning","compress-lzf")
exclude("io.netty", "netty")
exclude("jline","jline")
exclude("org.apache.cassandra.deps", "avro")
SPARK-1084.2 (resubmitted) (Ported from https://github.com/apache/incubator-spark/pull/650 ) This adds one more change though, to fix the scala version warning introduced by json4s recently. Author: Sean Owen <sowen@cloudera.com> Closes #32 from srowen/SPARK-1084.2 and squashes the following commits: 9240abd [Sean Owen] Avoid scala version conflict in scalap induced by json4s dependency 1561cec [Sean Owen] Remove "exclude *" dependencies that are causing Maven warnings, and that are apparently unneeded anyway
2014-03-02 17:27:53 -05:00
excludeAll(excludeSLF4J)
Add hBase example
2013-05-09 19:33:38 -04:00
)
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
) ++ assemblySettings ++ extraAssemblySettings
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 17:41:07 -04:00
Add JavaAPICompletenessChecker. This is used to find methods in the Scala API that need to be ported to the Java API. To use it: ./run spark.tools.JavaAPICompletenessChecker Conflicts: project/SparkBuild.scala run run2.cmd
2013-07-18 01:59:00 -04:00
def toolsSettings = sharedSettings ++ Seq(
name := "spark-tools"
Add spark-tools assembly to spark-class classpath. This allows the JavaAPICompletenessChecker to be run with Spark 0.8+.
2013-11-09 16:37:11 -05:00
) ++ assemblySettings ++ extraAssemblySettings
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 17:41:07 -04:00
graph -> graphx
2014-01-09 17:31:33 -05:00
def graphxSettings = sharedSettings ++ Seq(
Also add graphx commons-math3 dependeny in sbt build
2014-01-22 17:40:41 -05:00
name := "spark-graphx",
libraryDependencies ++= Seq(
Add jblas dependency
2014-01-23 06:54:01 -05:00
"org.jblas" % "jblas" % "1.2.3"
Also add graphx commons-math3 dependeny in sbt build
2014-01-22 17:40:41 -05:00
)
GraphX now builds with all merged changes.
2013-09-18 01:42:12 -04:00
)
Add bagel, mllib to SBT assembly. Also add jblas dependency to mllib pom.xml
2013-07-30 17:03:15 -04:00
def bagelSettings = sharedSettings ++ Seq(
name := "spark-bagel"
Add mllib, bagel to repl dependencies Also don't build an assembly jar for them
2013-07-30 21:31:11 -04:00
)
Fixed issues duplicate class issues in sbt assembly.
2012-06-22 18:03:09 -04:00
Renamed ML package to MLlib and added it to classpath
2013-07-05 14:38:53 -04:00
def mllibSettings = sharedSettings ++ Seq(
name := "spark-mllib",
Import RidgeRegression example Conflicts: run
2013-07-05 14:13:41 -04:00
libraryDependencies ++= Seq(
Removed dependency on Apache Commons Math
2013-06-21 09:58:01 -04:00
"org.jblas" % "jblas" % "1.2.3"
Import RidgeRegression example Conflicts: run
2013-07-05 14:13:41 -04:00
)
)
Added transformRDD DStream operation and TransformedDStream. Added sbt assembly option for streaming project.
2012-09-02 05:35:17 -04:00
def streamingSettings = sharedSettings ++ Seq(
Working on kafka.
2012-11-06 12:41:42 -05:00
name := "spark-streaming",
libraryDependencies ++= Seq(
Upgrade Kafka dependecy to 0.8.0 release version
2014-01-09 19:53:59 -05:00
"commons-io" % "commons-io" % "2.4"
Add Maven build file for streaming, and fix some issues in SBT file As part of this, changed our Scala 2.9.2 Kafka library to be available as a local Maven repository, following the example in (http://blog.dub.podval.org/2010/01/maven-in-project-repository.html)
2013-01-20 22:22:24 -05:00
)
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
)
Added the Spark Streaing code, ported to Akka 2
2012-07-28 23:03:26 -04:00
Use unmanaged source dir to include common yarn code
2013-12-23 00:32:20 -05:00
def yarnCommonSettings = sharedSettings ++ Seq(
unmanagedSourceDirectories in Compile <++= baseDirectory { base =>
Seq(
base / "../common/src/main/scala"
)
Add yarn/common/src/test dir in building script
2014-01-01 21:12:06 -05:00
},
unmanagedSourceDirectories in Test <++= baseDirectory { base =>
Seq(
base / "../common/src/test/scala"
)
Use unmanaged source dir to include common yarn code
2013-12-23 00:32:20 -05:00
}
Add yarn/common/src/test dir in building script
2014-01-01 21:12:06 -05:00
Update Maven build to create assemblies expected by new scripts This includes the following changes: - The "assembly" package now builds in Maven by default, and creates an assembly containing both hadoop-client and Spark, unlike the old BigTop distribution assembly that skipped hadoop-client - There is now a bigtop-dist package to build the old BigTop assembly - The repl-bin package is no longer built by default since the scripts don't reply on it; instead it can be enabled with -Prepl-bin - Py4J is now included in the assembly/lib folder as a local Maven repo, so that the Maven package can link to it - run-example now adds the original Spark classpath as well because the Maven examples assembly lists spark-core and such as provided - The various Maven projects add a spark-yarn dependency correctly
2013-08-27 22:23:54 -04:00
) ++ extraYarnSettings
Fix SBT generation of IDE project files
2013-08-23 13:26:37 -04:00
Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2
2014-01-01 22:23:33 -05:00
def yarnAlphaSettings = yarnCommonSettings ++ Seq(
name := "spark-yarn-alpha"
Reorganize yarn related codes into sub projects to remove duplicate files.
2013-12-04 00:20:27 -05:00
)
Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2
2014-01-01 22:23:33 -05:00
def yarnSettings = yarnCommonSettings ++ Seq(
name := "spark-yarn"
Reorganize yarn related codes into sub projects to remove duplicate files.
2013-12-04 00:20:27 -05:00
)
SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues... This patch removes Ganglia integration from the default build. It allows users willing to link against LGPL code to use Ganglia by adding build flags or linking against a new Spark artifact called spark-ganglia-lgpl. This brings Spark in line with the Apache policy on LGPL code enumerated here: https://www.apache.org/legal/3party.html#options-optional Author: Patrick Wendell <pwendell@gmail.com> Closes #108 from pwendell/ganglia and squashes the following commits: 326712a [Patrick Wendell] Responding to review feedback 5f28ee4 [Patrick Wendell] SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues.
2014-03-11 14:16:59 -04:00
def gangliaSettings = sharedSettings ++ Seq(
name := "spark-ganglia-lgpl",
libraryDependencies += "com.codahale.metrics" % "metrics-ganglia" % "3.0.0"
)
[java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs Author: Prashant Sharma <prashant.s@imaginea.com> Author: Patrick Wendell <pwendell@gmail.com> Closes #17 from ScrapCodes/java8-lambdas and squashes the following commits: 95850e6 [Patrick Wendell] Some doc improvements and build changes to the Java 8 patch. 85a954e [Prashant Sharma] Nit. import orderings. 673f7ac [Prashant Sharma] Added support for -java-home as well 80a13e8 [Prashant Sharma] Used fake class tag syntax 26eb3f6 [Prashant Sharma] Patrick's comments on PR. 35d8d79 [Prashant Sharma] Specified java 8 building in the docs 31d4cd6 [Prashant Sharma] Maven build to support -Pjava8-tests flag. 4ab87d3 [Prashant Sharma] Review feedback on the pr c33dc2c [Prashant Sharma] SPARK-964, Java 8 API Support.
2014-03-04 01:31:30 -05:00
def java8TestsSettings = sharedSettings ++ Seq(
name := "java8-tests",
javacOptions := Seq("-target", "1.8", "-source", "1.8"),
testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a")
)
Fix SBT generation of IDE project files
2013-08-23 13:26:37 -04:00
// Conditionally include the YARN dependencies because some tools look at all sub-projects and will complain
// if we refer to nonexistent dependencies (e.g. hadoop-yarn-api from a Hadoop version without YARN).
def extraYarnSettings = if(isYarnEnabled) yarnEnabledSettings else Seq()
def yarnEnabledSettings = Seq(
yarn sbt
2013-07-18 16:36:34 -04:00
libraryDependencies ++= Seq(
// Exclude rule required for all ?
SPARK-782 Clean up for ASM dependency. This makes two changes. 1) Spark uses the shaded version of asm that is (conveniently) published with Kryo. 2) Existing exclude rules around asm are updated to reflect the new groupId of `org.ow2.asm`. This made all of the old rules not work with newer Hadoop versions that pull in new asm versions. Author: Patrick Wendell <pwendell@gmail.com> Closes #100 from pwendell/asm and squashes the following commits: 9235f3f [Patrick Wendell] SPARK-782 Clean up for ASM dependency.
2014-03-09 16:17:07 -04:00
"org.apache.hadoop" % hadoopClient % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm),
"org.apache.hadoop" % "hadoop-yarn-api" % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm),
"org.apache.hadoop" % "hadoop-yarn-common" % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm),
"org.apache.hadoop" % "hadoop-yarn-client" % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm)
yarn sbt
2013-07-18 16:36:34 -04:00
)
Fix SBT generation of IDE project files
2013-08-23 13:26:37 -04:00
)
yarn sbt
2013-07-18 16:36:34 -04:00
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
def assemblyProjSettings = sharedSettings ++ Seq(
Deleted py4j jar and added to assembly dependency
2014-01-02 02:37:52 -05:00
libraryDependencies += "net.sf.py4j" % "py4j" % "0.8.1",
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
name := "spark-assembly",
Rename SBT target to assemble-deps.
2013-10-16 20:05:46 -04:00
assembleDeps in Compile <<= (packageProjects.map(packageBin in Compile in _) ++ Seq(packageDependency in Compile)).dependOn,
Add new SBT target for dependency assembly
2013-10-09 07:24:34 -04:00
jarName in assembly <<= version map { v => "spark-assembly-" + v + "-hadoop" + hadoopVersion + ".jar" },
jarName in packageDependency <<= version map { v => "spark-assembly-" + v + "-hadoop" + hadoopVersion + "-deps.jar" }
Added transformRDD DStream operation and TransformedDStream. Added sbt assembly option for streaming project.
2012-09-02 05:35:17 -04:00
) ++ assemblySettings ++ extraAssemblySettings
Added the Spark Streaing code, ported to Akka 2
2012-07-28 23:03:26 -04:00
Change build and run instructions to use assemblies This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
2013-08-24 02:30:17 -04:00
def extraAssemblySettings() = Seq(
test in assembly := {},
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 03:17:13 -04:00
mergeStrategy in assembly := {
case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
Don't use forward slash in exclusion for JAR signature files
2013-06-25 22:08:19 -04:00
case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard
Fixing spark streaming example and a bug in examples build. - Examples assembly included a log4j.properties which clobbered Spark's - Example had an error where some classes weren't serializable - Did some other clean-up in this example
2013-10-16 01:51:58 -04:00
case "log4j.properties" => MergeStrategy.discard
spark-assembly.jar fails to authenticate with YARN ResourceManager sbt-assembly is setup to pick the first META-INF/services/org.apache.hadoop.security.SecurityInfo file instead of merging them. This causes Kerberos authentication to fail, this manifests itself in the "info:null" debug log statement: DEBUG SaslRpcClient: Get token info proto:interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB info:null DEBUG SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB info:null ERROR UserGroupInformation: PriviledgedActionException as:foo@BAR (auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] DEBUG UserGroupInformation: PrivilegedAction as:foo@BAR (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:583) WARN Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] ERROR UserGroupInformation: PriviledgedActionException as:foo@BAR (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] This previously would just contain a single class: $ unzip -c assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.2.0.jar META-INF/services/org.apache.hadoop.security.SecurityInfo Archive: assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.2.0.jar inflating: META-INF/services/org.apache.hadoop.security.SecurityInfo org.apache.hadoop.security.AnnotatedSecurityInfo And now has the full list of classes: $ unzip -c assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.2.0.jar META-INF/services/org.apache.hadoop.security.SecurityInfoArchive: assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.2.0.jar inflating: META-INF/services/org.apache.hadoop.security.SecurityInfo org.apache.hadoop.security.AnnotatedSecurityInfo org.apache.hadoop.mapreduce.v2.app.MRClientSecurityInfo org.apache.hadoop.mapreduce.v2.security.client.ClientHSSecurityInfo org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo org.apache.hadoop.yarn.security.ContainerManagerSecurityInfo org.apache.hadoop.yarn.security.SchedulerSecurityInfo org.apache.hadoop.yarn.security.admin.AdminSecurityInfo org.apache.hadoop.yarn.server.RMNMSecurityInfoClass
2013-11-12 16:17:48 -05:00
case m if m.toLowerCase.startsWith("meta-inf/services/") => MergeStrategy.filterDistinctLines
Use sbt mergeStrategy for reference.conf files. Cleans up #158 / 509b721.
2012-08-02 13:12:00 -04:00
case "reference.conf" => MergeStrategy.concat
Fixed issues duplicate class issues in sbt assembly.
2012-06-22 18:03:09 -04:00
case _ => MergeStrategy.first
}
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 03:17:13 -04:00
)
Refactored streaming project to separate out the twitter functionality.
2013-12-26 21:02:49 -05:00
Added pom.xml for external projects and removed unnecessary dependencies and repositoris from other poms and sbt.
2013-12-31 03:28:57 -05:00
def twitterSettings() = sharedSettings ++ Seq(
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
name := "spark-streaming-twitter",
Refactored streaming project to separate out the twitter functionality.
2013-12-26 21:02:49 -05:00
libraryDependencies ++= Seq(
"org.twitter4j" % "twitter4j-stream" % "3.0.3" excludeAll(excludeNetty)
)
)
Upgrade Kafka dependecy to 0.8.0 release version
2014-01-09 19:53:59 -05:00
Added pom.xml for external projects and removed unnecessary dependencies and repositoris from other poms and sbt.
2013-12-31 03:28:57 -05:00
def kafkaSettings() = sharedSettings ++ Seq(
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
name := "spark-streaming-kafka",
libraryDependencies ++= Seq(
"com.github.sgroschupf" % "zkclient" % "0.1" excludeAll(excludeNetty),
Upgrade Kafka dependecy to 0.8.0 release version
2014-01-09 19:53:59 -05:00
"org.apache.kafka" %% "kafka" % "0.8.0"
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
exclude("com.sun.jdmk", "jmxtools")
exclude("com.sun.jmx", "jmxri")
exclude("net.sf.jopt-simple", "jopt-simple")
SPARK-1071: Tidy logging strategy and use of log4j Prompted by a recent thread on the mailing list, I tried and failed to see if Spark can be made independent of log4j. There are a few cases where control of the underlying logging is pretty useful, and to do that, you have to bind to a specific logger. Instead I propose some tidying that leaves Spark's use of log4j, but gets rid of warnings and should still enable downstream users to switch. The idea is to pipe everything (except log4j) through SLF4J, and have Spark use SLF4J directly when logging, and where Spark needs to output info (REPL and tests), bind from SLF4J to log4j. This leaves the same behavior in Spark. It means that downstream users who want to use something except log4j should: - Exclude dependencies on log4j, slf4j-log4j12 from Spark - Include dependency on log4j-over-slf4j - Include dependency on another logger X, and another slf4j-X - Recreate any log config that Spark does, that is needed, in the other logger's config That sounds about right. Here are the key changes: - Include the jcl-over-slf4j shim everywhere by depending on it in core. - Exclude dependencies on commons-logging from third-party libraries. - Include the jul-to-slf4j shim everywhere by depending on it in core. - Exclude slf4j-* dependencies from third-party libraries to prevent collision or warnings - Added missing slf4j-log4j12 binding to GraphX, Bagel module tests And minor/incidental changes: - Update to SLF4J 1.7.5, which happily matches Hadoop 2’s version and is a recommended update over 1.7.2 - (Remove a duplicate HBase dependency declaration in SparkBuild.scala) - (Remove a duplicate mockito dependency declaration that was causing warnings and bugging me) Author: Sean Owen <sowen@cloudera.com> Closes #570 from srowen/SPARK-1071 and squashes the following commits: 52eac9f [Sean Owen] Add slf4j-over-log4j12 dependency to core (non-test) and remove it from things that depend on core. 77a7fa9 [Sean Owen] SPARK-1071: Tidy logging strategy and use of log4j
2014-02-23 14:40:55 -05:00
excludeAll(excludeNetty, excludeSLF4J)
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
)
)
Upgrade Kafka dependecy to 0.8.0 release version
2014-01-09 19:53:59 -05:00
Added pom.xml for external projects and removed unnecessary dependencies and repositoris from other poms and sbt.
2013-12-31 03:28:57 -05:00
def flumeSettings() = sharedSettings ++ Seq(
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
name := "spark-streaming-flume",
libraryDependencies ++= Seq(
SPARK-1084.2 (resubmitted) (Ported from https://github.com/apache/incubator-spark/pull/650 ) This adds one more change though, to fix the scala version warning introduced by json4s recently. Author: Sean Owen <sowen@cloudera.com> Closes #32 from srowen/SPARK-1084.2 and squashes the following commits: 9240abd [Sean Owen] Avoid scala version conflict in scalap induced by json4s dependency 1561cec [Sean Owen] Remove "exclude *" dependencies that are causing Maven warnings, and that are apparently unneeded anyway
2014-03-02 17:27:53 -05:00
"org.apache.flume" % "flume-ng-sdk" % "1.2.0" % "compile" excludeAll(excludeNetty)
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
)
)
Added pom.xml for external projects and removed unnecessary dependencies and repositoris from other poms and sbt.
2013-12-31 03:28:57 -05:00
def zeromqSettings() = sharedSettings ++ Seq(
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
name := "spark-streaming-zeromq",
libraryDependencies ++= Seq(
"org.spark-project.akka" %% "akka-zeromq" % "2.2.3-shaded-protobuf" excludeAll(excludeNetty)
)
)
def mqttSettings() = streamingSettings ++ Seq(
name := "spark-streaming-mqtt",
Added pom.xml for external projects and removed unnecessary dependencies and repositoris from other poms and sbt.
2013-12-31 03:28:57 -05:00
resolvers ++= Seq("Eclipse Repo" at "https://repo.eclipse.org/content/repositories/paho-releases/"),
libraryDependencies ++= Seq("org.eclipse.paho" % "mqtt-client" % "0.4.0")
Refactored kafka, flume, zeromq, mqtt as separate external projects, with their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
2013-12-30 14:13:24 -05:00
)
Initial work on converting build to SBT 0.10.1
2011-07-14 22:38:25 -04:00
}
Reference in a new issue Copy permalink