ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3851 commits 6 branches 181 tags 397 MiB
Commit graph

256 commits

Author SHA1 Message Date
Reynold Xin 9db1e50344 Revert "Merge pull request #841 from rxin/json" 
This reverts commit 1fb1b09928, reversing
changes made to c69c48947d.
 2013-08-26 11:05:14 -07:00
Jey Kottalam a9db1b7b6e Upgrade SBT IDE project generators 2013-08-23 10:27:18 -07:00
Jey Kottalam b7f9e6374a Fix SBT generation of IDE project files 2013-08-23 10:26:37 -07:00
Jey Kottalam 281b6c5f28 Re-add removed dependency on 'commons-daemon' 
Fixes SBT build under Hadoop 0.23.9 and 2.0.4
 2013-08-22 15:45:45 -07:00
Matei Zaharia ae8ba83ef2 Merge pull request #855 from jey/update-build-docs 
Update build docs
 2013-08-22 10:14:54 -07:00
Matei Zaharia 8a36fd09dd Merge pull request #854 from markhamstra/pomUpdate 
Synced sbt and maven builds to use the same dependencies, etc.
 2013-08-22 10:13:35 -07:00
Jey Kottalam f9cc1fbf27 Remove references to unsupported Hadoop versions 2013-08-21 17:14:36 -07:00
Mark Hamstra ff6f1b0500 Synced sbt and maven builds 2013-08-21 13:50:24 -07:00
Reynold Xin af602ba9d3 Downgraded default build hadoop version to 1.0.4. 2013-08-21 11:38:24 -07:00
Matei Zaharia aa2b89d98d Merge remote-tracking branch 'jey/hadoop-agnostic' 
Conflicts:
	core/src/main/scala/spark/PairRDDFunctions.scala
 2013-08-20 10:14:15 -07:00
Jey Kottalam 6f6944c807 Update SBT build to use simpler fix for Hadoop 0.23.9 2013-08-19 12:33:13 -07:00
Jey Kottalam 67b593607c Rename YARN build flag to SPARK_WITH_YARN 2013-08-16 14:00:05 -07:00
Jey Kottalam b1d99744a8 Fix SBT build under Hadoop 0.23.x 2013-08-16 13:50:12 -07:00
Jey Kottalam 8add2d7a59 Fix repl/assembly when YARN enabled 2013-08-16 13:50:12 -07:00
Jey Kottalam 3f98eff63a Allow make-distribution.sh to specify Hadoop version used 2013-08-16 13:50:09 -07:00
Reynold Xin c961c19b7b Use the JSON formatter from Scala library and removed dependency on lift-json. 
It made the JSON creation slightly more complicated, but reduces one external dependency. The scala library also properly escape "/" (which lift-json doesn't).
 2013-08-15 18:23:01 -07:00
Jey Kottalam a0f0848463 Update default version of Hadoop to 1.2.1 2013-08-15 16:50:37 -07:00
Jey Kottalam cb4ef19214 yarn support 2013-08-15 16:50:37 -07:00
Jey Kottalam 273b499b9a yarn sbt 2013-08-15 16:50:37 -07:00
Jey Kottalam 69c3bbf688 dynamically detect hadoop version 2013-08-15 16:50:37 -07:00
Matei Zaharia d9588183fa Update to Mesos 0.12.1 2013-08-13 18:51:35 -07:00
jerryshao 320e87e7ab Add MetricsServlet for Spark metrics system 2013-08-12 13:23:23 +08:00
Matei Zaharia dce5e47435 Merge pull request #800 from dlyubimov/HBASE_VERSION 
Pull HBASE_VERSION in the head of sbt build
 2013-08-09 21:53:45 -07:00
Matei Zaharia cd247ba5bb Merge pull request #786 from shivaram/mllib-java 
Java fixes, tests and examples for ALS, KMeans
 2013-08-09 20:41:13 -07:00
Dmitriy Lyubimov 27f674f82b fewer words 2013-08-09 13:54:41 -07:00
Dmitriy Lyubimov ae95b57469 Pull HBASE_VERSION in the head of sbt build 2013-08-09 12:45:18 -07:00
Matei Zaharia 5a4003c1ac Update to Chill 0.3.1 2013-08-08 13:30:27 -07:00
Shivaram Venkataraman 471fbadd0c Java examples, tests for KMeans and ALS 
- Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it
  easier to call from Java
- Renames class methods from `train` to `run` to enable static methods to be
  called from Java.
- Add unit tests which check if both static / class methods can be called.
- Also add examples which port the main() function in ALS, KMeans to the
  examples project.

Couple of minor changes to existing code:
- Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily
- Workaround a bug where using double[] from Java leads to class cast exception in
  KMeans init
 2013-08-06 15:43:46 -07:00
Matei Zaharia e466a55a6b Revert Mesos version to 0.9 since the 0.12 artifact has target Java 7 2013-08-01 15:45:21 -07:00
Matei Zaharia b2b86c2575 Merge pull request #753 from shivaram/glm-refactor 
Build changes for ML lib
 2013-07-31 15:51:39 -07:00
Matei Zaharia 14bf2fe039 Merge pull request #749 from benh/spark-executor-uri 
Added property 'spark.executor.uri' for launching on Mesos.
 2013-07-31 14:18:16 -07:00
Shivaram Venkataraman 15fd0d619d Add mllib, bagel to repl dependencies 
Also don't build an assembly jar for them
 2013-07-30 18:31:11 -07:00
Reynold Xin 3b1ced83fb Exclude older version of Snappy in streaming and examples. 2013-07-30 17:25:36 -07:00
Reynold Xin 368c58eac5 Merge branch 'lazy_file_open' of github.com:lyogavin/spark into compression 
Conflicts:
	project/SparkBuild.scala
 2013-07-30 16:04:18 -07:00
Shivaram Venkataraman 48851d4dd9 Add bagel, mllib to SBT assembly. 
Also add jblas dependency to mllib pom.xml
 2013-07-30 14:03:15 -07:00
Benjamin Hindman f6f46455eb Added property 'spark.executor.uri' for launching on Mesos without 
requiring Spark to be installed. Using 'make_distribution.sh' a user
can put a Spark distribution at a URI supported by Mesos (e.g.,
'hdfs://...') and then set that when launching their job. Also added
SPARK_EXECUTOR_URI for the REPL.
 2013-07-29 23:32:52 -07:00
ryanlecompte 8e0939f5a9 refactor Kryo serializer support to use chill/chill-java 2013-07-24 20:43:57 -07:00
jerryshao 5730193e0c Fix some typos 2013-07-24 14:57:47 +08:00
jerryshao 576528f0f9 Add dependency of Codahale's metrics library 2013-07-24 14:57:46 +08:00
Josh Rosen c83680434b Add JavaAPICompletenessChecker. 
This is used to find methods in the Scala API that
need to be ported to the Java API.  To use it:

  ./run spark.tools.JavaAPICompletenessChecker
Conflicts:
	project/SparkBuild.scala
	run
	run2.cmd
 2013-07-22 16:11:49 -07:00
Liang-Chi Hsieh d1738d72ba also exclude asm for hadoop2. hadoop1 looks like no need to do that too. 2013-07-20 00:37:24 +08:00
Liang-Chi Hsieh 3aad452653 fix a bug in build process that pulls in two versionf of ASM. 2013-07-19 02:29:46 +08:00
Matei Zaharia cad48edb70 Merge pull request #708 from ScrapCodes/dependencies-upgrade 
Dependency upgrade Akka 2.0.3 -> 2.0.5
 2013-07-16 21:41:28 -07:00
Matei Zaharia af3c9d5042 Add Apache license headers and LICENSE and NOTICE files 2013-07-16 17:21:33 -07:00
Prashant Sharma 2748e73eb9 Dependency upgrade Akka 2.0.3 -> 2.0.5 2013-07-16 16:08:46 +05:30
Matei Zaharia 668b0dc6a7 Merge branch 'master' of github.com:mesos/spark 2013-07-13 19:10:46 -07:00
Matei Zaharia cd28d9c147 Merge remote-tracking branch 'origin/pr/662' 
Conflicts:
	bin/compute-classpath.sh
 2013-07-13 19:10:00 -07:00
seanm c4d5b01e44 changing com.google.code.findbugs maven coordinates 2013-07-13 14:56:23 -06:00
Matei Zaharia 3cc6818f13 Merge pull request #668 from shimingfei/guava-14.0.1 
update guava version from 11.0.1 to 14.0.1
 2013-07-06 19:51:20 -07:00
Matei Zaharia 1ffadb2d9e Merge remote-tracking branch 'pwendell/ui-updates' 
Conflicts:
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	pom.xml
 2013-07-06 15:51:41 -07:00