ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3252 commits 6 branches 181 tags 397 MiB
Commit graph

3252 commits

This branch
This branch
All branches
Author SHA1 Message Date
Evan Chan efd6418c1b Move getPersistentRDDs testing to a new Suite 2013-07-23 10:40:41 -07:00
Evan Chan 4830e22562 Rename method per rxin feedback 2013-07-23 09:50:13 -07:00
Evan Chan 2c2bfbe294 Add toMap method to TimeStampedHashMap and use it 2013-07-23 01:36:44 -07:00
Evan Chan 0337d88321 Add a public method getCachedRdds to SparkContext 2013-07-21 18:26:14 -07:00
Matei Zaharia c40f0f21f1 Merge pull request #711 from shivaram/ml-generators 
Move ML lib data generator files to util/
 2013-07-19 13:33:04 -07:00
Matei Zaharia 413b84172e Merge pull request #717 from viirya/dev1 
Do not copy local jars given to SparkContext in yarn mode
 2013-07-19 13:31:38 -07:00
Liang-Chi Hsieh d1738d72ba also exclude asm for hadoop2. hadoop1 looks like no need to do that too. 2013-07-20 00:37:24 +08:00
Liang-Chi Hsieh 4530e8a9bf fix typo. 2013-07-20 00:04:25 +08:00
Liang-Chi Hsieh aa6f83289b A better fix for giving local jars unde Yarn mode. 2013-07-19 22:25:28 +08:00
Liang-Chi Hsieh a613628c50 Do not copy local jars given to SparkContext in yarn mode since the Context is not running on local. This bug causes failure when jars can not be found. Example codes (such as spark.examples.SparkPi) can not work without this fix under yarn mode. 2013-07-19 16:59:12 +08:00
Matei Zaharia 0d0a47cb4f Merge pull request #710 from shivaram/ml-updates 
Updates to LogisticRegression
 2013-07-18 12:06:37 -07:00
Shivaram Venkataraman 2c9ea56db4 Rename classes to be called DataGenerator 2013-07-18 11:57:14 -07:00
Shivaram Venkataraman 7ab1170503 Refactor data generators to have a function that can be used in tests. 2013-07-18 11:55:19 -07:00
Matei Zaharia c6235b51ba Merge pull request #714 from adatao/master 
[BUGFIX]  Fix for sbt/sbt script SPARK_HOME setting
 2013-07-18 11:43:48 -07:00
Matei Zaharia 009c79e5a5 Merge pull request #715 from viirya/dev1 
fix a bug in build process that pulls in two versions of ASM.
 2013-07-18 11:41:52 -07:00
Liang-Chi Hsieh 3aad452653 fix a bug in build process that pulls in two versionf of ASM. 2013-07-19 02:29:46 +08:00
Matei Zaharia 985a9e324a Merge pull request #712 from stayhf/SPARK-817 
Consistently invoke bash with /usr/bin/env bash in scripts to make code ...
 2013-07-17 22:27:19 -07:00
Ubuntu 88a0823c58 Consistently invoke bash with /usr/bin/env bash in scripts to make code more portable (JIRA Ticket SPARK-817) 2013-07-18 00:51:18 +00:00
Shivaram Venkataraman 217667174e Return Array[Double] from SGD instead of DoubleMatrix 2013-07-17 16:08:34 -07:00
Shivaram Venkataraman 45f3c85518 Change weights to be Array[Double] in LR model. 
Also ensure weights are initialized to a column vector.
 2013-07-17 16:03:29 -07:00
Shivaram Venkataraman 3bf9897136 Rename loss -> stochasticLoss and add a note to explain why we have 
multiple train methods.
 2013-07-17 14:20:24 -07:00
Shivaram Venkataraman 64b88e039a Move ML lib data generator files to util/ 2013-07-17 14:11:44 -07:00
Shivaram Venkataraman 84fa20c2a1 Allow initial weight vectors in LogisticRegression. 
Also move LogisticGradient to the LogisticRegression file and fix the
unit tests log path.
 2013-07-17 14:04:05 -07:00
ctn a1d2c34346 [BUGFIX] Fix for sbt/sbt script SPARK_HOME setting 
In some environments, this command

    export SPARK_HOME=$(cd "$(dirname $0)/.."; pwd)

echoes two paths, one by the "cd ..", and one by the "pwd". Note the resulting
erroneous -jar paths below:

    ctn@ubuntu:~/src/spark$ sbt/sbt
    + EXTRA_ARGS=
    + '[' '' '!=' '' ']'
    +++ dirname sbt/sbt
    ++ cd sbt/..
    ++ pwd
    + export 'SPARK_HOME=/home/ctn/src/spark
    /home/ctn/src/spark'
    + SPARK_HOME='/home/ctn/src/spark
    /home/ctn/src/spark'
    + export SPARK_TESTING=1
    + SPARK_TESTING=1
    + java -Xmx1200m -XX:MaxPermSize=350m -XX:ReservedCodeCacheSize=128m -jar /home/ctn/src/spark /home/ctn/src/spark/sbt/sbt-launch-0.11.3-2.jar
    Error: Invalid or corrupt jarfile /home/ctn/src/spark

Committer: ctn <ctn@adatao.com>

On branch master
Changes to be committed:

- Send output of the "cd .." part to /dev/null
	modified:   sbt/sbt
 2013-07-17 11:27:11 -07:00
ctn b29c53e6aa Merge remote-tracking branch 'upstream/master' 2013-07-17 11:24:03 -07:00
Matei Zaharia cad48edb70 Merge pull request #708 from ScrapCodes/dependencies-upgrade 
Dependency upgrade Akka 2.0.3 -> 2.0.5
 2013-07-16 21:41:28 -07:00
Matei Zaharia 7e74ab5b0f Merge remote-tracking branch 'origin/pr/704' 
Conflicts:
	make-distribution.sh
 2013-07-16 19:13:07 -07:00
Matei Zaharia 4ff494de20 Some missing license headers 2013-07-16 17:26:48 -07:00
Matei Zaharia af3c9d5042 Add Apache license headers and LICENSE and NOTICE files 2013-07-16 17:21:33 -07:00
Matei Zaharia 00a14deb6d Update to latest Scala Maven plugin and allow Zinc external compiler 2013-07-16 11:52:20 -07:00
Matei Zaharia b1f9f64743 Merge branch 'master' of github.com:mesos/spark 2013-07-16 11:01:53 -07:00
Matei Zaharia 5c388808a8 SPARK-814: Result stages should be named after action 2013-07-16 11:01:14 -07:00
Matei Zaharia f347cc3f65 Fix deprecation warning and style issues 2013-07-16 10:53:30 -07:00
Prashant Sharma 2748e73eb9 Dependency upgrade Akka 2.0.3 -> 2.0.5 2013-07-16 16:08:46 +05:30
Matei Zaharia 8a8a8f2de2 Merge pull request #705 from rxin/errormessages 
Throw a more meaningful message when runJob is called to launch tasks on non-existent partitions.
 2013-07-15 23:09:21 -07:00
Reynold Xin 69316603d6 Throw a more meaningful message when runJob is called to launch tasks on non-existent partitions. 2013-07-15 22:50:11 -07:00
seanm 90b0142985 adding files generated from make-distribution.sh to .gitignore 2013-07-15 19:13:39 -06:00
seanm a96b4ef761 dding tgz option to make-distribution.sh 2013-07-15 19:13:17 -06:00
Matei Zaharia ed8415b2fa Merge pull request #703 from karenfeng/ui-802 
Link to job UI from standalone deploy cluster web UI
 2013-07-15 16:41:04 -07:00
Karen Feng 6dc7c9bfb1 Removed job UI column, linked description to job UI 2013-07-15 16:33:50 -07:00
Karen Feng fbf5aa761e Removed log message, added field in master UI to link to log UI 2013-07-15 15:50:03 -07:00
Karen Feng eac381a957 Merge branch 'ui-802' of https://github.com/karenfeng/spark into ui-802 2013-07-15 15:48:44 -07:00
Karen Feng 3955711250 Added field to master UI with link to job UI 2013-07-15 15:47:21 -07:00
Matei Zaharia e3d3e6f0ab Merge pull request #702 from karenfeng/ui-fixes 
Adds app name in HTML page titles on job web UI
 2013-07-15 14:59:44 -07:00
Karen Feng 0d78b6d9cd Links to job UI from standalone deploy cluster web UI: fixes SPARK-802 2013-07-15 13:47:38 -07:00
Karen Feng b2aaa1199e Adds app name in HTML page titles on job web UI: fixes SPARK-806 2013-07-15 11:44:42 -07:00
Matei Zaharia 4698a0d688 Shuffle ratings in a more efficient way at start of ALS 2013-07-15 02:54:11 +00:00
Matei Zaharia d47c16f78d Add an option to disable reference tracking in Kryo 2013-07-15 01:55:54 +00:00
Matei Zaharia 238d0e6893 Merge branch 'master' of github.com:mesos/spark 2013-07-15 00:30:32 +00:00
Matei Zaharia ed7fd501cf Make number of blocks in ALS configurable and lower the default 2013-07-15 00:30:10 +00:00