ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3465 commits 6 branches 181 tags 397 MiB
Commit graph

3465 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 8e38e77232 Fix a test that was using an outdated config setting 2013-07-22 16:05:32 -07:00
Matei Zaharia 8ae1436981 Merge pull request #722 from JoshRosen/spark-825 
Fix bug: DoubleRDDFunctions.sampleStdev() computed non-sample stdev()
 2013-07-22 16:03:04 -07:00
Karen Feng 2eea974795 Executors UI now calls executor ID from TaskInfo instead of TaskMetrics 2013-07-22 15:15:54 -07:00
Dmitriy Lyubimov 8ca0c31944 removing non-pertinent comment 2013-07-22 14:48:46 -07:00
Dmitriy Lyubimov b4b230e606 Fixing for LocalScheduler with test, that much works .. 2013-07-22 14:42:47 -07:00
Karen Feng 85c4d7bf3b Shows number of complete/total/failed tasks (bug: failed tasks assigned to null executor) 2013-07-22 14:35:47 -07:00
Josh Rosen f649dabb4a Fix bug: DoubleRDDFunctions.sampleStdev() computed non-sample stdev(). 
Update JavaDoubleRDD to add new methods and docs.

Fixes SPARK-825.
 2013-07-22 13:21:48 -07:00
Karen Feng 8901f379c9 Fixed memory used/remaining/total bug 2013-07-22 09:58:03 -07:00
Karen Feng 636b19f833 Merge branch 'master' of https://github.com/mesos/spark into ui-808 2013-07-22 09:53:26 -07:00
Evan Chan 0337d88321 Add a public method getCachedRdds to SparkContext 2013-07-21 18:26:14 -07:00
Konstantin Boudnik f4d514810e Building spark assembly for further consumption of the Spark project with a deployed cluster 2013-07-21 11:47:29 -07:00
Matei Zaharia 15fb394833 Merge pull request #716 from c0s/webui-port 
Regression: default webui-port can't be set via command line "--webui-port" anymore
 2013-07-21 10:33:38 -07:00
Karen Feng 865dc63bac Changed table format for executors 2013-07-19 15:57:01 -07:00
Karen Feng 81bb5dc640 Creates Executors tab for application with RDD block and memory/disk used, solves SPARK-808 2013-07-19 14:08:30 -07:00
Konstantin Boudnik cfce9a6a36 Regression: default webui-port can't be set via command line "--webui-port" anymore 2013-07-19 14:00:58 -07:00
Matei Zaharia c40f0f21f1 Merge pull request #711 from shivaram/ml-generators 
Move ML lib data generator files to util/
 2013-07-19 13:33:04 -07:00
Matei Zaharia 413b84172e Merge pull request #717 from viirya/dev1 
Do not copy local jars given to SparkContext in yarn mode
 2013-07-19 13:31:38 -07:00
Liang-Chi Hsieh d1738d72ba also exclude asm for hadoop2. hadoop1 looks like no need to do that too. 2013-07-20 00:37:24 +08:00
Liang-Chi Hsieh 4530e8a9bf fix typo. 2013-07-20 00:04:25 +08:00
Liang-Chi Hsieh aa6f83289b A better fix for giving local jars unde Yarn mode. 2013-07-19 22:25:28 +08:00
Liang-Chi Hsieh a613628c50 Do not copy local jars given to SparkContext in yarn mode since the Context is not running on local. This bug causes failure when jars can not be found. Example codes (such as spark.examples.SparkPi) can not work without this fix under yarn mode. 2013-07-19 16:59:12 +08:00
Matei Zaharia 0d0a47cb4f Merge pull request #710 from shivaram/ml-updates 
Updates to LogisticRegression
 2013-07-18 12:06:37 -07:00
Shivaram Venkataraman 2c9ea56db4 Rename classes to be called DataGenerator 2013-07-18 11:57:14 -07:00
Shivaram Venkataraman 7ab1170503 Refactor data generators to have a function that can be used in tests. 2013-07-18 11:55:19 -07:00
Matei Zaharia c6235b51ba Merge pull request #714 from adatao/master 
[BUGFIX]  Fix for sbt/sbt script SPARK_HOME setting
 2013-07-18 11:43:48 -07:00
Matei Zaharia 009c79e5a5 Merge pull request #715 from viirya/dev1 
fix a bug in build process that pulls in two versions of ASM.
 2013-07-18 11:41:52 -07:00
Liang-Chi Hsieh 3aad452653 fix a bug in build process that pulls in two versionf of ASM. 2013-07-19 02:29:46 +08:00
Matei Zaharia 985a9e324a Merge pull request #712 from stayhf/SPARK-817 
Consistently invoke bash with /usr/bin/env bash in scripts to make code ...
 2013-07-17 22:27:19 -07:00
Ubuntu 88a0823c58 Consistently invoke bash with /usr/bin/env bash in scripts to make code more portable (JIRA Ticket SPARK-817) 2013-07-18 00:51:18 +00:00
Shivaram Venkataraman 217667174e Return Array[Double] from SGD instead of DoubleMatrix 2013-07-17 16:08:34 -07:00
Shivaram Venkataraman 45f3c85518 Change weights to be Array[Double] in LR model. 
Also ensure weights are initialized to a column vector.
 2013-07-17 16:03:29 -07:00
Shivaram Venkataraman 3bf9897136 Rename loss -> stochasticLoss and add a note to explain why we have 
multiple train methods.
 2013-07-17 14:20:24 -07:00
Shivaram Venkataraman 64b88e039a Move ML lib data generator files to util/ 2013-07-17 14:11:44 -07:00
Shivaram Venkataraman 84fa20c2a1 Allow initial weight vectors in LogisticRegression. 
Also move LogisticGradient to the LogisticRegression file and fix the
unit tests log path.
 2013-07-17 14:04:05 -07:00
ctn a1d2c34346 [BUGFIX] Fix for sbt/sbt script SPARK_HOME setting 
In some environments, this command

    export SPARK_HOME=$(cd "$(dirname $0)/.."; pwd)

echoes two paths, one by the "cd ..", and one by the "pwd". Note the resulting
erroneous -jar paths below:

    ctn@ubuntu:~/src/spark$ sbt/sbt
    + EXTRA_ARGS=
    + '[' '' '!=' '' ']'
    +++ dirname sbt/sbt
    ++ cd sbt/..
    ++ pwd
    + export 'SPARK_HOME=/home/ctn/src/spark
    /home/ctn/src/spark'
    + SPARK_HOME='/home/ctn/src/spark
    /home/ctn/src/spark'
    + export SPARK_TESTING=1
    + SPARK_TESTING=1
    + java -Xmx1200m -XX:MaxPermSize=350m -XX:ReservedCodeCacheSize=128m -jar /home/ctn/src/spark /home/ctn/src/spark/sbt/sbt-launch-0.11.3-2.jar
    Error: Invalid or corrupt jarfile /home/ctn/src/spark

Committer: ctn <ctn@adatao.com>

On branch master
Changes to be committed:

- Send output of the "cd .." part to /dev/null
	modified:   sbt/sbt
 2013-07-17 11:27:11 -07:00
ctn b29c53e6aa Merge remote-tracking branch 'upstream/master' 2013-07-17 11:24:03 -07:00
Matei Zaharia cad48edb70 Merge pull request #708 from ScrapCodes/dependencies-upgrade 
Dependency upgrade Akka 2.0.3 -> 2.0.5
 2013-07-16 21:41:28 -07:00
Matei Zaharia 7e74ab5b0f Merge remote-tracking branch 'origin/pr/704' 
Conflicts:
	make-distribution.sh
 2013-07-16 19:13:07 -07:00
Matei Zaharia 4ff494de20 Some missing license headers 2013-07-16 17:26:48 -07:00
Matei Zaharia af3c9d5042 Add Apache license headers and LICENSE and NOTICE files 2013-07-16 17:21:33 -07:00
Matei Zaharia 00a14deb6d Update to latest Scala Maven plugin and allow Zinc external compiler 2013-07-16 11:52:20 -07:00
Matei Zaharia b1f9f64743 Merge branch 'master' of github.com:mesos/spark 2013-07-16 11:01:53 -07:00
Matei Zaharia 5c388808a8 SPARK-814: Result stages should be named after action 2013-07-16 11:01:14 -07:00
Matei Zaharia f347cc3f65 Fix deprecation warning and style issues 2013-07-16 10:53:30 -07:00
Prashant Sharma 2748e73eb9 Dependency upgrade Akka 2.0.3 -> 2.0.5 2013-07-16 16:08:46 +05:30
Matei Zaharia 8a8a8f2de2 Merge pull request #705 from rxin/errormessages 
Throw a more meaningful message when runJob is called to launch tasks on non-existent partitions.
 2013-07-15 23:09:21 -07:00
Reynold Xin 69316603d6 Throw a more meaningful message when runJob is called to launch tasks on non-existent partitions. 2013-07-15 22:50:11 -07:00
seanm 90b0142985 adding files generated from make-distribution.sh to .gitignore 2013-07-15 19:13:39 -06:00
seanm a96b4ef761 dding tgz option to make-distribution.sh 2013-07-15 19:13:17 -06:00
Matei Zaharia ed8415b2fa Merge pull request #703 from karenfeng/ui-802 
Link to job UI from standalone deploy cluster web UI
 2013-07-15 16:41:04 -07:00