ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3563 commits 6 branches 181 tags 397 MiB
Commit graph

3563 commits

This branch
This branch
All branches
Author SHA1 Message Date
Konstantin Boudnik a6f43a97d1 SPARK-842. Maven assembly is including examples libs and dependencies 2013-07-31 17:26:55 -07:00
Karen Feng ef1f22bda9 Merge branch 'master' of https://github.com/mesos/spark 2013-07-31 16:13:49 -07:00
Matei Zaharia 39c75f3033 Merge pull request #757 from BlackNiuza/result_task_generation 
Bug fix: SPARK-837
 2013-07-31 15:52:36 -07:00
Matei Zaharia b2b86c2575 Merge pull request #753 from shivaram/glm-refactor 
Build changes for ML lib
 2013-07-31 15:51:39 -07:00
Matei Zaharia f607ffb9e1 Added data generator for K-means 
Also made it possible to specify the number of runs in KMeans.main().
 2013-07-31 14:31:07 -07:00
Matei Zaharia 14bf2fe039 Merge pull request #749 from benh/spark-executor-uri 
Added property 'spark.executor.uri' for launching on Mesos.
 2013-07-31 14:18:16 -07:00
shivaram 4ba4c3fe1a Merge pull request #759 from mateiz/split-fix 
Use the Char version of split() instead of the String one in MLUtils
 2013-07-31 13:14:49 -07:00
Benjamin Hindman 529ac81195 Do not try and use 'scala' in 'run' from within a "release". 2013-07-31 12:50:12 -07:00
Benjamin Hindman 4692ea4892 Used 'uri.split('/').last' instead of 'new File(uri).getName()'. 2013-07-31 12:29:44 -07:00
Karen Feng c453967f9a Reduced size of heading 2013-07-31 11:57:50 -07:00
Matei Zaharia 9a444cffe7 Use the Char version of split() instead of the String one for efficiency 2013-07-31 11:28:39 -07:00
Matei Zaharia a386ced2c6 Merge pull request #754 from rxin/compression 
Compression codec change
 2013-07-31 11:22:50 -07:00
Karen Feng 49e6344142 Removed master URL from job UI, reduced heading size of basic spark pages 2013-07-31 11:17:59 -07:00
Reynold Xin 0be071abfc Merge pull request #756 from cdshines/patch-1 
Refactored Vector.apply(length, initializer) replacing excessive code with library method
 2013-07-31 11:11:59 -07:00
Reynold Xin c61843a69f Changed other LZF uses to use the compression codec interface. 2013-07-31 10:32:13 -07:00
Patrick Wendell 89da9d94b3 Add JSON path to master index page 2013-07-31 09:47:53 -07:00
BlackNiuza 9a815de4bf write and read generation in ResultTask 2013-08-01 00:36:47 +08:00
Roman Tkalenko 0c6553714a Refactored Vector.apply(length, initializer) replacing excessive code with library method 
(also removed unused variable ```ans``` as minor change)
 2013-07-31 19:05:46 +03:00
Matei Zaharia 12553e5c55 Simplified nonNegativeMod to match previous version 2013-07-31 08:50:28 -07:00
Matei Zaharia d4556f4207 Merge pull request #751 from cdshines/master 
Cleaned Partitioner & PythonPartitioner source by taking out non-related logic to Utils
 2013-07-31 08:48:14 -07:00
Andrew xia 5670c96f29 Merge branch 'master' into Pool_UI 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/spark/scheduler/SparkListener.scala
	core/src/main/scala/spark/scheduler/cluster/ClusterTaskSetManager.scala
	core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
	core/src/main/scala/spark/scheduler/local/LocalTaskSetManager.scala
	core/src/main/scala/spark/ui/jobs/IndexPage.scala
	core/src/main/scala/spark/ui/jobs/JobProgressUI.scala
 2013-07-31 19:36:36 +08:00
cdshines fefb03cbd7 Eliminated code duplication, refactored to pattern-matching style Partitioner and PythonPartitioner 2013-07-31 13:19:42 +03:00
Matei Zaharia 29b8cd3616 Merge pull request #755 from jerryshao/add-apache-header 
Add Apache license header to metrics system
 2013-07-30 21:30:33 -07:00
jerryshao bf9318091a Add Apache license header to metrics system 2013-07-31 09:42:16 +08:00
Shivaram Venkataraman 15fd0d619d Add mllib, bagel to repl dependencies 
Also don't build an assembly jar for them
 2013-07-30 18:31:11 -07:00
Reynold Xin 98024eadc3 Renamed compressionOutputStream and compressionInputStream to compressedOutputStream and compressedInputStream. 2013-07-30 18:28:46 -07:00
Reynold Xin dae12fef9e Updated the configuration option for Snappy block size to be consistent with the documentation. 2013-07-30 17:49:31 -07:00
Reynold Xin 311aae76a2 Added Snappy dependency to Maven build files. 2013-07-30 17:25:42 -07:00
Reynold Xin 3b1ced83fb Exclude older version of Snappy in streaming and examples. 2013-07-30 17:25:36 -07:00
Reynold Xin 56774b176e Added unit test for compression codecs. 2013-07-30 17:12:33 -07:00
Reynold Xin 5227043f84 Documentation update for compression codec. 2013-07-30 17:12:16 -07:00
Reynold Xin ad7e9d0d64 CompressionCodec cleanup. Moved it to spark.io package. 2013-07-30 17:11:54 -07:00
Benjamin Hindman 7bdafa918a Format cleanup. 2013-07-30 17:01:00 -07:00
Reynold Xin 368c58eac5 Merge branch 'lazy_file_open' of github.com:lyogavin/spark into compression 
Conflicts:
	project/SparkBuild.scala
 2013-07-30 16:04:18 -07:00
Patrick Wendell e87de037d6 Merge pull request #744 from karenfeng/bootstrap-update 
Use Bootstrap progress bars in web UI
 2013-07-30 15:00:08 -07:00
shivaram ae57020598 Merge pull request #752 from rxin/master 
Minor mllib cleanup
 2013-07-30 14:56:41 -07:00
Shivaram Venkataraman 48851d4dd9 Add bagel, mllib to SBT assembly. 
Also add jblas dependency to mllib pom.xml
 2013-07-30 14:03:15 -07:00
Reynold Xin 366f7735eb Minor style cleanup of mllib. 2013-07-30 13:59:32 -07:00
Reynold Xin 47011e6854 Use a tigher bound in logistic regression unit test's prediction validation. 2013-07-30 13:58:23 -07:00
Reynold Xin e35966ae9a Renamed Classification.scala to ClassificationModel.scala and Regression.scala to RegressionModel.scala 2013-07-30 13:28:31 -07:00
Karen Feng 26144c400f Fixed wrap style 2013-07-30 12:40:41 -07:00
Karen Feng 218d7c4ed8 Fixed style, lowered height of progress bars 2013-07-30 12:39:17 -07:00
Karen Feng f1cab31b73 Removed intermediate set for activeTasks, removed progress bar margin 2013-07-30 11:06:47 -07:00
shivaram 8aee118216 Merge pull request #748 from atalwalkar/master 
made SimpleUpdater consistent with other updaters
 2013-07-30 10:27:54 -07:00
Benjamin Hindman f6f46455eb Added property 'spark.executor.uri' for launching on Mesos without 
requiring Spark to be installed. Using 'make_distribution.sh' a user
can put a Spark distribution at a URI supported by Mesos (e.g.,
'hdfs://...') and then set that when launching their job. Also added
SPARK_EXECUTOR_URI for the REPL.
 2013-07-29 23:32:52 -07:00
Ameet Talwalkar e4387ddf5d made SimpleUpdater consistent with other updaters 2013-07-29 22:21:50 -07:00
Josh Rosen 49be084ed3 Use File.pathSeparator instead of hardcoding ':'. 2013-07-29 22:08:57 -07:00
Josh Rosen b95732632b Do not inherit master's PYTHONPATH on workers. 
This fixes SPARK-832, an issue where PySpark
would not work when the master and workers used
different SPARK_HOME paths.

This change may potentially break code that relied
on the master's PYTHONPATH being used on workers.
To have custom PYTHONPATH additions used on the
workers, users should set a custom PYTHONPATH in
spark-env.sh rather than setting it in the shell.
 2013-07-29 22:08:57 -07:00
Andrew xia 5406013997 refactor codes less than 100 character per line 2013-07-30 11:41:38 +08:00
Andrew xia 614ee16cc4 refactor job ui with pool information 2013-07-30 10:57:26 +08:00