ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3600 commits 6 branches 181 tags 397 MiB
Commit graph

3600 commits

This branch
This branch
All branches
Author SHA1 Message Date
Andrew xia 5670c96f29 Merge branch 'master' into Pool_UI 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/spark/scheduler/SparkListener.scala
	core/src/main/scala/spark/scheduler/cluster/ClusterTaskSetManager.scala
	core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
	core/src/main/scala/spark/scheduler/local/LocalTaskSetManager.scala
	core/src/main/scala/spark/ui/jobs/IndexPage.scala
	core/src/main/scala/spark/ui/jobs/JobProgressUI.scala
 2013-07-31 19:36:36 +08:00
cdshines fefb03cbd7 Eliminated code duplication, refactored to pattern-matching style Partitioner and PythonPartitioner 2013-07-31 13:19:42 +03:00
Dmitriy Lyubimov 96664431cb IDEA flipped JavaSerialized import at some point to a wrong class. 2013-07-30 23:10:09 -07:00
Dmitriy Lyubimov c219fc94fd Minor, style 2013-07-30 22:08:39 -07:00
Matei Zaharia 29b8cd3616 Merge pull request #755 from jerryshao/add-apache-header 
Add Apache license header to metrics system
 2013-07-30 21:30:33 -07:00
Dmitriy Lyubimov f4b4b8836e reverting back to one-by-one serialization for parallelize() 2013-07-30 19:00:58 -07:00
jerryshao bf9318091a Add Apache license header to metrics system 2013-07-31 09:42:16 +08:00
Shivaram Venkataraman 15fd0d619d Add mllib, bagel to repl dependencies 
Also don't build an assembly jar for them
 2013-07-30 18:31:11 -07:00
Reynold Xin 98024eadc3 Renamed compressionOutputStream and compressionInputStream to compressedOutputStream and compressedInputStream. 2013-07-30 18:28:46 -07:00
Dmitriy Lyubimov abada94ebf removing default constructor (not Externalizable any more) 2013-07-30 18:04:02 -07:00
Dmitriy Lyubimov 943c6590c9 realiging "extends" back manually 2013-07-30 18:01:35 -07:00
Dmitriy Lyubimov ca33b12e98 resetting wrap and continuation indent = 4 2013-07-30 17:51:44 -07:00
Reynold Xin dae12fef9e Updated the configuration option for Snappy block size to be consistent with the documentation. 2013-07-30 17:49:31 -07:00
Dmitriy Lyubimov 483d8dd694 Reverting SparkBuild w.r.t to branch fork ratherh than current trunk to avoid diffs appearing 2013-07-30 17:41:24 -07:00
Dmitriy Lyubimov 7ef2551af0 reverting build file..again 2013-07-30 17:39:43 -07:00
Dmitriy Lyubimov 984b56155a changing approaches for parallelize(): java serialization needs to avoid writing headers! 2013-07-30 17:36:59 -07:00
Reynold Xin 311aae76a2 Added Snappy dependency to Maven build files. 2013-07-30 17:25:42 -07:00
Reynold Xin 3b1ced83fb Exclude older version of Snappy in streaming and examples. 2013-07-30 17:25:36 -07:00
Reynold Xin 56774b176e Added unit test for compression codecs. 2013-07-30 17:12:33 -07:00
Reynold Xin 5227043f84 Documentation update for compression codec. 2013-07-30 17:12:16 -07:00
Reynold Xin ad7e9d0d64 CompressionCodec cleanup. Moved it to spark.io package. 2013-07-30 17:11:54 -07:00
Benjamin Hindman 7bdafa918a Format cleanup. 2013-07-30 17:01:00 -07:00
Dmitriy Lyubimov ef9529a943 refactoring using writeByteBuffer() from Utils. 2013-07-30 16:24:23 -07:00
Dmitriy Lyubimov 43394b9a6d fixing formatting 2013-07-30 16:13:41 -07:00
Dmitriy Lyubimov 13a9d66645 adding === 2013-07-30 16:10:55 -07:00
Reynold Xin 368c58eac5 Merge branch 'lazy_file_open' of github.com:lyogavin/spark into compression 
Conflicts:
	project/SparkBuild.scala
 2013-07-30 16:04:18 -07:00
Patrick Wendell e87de037d6 Merge pull request #744 from karenfeng/bootstrap-update 
Use Bootstrap progress bars in web UI
 2013-07-30 15:00:08 -07:00
shivaram ae57020598 Merge pull request #752 from rxin/master 
Minor mllib cleanup
 2013-07-30 14:56:41 -07:00
Shivaram Venkataraman 48851d4dd9 Add bagel, mllib to SBT assembly. 
Also add jblas dependency to mllib pom.xml
 2013-07-30 14:03:15 -07:00
Reynold Xin 366f7735eb Minor style cleanup of mllib. 2013-07-30 13:59:32 -07:00
Reynold Xin 47011e6854 Use a tigher bound in logistic regression unit test's prediction validation. 2013-07-30 13:58:23 -07:00
Reynold Xin e35966ae9a Renamed Classification.scala to ClassificationModel.scala and Regression.scala to RegressionModel.scala 2013-07-30 13:28:31 -07:00
Karen Feng 26144c400f Fixed wrap style 2013-07-30 12:40:41 -07:00
Karen Feng 218d7c4ed8 Fixed style, lowered height of progress bars 2013-07-30 12:39:17 -07:00
Karen Feng f1cab31b73 Removed intermediate set for activeTasks, removed progress bar margin 2013-07-30 11:06:47 -07:00
Dmitriy Lyubimov 1bca91633e + bug fixes; 
test added

Conflicts:

	core/src/test/scala/spark/KryoSerializerSuite.scala
 2013-07-30 11:04:11 -07:00
shivaram 8aee118216 Merge pull request #748 from atalwalkar/master 
made SimpleUpdater consistent with other updaters
 2013-07-30 10:27:54 -07:00
Benjamin Hindman f6f46455eb Added property 'spark.executor.uri' for launching on Mesos without 
requiring Spark to be installed. Using 'make_distribution.sh' a user
can put a Spark distribution at a URI supported by Mesos (e.g.,
'hdfs://...') and then set that when launching their job. Also added
SPARK_EXECUTOR_URI for the REPL.
 2013-07-29 23:32:52 -07:00
Ameet Talwalkar e4387ddf5d made SimpleUpdater consistent with other updaters 2013-07-29 22:21:50 -07:00
Josh Rosen 49be084ed3 Use File.pathSeparator instead of hardcoding ':'. 2013-07-29 22:08:57 -07:00
Josh Rosen b95732632b Do not inherit master's PYTHONPATH on workers. 
This fixes SPARK-832, an issue where PySpark
would not work when the master and workers used
different SPARK_HOME paths.

This change may potentially break code that relied
on the master's PYTHONPATH being used on workers.
To have custom PYTHONPATH additions used on the
workers, users should set a custom PYTHONPATH in
spark-env.sh rather than setting it in the shell.
 2013-07-29 22:08:57 -07:00
Andrew xia 5406013997 refactor codes less than 100 character per line 2013-07-30 11:41:38 +08:00
Andrew xia 614ee16cc4 refactor job ui with pool information 2013-07-30 10:57:26 +08:00
Matei Zaharia 468a36c005 Merge pull request #746 from rxin/cleanup 
Internal cleanup
 2013-07-29 19:44:33 -07:00
atalwalkar 1e1ffb192a Merge pull request #745 from shivaram/loss-update-fix 
Remove duplicate loss history in Gradient Descent
 2013-07-29 19:26:19 -07:00
Matei Zaharia 01f94931d5 Update the Python logistic regression example to read from a file and 
batch input records for more efficient NumPy computations
 2013-07-29 19:23:41 -07:00
Dmitriy Lyubimov 8e5cd041bb initial externalization of ParallelCollectionRDD's split 2013-07-29 19:02:53 -07:00
Shivaram Venkataraman 3ca9faa341 Clarify how regVal is computed in Updater docs 2013-07-29 18:37:28 -07:00
Reynold Xin 81720e13fc Moved all StandaloneClusterMessage's into StandaloneClusterMessages object. 2013-07-29 17:53:01 -07:00
Reynold Xin 23b5da14ed Moved block manager messages into BlockManagerMessages object. 2013-07-29 17:42:05 -07:00