ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5085 commits 6 branches 181 tags 397 MiB
Commit graph

5085 commits

This branch
This branch
All branches
Author SHA1 Message Date
Reynold Xin 72a17b69f5 Revert "Merge pull request #310 from jyunfan/master" 
This reverts commit 79b20e4dbe, reversing
changes made to 7375047d51.
 2013-12-28 21:25:40 -10:00
Reynold Xin 79b20e4dbe Merge pull request #310 from jyunfan/master 
Fix typo in the Accumulators section

Change 'val' to 'var'
 2013-12-28 21:13:36 -10:00
Matei Zaharia 1c11f54a9b Fix Python use of getLocalDir 2013-12-29 00:11:36 -05:00
Tor Myklebust fec01664a7 Make Python function/line appear in the UI. 2013-12-28 23:34:16 -05:00
Tor Myklebust d812aeece9 Factor call site reporting out to SparkContext. 2013-12-28 23:21:49 -05:00
Matei Zaharia 20631348d1 Fix other failing tests 2013-12-28 23:17:58 -05:00
Jyun-Fan Tsai 17f6620a71 Fix typo in the Accumulators section 
val => var
 2013-12-29 11:30:02 +08:00
Matei Zaharia 0900d5c72a Add a StreamingContext constructor that takes a conf object 2013-12-28 21:38:07 -05:00
Matei Zaharia a8f316386a Fix CheckpointSuite test failures 2013-12-28 21:26:43 -05:00
Matei Zaharia 578bd1fc28 Fix test failures due to setting / clearing clock type in Streaming 2013-12-28 21:21:06 -05:00
Matei Zaharia 5bbe73864e Fix Executor not getting properties in local mode 2013-12-28 17:31:58 -05:00
Matei Zaharia a16c52ed1b Check for SPARK_YARN_MODE through a system property too since it can 
sometimes be set that way (undoes a change in previous commit)
 2013-12-28 17:24:21 -05:00
Matei Zaharia 642029e7f4 Various fixes to configuration code 
- Got rid of global SparkContext.globalConf
- Pass SparkConf to serializers and compression codecs
- Made SparkConf public instead of private[spark]
- Improved API of SparkContext and SparkConf
- Switched executor environment vars to be passed through SparkConf
- Fixed some places that were still using system properties
- Fixed some tests, though others are still failing

This still fails several tests in core, repl and streaming, likely due
to properties not being set or cleared correctly (some of the tests run
fine in isolation).
 2013-12-28 17:13:15 -05:00
Patrick Wendell 7375047d51 Merge pull request #304 from kayousterhout/remove_unused 
Removed unused failed and causeOfFailure variables (in TaskSetManager)
 2013-12-28 13:25:06 -08:00
Matei Zaharia ad3dfd1531 Merge pull request #307 from kayousterhout/other_failure 
Removed unused OtherFailure TaskEndReason.

The OtherFailure TaskEndReason was added by @mateiz 3 years ago in this commit: 24a1e7f838

Unless I am missing something, it doesn't seem to have been used then, and is not used now, so seems safe for deletion.
 2013-12-27 22:10:14 -05:00
Matei Zaharia b579b83277 Merge pull request #306 from kayousterhout/remove_pending 
Remove unused hasPendingTasks methods
 2013-12-27 22:09:04 -05:00
Kay Ousterhout b4619e509b Changed naming of StageCompleted event to be consistent 
The rest of the SparkListener events are named with "SparkListener"
as the prefix of the name; this commit renames the StageCompleted
event to SparkListenerStageCompleted for consistency.
 2013-12-27 17:45:20 -08:00
Kay Ousterhout e17d7518ab Removed unused OtherFailure TaskEndReason. 2013-12-27 15:51:27 -08:00
Kay Ousterhout 8419148e5f Remove unused hasPendingTasks methods 2013-12-27 15:19:42 -08:00
Patrick Wendell 19672dca32 Merge pull request #305 from kayousterhout/line_spacing 
Fixed >100char lines in DAGScheduler.scala

There's no changed functionality here -- only line spacing and one grammatical fix in a comment.
 2013-12-27 13:37:10 -08:00
Tathagata Das 271e3237f3 Minor changes in comments and strings to address comments in PR 289. 2013-12-27 12:26:57 -08:00
Kay Ousterhout 0c71ffe924 Style fixes as per Reynold's review 2013-12-27 12:19:38 -08:00
Kay Ousterhout 8c81068e16 Fixed >100char lines in DAGScheduler.scala 2013-12-27 11:36:54 -08:00
Binh Nguyen 2c5bade4ee Fix failed unit tests 
Also clean up a bit.
 2013-12-27 11:24:30 -08:00
Kay Ousterhout baaabcedc9 Removed unused failed and causeOfFailure variables 2013-12-27 11:12:36 -08:00
Reynold Xin 7be1e57786 Merge pull request #298 from aarondav/minor 
Minor: Decrease margin of left side of Log page

Before
![before](https://f.cloud.github.com/assets/1400247/1812647/1a4be53e-6e87-11e3-9d5b-f851274be0e9.png)

After
![after](https://f.cloud.github.com/assets/1400247/1812648/1ca1ea2c-6e87-11e3-946c-31be9258f450.png)

It's a start anyway...
 2013-12-26 23:41:40 -10:00
Reynold Xin 7d811ba6f2 Merge pull request #302 from pwendell/SPARK-1007 
SPARK-1007: spark-class2.cmd should change SCALA_VERSION to be 2.10

Reported by Qiuzhuang Lian
 2013-12-26 23:39:58 -10:00
Patrick Wendell 0cc1e0d43d SPARK-1007: spark-class2.cmd should change SCALA_VERSION to be 2.10 2013-12-26 23:21:08 -08:00
Matei Zaharia 5e69fc5bb4 Merge pull request #295 from markhamstra/JobProgressListenerNPE 
Avoid a lump of coal (NPE) in JobProgressListener's stocking.
 2013-12-26 19:10:39 -05:00
Aaron Davidson 4f2fb761b0 Decrease margin of left side of log page 2013-12-26 15:38:45 -08:00
Tathagata Das 5fde4566ea Added Apache boilerplate and class docs to PartitionerAwareUnionRDD. 2013-12-26 14:33:37 -08:00
Tathagata Das 577c8cc834 Removed unncessary options from WindowedDStream. 2013-12-26 14:17:16 -08:00
Tathagata Das 3618d70b2a Added warning if filestream adds files with no data in them (file RDDs have 0 partitions). 2013-12-26 12:45:40 -08:00
Tathagata Das be64719138 Changed file stream to not catch any exceptions related to finding new files (FileNotFound exception is still caught and ignored). 2013-12-26 12:33:12 -08:00
Tathagata Das 3579647cdc Merge branch 'apache-master' into window-improvement 2013-12-26 12:12:10 -08:00
Tathagata Das c4a54f51b5 Merge branch 'master' into window-improvement 2013-12-26 12:03:11 -08:00
Matei Zaharia e240bad03b Merge pull request #296 from witgo/master 
Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn package
 2013-12-26 12:30:48 -05:00
Tathagata Das 069cb14bdc Updated groupByKeyAndWindow to be computed incrementally, and added mapSideCombine to combineByKeyAndWindow. 2013-12-26 02:58:29 -08:00
Tathagata Das bacc65cf28 Removed slack time in file stream and added better handling of exceptions due to failures due FileNotFound exceptions. 2013-12-26 10:18:46 +00:00
liguoqiang b662c88a24 fix this import order 2013-12-26 15:49:33 +08:00
Mark Hamstra c529dceaff Avoid a lump of coal (NPE) in JobProgressListener's stocking. 2013-12-25 23:10:02 -08:00
Matei Zaharia c344ed04c7 Merge pull request #283 from tmyklebu/master 
Python bindings for mllib

This pull request contains Python bindings for the regression, clustering, classification, and recommendation tools in mllib.

For each 'train' frontend exposed, there is a Scala stub in PythonMLLibAPI.scala and a Python stub in mllib.py.  The Python stub serialises the input RDD and any vector/matrix arguments into a mutually-understood format and calls the Scala stub.  The Scala stub deserialises the RDD and the vector/matrix arguments, calls the appropriate 'train' function, serialises the resulting model, and returns the serialised model.

ALSModel is slightly different since a MatrixFactorizationModel has RDDs inside.  The Scala stub returns a handle to a Scala MatrixFactorizationModel; prediction is done by calling the Scala predict method.

I have tested these bindings on an x86_64 machine running Linux.  There is a risk that these bindings may fail on some choose-your-own-endian platform if Python's endian differs from java.nio.ByteBuffer's idea of the native byte order.
 2013-12-26 01:31:06 -05:00
liguoqiang 2bd76f693d Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn 2013-12-26 11:10:35 +08:00
liguoqiang 14fcef72db Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn 2013-12-26 11:05:07 +08:00
Tathagata Das 94479673eb Fixed bug in PartitionAwareUnionRDD 2013-12-26 00:07:45 +00:00
Tor Myklebust 9cbcf81453 Remove commented code in __init__.py. 2013-12-25 14:12:42 -05:00
Tor Myklebust 5e71354cb7 Fix copypasta in __init__.py. Don't import anything directly into pyspark.mllib. 2013-12-25 14:10:55 -05:00
Matei Zaharia 56094bcd8d Merge pull request #290 from ash211/patch-3 
Typo: avaiable -> available
 2013-12-25 13:14:33 -05:00
Reynold Xin 4842a07da8 Merge pull request #287 from azuryyu/master 
Fixed job name in the java streaming example.
 2013-12-25 01:52:15 -08:00
Tor Myklebust 02208a175c Initial weights in Scala are ones; do that too. Also fix some errors. 2013-12-25 00:53:48 -05:00