ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4028 commits 6 branches 181 tags 397 MiB
Commit graph

4028 commits

This branch
This branch
All branches
Author SHA1 Message Date
Patrick Wendell bddf135670 Change port from 3030 to 4040 2013-09-11 10:01:38 -07:00
Matei Zaharia 2425eb85ca Update Python API features 2013-09-10 11:12:59 -07:00
shivaram 8c14f4b722 Merge pull request #917 from pwendell/master 
Document libgfortran dependency for MLBase
 2013-09-09 22:07:58 -07:00
Patrick Wendell cefee1ed1a Document fortran dependency for MLBase 2013-09-09 21:45:04 -07:00
Matei Zaharia c81377b9ed Merge pull request #915 from ooyala/master 
Get rid of / improve ugly NPE when Utils.deleteRecursively() fails
 2013-09-09 20:16:19 -07:00
Matei Zaharia 61d2a010e1 Merge pull request #916 from mateiz/mkdist-fix 
Fix copy issue in https://github.com/mesos/spark/pull/899
 2013-09-09 18:21:01 -07:00
Evan Chan fdb8b0eec3 Style fix: put body of if within curly braces 2013-09-09 14:29:32 -07:00
Matei Zaharia f5a8afa6c3 Fix copy issue in https://github.com/mesos/spark/pull/899 2013-09-09 13:47:56 -07:00
Matei Zaharia a85758c200 Merge pull request #907 from stephenh/document_coalesce_shuffle 
Add better docs for coalesce.
 2013-09-09 13:45:40 -07:00
Evan Chan 27726079e4 Print out more friendly error if listFiles() fails 
listFiles() could return null if the I/O fails, and this currently results in an ugly NPE which is hard to diagnose.
 2013-09-09 12:58:12 -07:00
Matei Zaharia 084fc36961 Merge pull request #912 from tgravescs/ganglia-pom 
Add metrics-ganglia to core pom file
 2013-09-09 12:01:35 -07:00
Y.CORP.YAHOO.COM\tgraves 2186d93285 Add metrics-ganglia to core pom file 2013-09-09 12:37:33 -05:00
Matei Zaharia 0456384939 Merge pull request #911 from pwendell/ganglia-sink 
Adding Manen dependency for Ganglia
 2013-09-09 09:57:54 -07:00
Stephen Haberman 59003d387d Use a set since shuffle could change order. 2013-09-09 11:45:03 -05:00
Stephen Haberman 6471bfec73 Reword 'evenly distributed' to 'distributed with a hash partitioner. 2013-09-09 11:44:15 -05:00
Patrick Wendell 528fdbae97 Adding Manen dependency 2013-09-09 09:32:18 -07:00
Matei Zaharia bf984e2745 Merge pull request #890 from mridulm/master 
Fix hash bug
 2013-09-08 23:50:24 -07:00
Reynold Xin e9d4f44a7a Merge pull request #909 from mateiz/exec-id-fix 
Fix an instance where full standalone mode executor IDs were passed to
 2013-09-08 23:36:48 -07:00
Matei Zaharia 2447b1c4e6 Merge pull request #910 from mateiz/ml-doc-tweaks 
Small tweaks to MLlib docs
 2013-09-08 22:27:49 -07:00
Matei Zaharia 7a5c4b647b Small tweaks to MLlib docs 2013-09-08 21:47:24 -07:00
Matei Zaharia 7d3204b056 Merge pull request #905 from mateiz/docs2 
Job scheduling and cluster mode docs
 2013-09-08 21:39:12 -07:00
Matei Zaharia f1f83712f4 Merge pull request #896 from atalwalkar/master 
updated content
 2013-09-08 21:26:11 -07:00
Matei Zaharia b458854977 Fix some review comments 2013-09-08 21:25:49 -07:00
Ameet Talwalkar 81a8bd46ac respose to PR comments 2013-09-08 19:21:30 -07:00
Ameet Talwalkar bf280c8b0f Merge remote-tracking branch 'upstream/master' 2013-09-08 18:41:38 -07:00
Patrick Wendell f68848d95d Merge pull request #906 from pwendell/ganglia-sink 
Clean-up of Metrics Code/Docs and Add Ganglia Sink
 2013-09-08 18:32:16 -07:00
Matei Zaharia f9b7f58de2 Fix an instance where full standalone mode executor IDs were passed to 
StandaloneSchedulerBackend instead of the smaller IDs used within Spark
(that lack the application name).

This was reported by ClearStory in
https://github.com/clearstorydata/spark/pull/9.

Also fixed some messages that said slave instead of executor.
 2013-09-08 18:27:50 -07:00
Matei Zaharia 170b3869ee Fix unit test failure due to changed default 2013-09-08 17:51:27 -07:00
Ameet Talwalkar 5ac62dbbd0 updates based on comments to PR 2013-09-08 17:39:08 -07:00
Patrick Wendell b4e382c210 Adding sc name in metrics source 2013-09-08 16:06:49 -07:00
Patrick Wendell 8026537597 Fixing package name in template conf 2013-09-08 16:06:32 -07:00
Matei Zaharia 0b957997ad Merge pull request #908 from pwendell/master 
Fix target JVM version in scala build
 2013-09-08 15:30:16 -07:00
Patrick Wendell 27bd74c8ad Fix target JVM version in scala build 2013-09-08 14:37:45 -07:00
Matei Zaharia 5a587fb98d Updated cluster diagram to show caches 2013-09-08 13:51:57 -07:00
Patrick Wendell c190b48bf5 Adding more docs and some code cleanup 2013-09-08 13:46:28 -07:00
Stephen Haberman df5fd35273 Add better docs for coalesce. 
Include the useful tip that if shuffle=true, coalesce can actually
increase the number of partitions.

This makes coalesce more like a generic `RDD.repartition` operation.

(Ideally this `RDD.repartition` could automatically choose either a coalesce or
a shuffle if numPartitions was either less than or greater than, respectively,
the current number of partitions.)
 2013-09-08 15:39:04 -05:00
Matei Zaharia af8ffdb73c Review comments 2013-09-08 13:36:50 -07:00
Matei Zaharia 04cfb3aa9d Merge pull request #898 from ilikerps/660 
SPARK-660: Add StorageLevel support in Python
 2013-09-08 10:33:20 -07:00
Patrick Wendell 8de8ee5d3c Ganglia sink 2013-09-08 10:08:18 -07:00
Matei Zaharia c0d375107f Some tweaks to CDH/HDP doc 2013-09-08 00:44:41 -07:00
Aaron Davidson a3868544be Whoopsy daisy 2013-09-08 00:30:47 -07:00
Matei Zaharia f261d2a60f Added cluster overview doc, made logo higher-resolution, and added more 
details on monitoring
 2013-09-08 00:29:11 -07:00
Matei Zaharia 651a96adf7 More fair scheduler docs and property names. 
Also changed uses of "job" terminology to "application" when they
referred to an entire Spark program, to avoid confusion.
 2013-09-08 00:29:11 -07:00
Matei Zaharia 98fb69822c Work in progress: 
- Add job scheduling docs
- Rename some fair scheduler properties
- Organize intro page better
- Link to Apache wiki for "contributing to Spark"
 2013-09-08 00:29:11 -07:00
Matei Zaharia 38488aca8a Merge pull request #900 from pwendell/cdh-docs 
Provide docs to describe running on CDH/HDP cluster.
 2013-09-08 00:28:53 -07:00
Patrick Wendell a8e376ec0f Merge pull request #904 from pwendell/master 
Adding Apache license to two files
 2013-09-07 21:16:01 -07:00
Patrick Wendell 6d2198643c Adding Apache license to two files 2013-09-07 20:46:58 -07:00
Aaron Davidson c1cc8c4da2 Export StorageLevel and refactor 2013-09-07 14:41:31 -07:00
Patrick Wendell 22b982d2bc File rename 2013-09-07 14:38:54 -07:00
Matei Zaharia cfde85e395 Merge pull request #901 from ooyala/2013-09/0.8-doc-changes 
0.8 Doc changes for make-distribution.sh
 2013-09-07 13:53:08 -07:00