Patrick Wendell
d40f1403f3
Merge pull request #921 from pwendell/master
...
Fix HDFS access bug with assembly build.
2013-09-10 23:05:29 -07:00
Haoyuan Li
56b9407848
fix run-example script
2013-09-10 23:03:09 -07:00
Patrick Wendell
0c1985b153
Fix HDFS access bug with assembly build.
...
Due to this change in HDFS:
https://issues.apache.org/jira/browse/HADOOP-7549
there is a bug when using the new assembly builds. The symptom is that any HDFS access
results in an exception saying "No filesystem for scheme 'hdfs'". This adds a merge
strategy in the assembly build which fixes the problem.
2013-09-10 22:05:13 -07:00
Matei Zaharia
2425eb85ca
Update Python API features
2013-09-10 11:12:59 -07:00
Matei Zaharia
f117dc6d0d
Add explicit jets3t dependency, which is excluded in hadoop-client
2013-09-10 06:39:25 +00:00
shivaram
8c14f4b722
Merge pull request #917 from pwendell/master
...
Document libgfortran dependency for MLBase
2013-09-09 22:07:58 -07:00
Patrick Wendell
cefee1ed1a
Document fortran dependency for MLBase
2013-09-09 21:45:04 -07:00
Matei Zaharia
c81377b9ed
Merge pull request #915 from ooyala/master
...
Get rid of / improve ugly NPE when Utils.deleteRecursively() fails
2013-09-09 20:16:19 -07:00
Matei Zaharia
61d2a010e1
Merge pull request #916 from mateiz/mkdist-fix
...
Fix copy issue in https://github.com/mesos/spark/pull/899
2013-09-09 18:21:01 -07:00
Evan Chan
fdb8b0eec3
Style fix: put body of if within curly braces
2013-09-09 14:29:32 -07:00
Matei Zaharia
f5a8afa6c3
Fix copy issue in https://github.com/mesos/spark/pull/899
2013-09-09 13:47:56 -07:00
Matei Zaharia
a85758c200
Merge pull request #907 from stephenh/document_coalesce_shuffle
...
Add better docs for coalesce.
2013-09-09 13:45:40 -07:00
Evan Chan
27726079e4
Print out more friendly error if listFiles() fails
...
listFiles() could return null if the I/O fails, and this currently results in an ugly NPE which is hard to diagnose.
2013-09-09 12:58:12 -07:00
Matei Zaharia
084fc36961
Merge pull request #912 from tgravescs/ganglia-pom
...
Add metrics-ganglia to core pom file
2013-09-09 12:01:35 -07:00
Y.CORP.YAHOO.COM\tgraves
2186d93285
Add metrics-ganglia to core pom file
2013-09-09 12:37:33 -05:00
Matei Zaharia
0456384939
Merge pull request #911 from pwendell/ganglia-sink
...
Adding Manen dependency for Ganglia
2013-09-09 09:57:54 -07:00
Stephen Haberman
59003d387d
Use a set since shuffle could change order.
2013-09-09 11:45:03 -05:00
Stephen Haberman
6471bfec73
Reword 'evenly distributed' to 'distributed with a hash partitioner.
2013-09-09 11:44:15 -05:00
Patrick Wendell
528fdbae97
Adding Manen dependency
2013-09-09 09:32:18 -07:00
Matei Zaharia
bf984e2745
Merge pull request #890 from mridulm/master
...
Fix hash bug
2013-09-08 23:50:24 -07:00
Reynold Xin
e9d4f44a7a
Merge pull request #909 from mateiz/exec-id-fix
...
Fix an instance where full standalone mode executor IDs were passed to
2013-09-08 23:36:48 -07:00
Matei Zaharia
2447b1c4e6
Merge pull request #910 from mateiz/ml-doc-tweaks
...
Small tweaks to MLlib docs
2013-09-08 22:27:49 -07:00
Matei Zaharia
7a5c4b647b
Small tweaks to MLlib docs
2013-09-08 21:47:24 -07:00
Matei Zaharia
7d3204b056
Merge pull request #905 from mateiz/docs2
...
Job scheduling and cluster mode docs
2013-09-08 21:39:12 -07:00
Matei Zaharia
f1f83712f4
Merge pull request #896 from atalwalkar/master
...
updated content
2013-09-08 21:26:11 -07:00
Matei Zaharia
b458854977
Fix some review comments
2013-09-08 21:25:49 -07:00
Ameet Talwalkar
81a8bd46ac
respose to PR comments
2013-09-08 19:21:30 -07:00
Ameet Talwalkar
bf280c8b0f
Merge remote-tracking branch 'upstream/master'
2013-09-08 18:41:38 -07:00
Patrick Wendell
f68848d95d
Merge pull request #906 from pwendell/ganglia-sink
...
Clean-up of Metrics Code/Docs and Add Ganglia Sink
2013-09-08 18:32:16 -07:00
Matei Zaharia
f9b7f58de2
Fix an instance where full standalone mode executor IDs were passed to
...
StandaloneSchedulerBackend instead of the smaller IDs used within Spark
(that lack the application name).
This was reported by ClearStory in
https://github.com/clearstorydata/spark/pull/9 .
Also fixed some messages that said slave instead of executor.
2013-09-08 18:27:50 -07:00
Matei Zaharia
170b3869ee
Fix unit test failure due to changed default
2013-09-08 17:51:27 -07:00
Ameet Talwalkar
5ac62dbbd0
updates based on comments to PR
2013-09-08 17:39:08 -07:00
Patrick Wendell
b4e382c210
Adding sc name in metrics source
2013-09-08 16:06:49 -07:00
Patrick Wendell
8026537597
Fixing package name in template conf
2013-09-08 16:06:32 -07:00
Matei Zaharia
0b957997ad
Merge pull request #908 from pwendell/master
...
Fix target JVM version in scala build
2013-09-08 15:30:16 -07:00
Patrick Wendell
27bd74c8ad
Fix target JVM version in scala build
2013-09-08 14:37:45 -07:00
Matei Zaharia
5a587fb98d
Updated cluster diagram to show caches
2013-09-08 13:51:57 -07:00
Patrick Wendell
c190b48bf5
Adding more docs and some code cleanup
2013-09-08 13:46:28 -07:00
Stephen Haberman
df5fd35273
Add better docs for coalesce.
...
Include the useful tip that if shuffle=true, coalesce can actually
increase the number of partitions.
This makes coalesce more like a generic `RDD.repartition` operation.
(Ideally this `RDD.repartition` could automatically choose either a coalesce or
a shuffle if numPartitions was either less than or greater than, respectively,
the current number of partitions.)
2013-09-08 15:39:04 -05:00
Matei Zaharia
af8ffdb73c
Review comments
2013-09-08 13:36:50 -07:00
Matei Zaharia
04cfb3aa9d
Merge pull request #898 from ilikerps/660
...
SPARK-660: Add StorageLevel support in Python
2013-09-08 10:33:20 -07:00
Patrick Wendell
8de8ee5d3c
Ganglia sink
2013-09-08 10:08:18 -07:00
Matei Zaharia
c0d375107f
Some tweaks to CDH/HDP doc
2013-09-08 00:44:41 -07:00
Aaron Davidson
a3868544be
Whoopsy daisy
2013-09-08 00:30:47 -07:00
Matei Zaharia
f261d2a60f
Added cluster overview doc, made logo higher-resolution, and added more
...
details on monitoring
2013-09-08 00:29:11 -07:00
Matei Zaharia
651a96adf7
More fair scheduler docs and property names.
...
Also changed uses of "job" terminology to "application" when they
referred to an entire Spark program, to avoid confusion.
2013-09-08 00:29:11 -07:00
Matei Zaharia
98fb69822c
Work in progress:
...
- Add job scheduling docs
- Rename some fair scheduler properties
- Organize intro page better
- Link to Apache wiki for "contributing to Spark"
2013-09-08 00:29:11 -07:00
Matei Zaharia
38488aca8a
Merge pull request #900 from pwendell/cdh-docs
...
Provide docs to describe running on CDH/HDP cluster.
2013-09-08 00:28:53 -07:00
Patrick Wendell
a8e376ec0f
Merge pull request #904 from pwendell/master
...
Adding Apache license to two files
2013-09-07 21:16:01 -07:00
Patrick Wendell
6d2198643c
Adding Apache license to two files
2013-09-07 20:46:58 -07:00