Andy Konwinski
bd688940a1
A start on scaladoc for the public APIs.
2012-10-08 21:13:29 -07:00
Matei Zaharia
a91ed6a3a7
Merge pull request #256 from pwendell/new-download-instr
...
Adding new download instructions
2012-10-08 17:25:25 -07:00
Andy Konwinski
e1a724f39c
Updating lots of docs to use the new special version number variables,
...
also adding the version to the navbar so it is easy to tell which
version of Spark these docs were compiled for.
2012-10-08 17:17:17 -07:00
Patrick Wendell
7887e171f4
Adding new download instructions
2012-10-08 17:05:13 -07:00
Matei Zaharia
1231eb12e6
Merge pull request #259 from mosharaf/bc-fix-dev
...
Synchronization bug fix in broadcast implementations
2012-10-08 16:54:28 -07:00
Mosharaf Chowdhury
edc67bfba8
Merge branch 'dev' into bc-fix-dev
2012-10-08 16:19:13 -07:00
Matei Zaharia
4f1e0325f1
Merge pull request #257 from andyk/add-version-vars-to-docs
...
Adds special version variables to docs templating system
2012-10-08 12:29:08 -07:00
Andy Konwinski
56b91ab380
Updates README.md with instructions for running jekyll without building
...
scaladoc (i.e. run `SKIP_SCALADOC=1 jekyll`).
2012-10-08 12:15:44 -07:00
Andy Konwinski
89f8e1c2e7
Merge remote-tracking branch 'public-spark/dev' into add-version-vars-to-docs
...
Conflicts:
docs/quick-start.md
2012-10-08 12:15:38 -07:00
Matei Zaharia
88152e2164
Merge pull request #254 from pwendell/quickstart-fix
...
Removing one link in quickstart
2012-10-08 11:08:29 -07:00
Andy Konwinski
45d03231d0
Adds liquid variables to docs templating system so that they can be used
...
throughout the docs: SPARK_VERSION, SCALA_VERSION, and MESOS_VERSION.
To use them, e.g. use {{site.SPARK_VERSION}}.
Also removes uses of {{HOME_PATH}} which were being resolved to ""
by the templating system anyway.
2012-10-08 10:30:38 -07:00
Mosharaf Chowdhury
46c3899836
Merge remote-tracking branch 'upstream/dev' into dev
2012-10-08 09:47:46 -07:00
Matei Zaharia
280286b6cd
Merge pull request #255 from andyk/fix-nav-gap
...
Removes the annoying small gap above the nav menu dropdown boxes
2012-10-08 09:45:03 -07:00
Andy Konwinski
cd710a7718
Fixes the small gap above the nav menu dropdown boxes and the hoverable
...
menu items that was causing the dropdowns to go away when the user
moved their mouse down towards them.
2012-10-08 09:33:40 -07:00
Patrick Wendell
99aac2f6c4
Removing one link in quickstart
2012-10-08 08:53:27 -07:00
Matei Zaharia
efc5423210
Made compression configurable separately for shuffle, broadcast and RDDs
2012-10-07 11:30:53 -07:00
Matei Zaharia
d1538ebdd9
Change version in REPL
2012-10-07 10:40:29 -07:00
Matei Zaharia
f31b6f92b0
Changes to run script and README to deal with updated Scala version and
...
with Mesos being in Maven
2012-10-07 10:00:48 -07:00
Matei Zaharia
039cc6228e
Merge pull request #251 from JoshRosen/docs/internals
...
Document Dependency classes and make minor interface improvements
2012-10-07 09:56:53 -07:00
Matei Zaharia
b97bde2b62
Merge pull request #252 from pwendell/0.5-with-sonatype
...
Adding Sonatype releases to SBT.
2012-10-07 09:47:46 -07:00
Reynold Xin
933e4f5123
Merge pull request #253 from rxin/dev
...
Changed the println to logInfo in Utils.fetchFile.
2012-10-07 01:54:18 -07:00
Reynold Xin
f66c0e9561
Changed the println to logInfo in Utils.fetchFile.
2012-10-07 01:53:24 -07:00
Patrick Wendell
14719b93ff
Adding Sonatype releases to SBT.
...
This does a few things to get this branch ready for release:
1. Upgrades the sbt and Scala version
2. Sets the release number to 0.5.1
3. Adds the Sonatype publishing target
4. Installs the PGP signing plugin
5. Removes the Mesos jar dependency
2012-10-07 01:20:39 -07:00
Matei Zaharia
d72db3d7dc
Merge pull request #250 from rxin/dev
...
Fixed a bug in addFile that if the file is specified as "file:///", the symlink is created incorrectly for local mode.
2012-10-07 00:56:53 -07:00
Reynold Xin
80f59e17e2
Fixed a bug in addFile that if the file is specified as "file:///", the
...
symlink is created wrong for local mode.
2012-10-07 00:54:38 -07:00
Josh Rosen
e10308f5a0
Make ShuffleDependency.aggregator explicitly optional.
...
It was confusing to be using
new Aggregator[K, V, V](null, null, null, false)
to represent the absence of an aggregator.
2012-10-07 00:36:04 -07:00
Matei Zaharia
f930fe5d81
Improve error message
2012-10-07 07:34:36 +00:00
Matei Zaharia
a3bf0ce57f
Don't crash on ask timeout exceptions in deploy.Client.stop() (fixes a crash in tests)
2012-10-07 07:25:41 +00:00
Matei Zaharia
eca570f66a
Removed the need to sleep in tests due to waiting for Akka to shut down
2012-10-07 00:17:59 -07:00
Josh Rosen
4f72066a9a
Document the Dependency classes.
2012-10-07 00:05:37 -07:00
Josh Rosen
3f2571fe98
Remove unused isShuffle field from Dependency.
2012-10-07 00:03:55 -07:00
Matei Zaharia
b2fc3dd902
Log message
2012-10-07 06:43:52 +00:00
Matei Zaharia
ea096f7cd5
More logging
2012-10-07 06:35:48 +00:00
root
554b42cb24
Log more info in MapOutputTracker
2012-10-07 05:02:18 +00:00
root
a73b25826b
Made Akka thread pool and message batch sizes configurable
2012-10-07 04:19:54 +00:00
root
ce915cadee
Made run script add test-classes onto the classpath only if SPARK_TESTING is set; fixes #216
2012-10-07 04:19:16 +00:00
root
975009d688
Avoid acquiring locks in BlockManager when fetching shuffle outputs
2012-10-07 04:02:10 +00:00
root
0bc63f7ef1
Log initial number of fetches in reducer
2012-10-07 03:51:04 +00:00
Matei Zaharia
dc28a3ac0a
Modified shuffle to limit the maximum outstanding data size in bytes,
...
instead of the maximum number of outstanding fetches. This should make
it faster when there are many small map output files, as well as more
robust to overallocating memory on large map outputs.
2012-10-06 20:07:10 -07:00
Matei Zaharia
9a3b3f32a3
Pass sizes of map outputs back to MapOutputTracker
2012-10-06 18:46:04 -07:00
Matei Zaharia
0e42832e6a
Made block store return the size of each block put in
2012-10-06 18:00:53 -07:00
Matei Zaharia
b0110de5b6
Warn about user programs that try to set spark.cache.class
2012-10-06 17:27:14 -07:00
Matei Zaharia
65113b7e1b
Only group elements ten at a time into SequenceFile records in
...
saveAsObjectFile
2012-10-06 17:14:41 -07:00
Matei Zaharia
dbf1f3dd5b
Make reduce logging less verbose
2012-10-06 17:10:09 -07:00
Matei Zaharia
716e10ca32
Minor formatting fixes
2012-10-05 22:03:06 -07:00
Matei Zaharia
70f02fa912
Merge branch 'dev' of github.com:mesos/spark into dev
2012-10-05 22:00:22 -07:00
Matei Zaharia
95ef307ef5
Merge pull request #249 from andyk/move-rdds-to-their-own-package
...
Move RDD classes/files to their own package/directory
2012-10-05 21:59:57 -07:00
Andy Konwinski
a242cdd0a6
Factor subclasses of RDD out of RDD.scala into their own classes
...
in the rdd package.
2012-10-05 19:53:54 -07:00
Andy Konwinski
d7363a6b8a
Moves all files in core/src/main/scala/ that have RDD in their name
...
from that directory to a new core/src/main/scala/rdd directory.
2012-10-05 19:23:45 -07:00
Andy Konwinski
e0067da082
Moves all files in core/src/main/scala/ that have RDD in them from
...
package spark to package spark.rdd and updates all references to them.
2012-10-05 19:23:45 -07:00