ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1621 commits 6 branches 181 tags 397 MiB
Commit graph

89 commits

Author SHA1 Message Date
Matei Zaharia fbb3fc4143 Merge pull request #346 from JoshRosen/python-api 
Python API (PySpark)
 2013-01-12 23:49:36 -08:00
Michael Heuer 480c4139bb add repositories section to simple job pom.xml 2013-01-11 11:40:17 -06:00
Josh Rosen b57dd0f160 Add mapPartitionsWithSplit() to PySpark. 2013-01-08 16:05:02 -08:00
Josh Rosen ce9f1bbe20 Add pyspark script to replace the other scripts. 
Expand the PySpark programming guide.
 2013-01-01 21:25:49 -08:00
Josh Rosen b58340dbd9 Rename top-level 'pyspark' directory to 'python' 2013-01-01 15:05:00 -08:00
Josh Rosen 170e451fbd Minor documentation and style fixes for PySpark. 2013-01-01 13:52:14 -08:00
Josh Rosen c5cee53f20 Merge remote-tracking branch 'origin/master' into python-api 
Conflicts:
	docs/quick-start.md
 2012-12-29 16:00:51 -08:00
Josh Rosen c2b105af34 Add documentation for Python API. 2012-12-28 22:51:28 -08:00
Josh Rosen 85b8f2c64f Add epydoc API documentation for PySpark. 2012-12-27 18:04:10 -08:00
Reynold Xin c68a076037 Updated Kryo documentation for Kryo version update. 2012-12-21 16:03:17 -08:00
Josh Rosen 1948f46093 Use spark-env.sh to configure standalone master. See SPARK-638. 
Also fixed a typo in the standalone mode documentation.
 2012-12-14 01:20:00 +00:00
Patrick Wendell 6ceb559994 Adding multi-jar constructor in quickstart 2012-11-27 23:32:24 -08:00
Matei Zaharia 6adc7c965f Doc fix 2012-11-16 20:49:02 -08:00
Matei Zaharia 51477e8874 Merge pull request #294 from JoshRosen/docs/quickstart 
Fix minor typos in quickstart and Scala programming guides
 2012-10-27 16:56:39 -07:00
Josh Rosen 33bea24f8e Fix Spark groupId in Scala Programming Guide. 2012-10-26 15:01:28 -07:00
Josh Rosen c4aa10154e Fix minor typos in quick start guide. 2012-10-23 13:49:52 -07:00
Matei Zaharia 0967e71a00 Bump up version to 0.7.0-SNAPSHOT for master branch 2012-10-22 11:49:42 -07:00
Matei Zaharia 902a608187 Update version to 0.6.1-SNAPSHOT to show this is in development 2012-10-22 11:43:57 -07:00
Matei Zaharia 6999724ce8 Fix a path in the web UI 2012-10-20 23:33:37 -07:00
Patrick Wendell 7a03a0e35d Adding dependency repos in quickstart example 2012-10-14 11:48:24 -07:00
Matei Zaharia 4be12d97ec Some doc fixes, including showing version number in nav bar again 2012-10-13 19:05:11 -07:00
Matei Zaharia 19910c00c3 tweaks 2012-10-13 16:22:39 -07:00
Matei Zaharia 4a3e9cf69c Document how to configure SPARK_MEM & co on a per-job basis 2012-10-13 16:20:25 -07:00
Matei Zaharia 8d7b77bcb5 Some doc and usability improvements: 
- Added a StorageLevels class for easy access to StorageLevel constants
  in Java
- Added doc comments on Function classes in Java
- Updated Accumulator and HadoopWriter docs slightly
 2012-10-12 17:53:20 -07:00
Matei Zaharia 1183b30941 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-12 14:40:07 -07:00
Matei Zaharia 603b419fdf Tweak 2012-10-12 14:40:00 -07:00
Patrick Wendell 5788a92953 Updating Bagel build instructions 2012-10-09 22:39:29 -07:00
Patrick Wendell 0f760a0bd3 Updating programming guide with new link instructions 2012-10-09 22:39:29 -07:00
Patrick Wendell 4de5cc1ad4 Removing reference to publish-local in the quickstart 2012-10-09 22:39:29 -07:00
Patrick Wendell 8321e7f0c2 Fixing YARN instructions 2012-10-09 22:39:28 -07:00
Patrick Wendell 5013c785fd Adding SNAPSHOT to Spark version in doc config 2012-10-09 22:39:28 -07:00
Matei Zaharia bc0bc672d0 Updates to documentation: 
- Edited quick start and tuning guide to simplify them a little
- Simplified top menu bar
- Made private a SparkContext constructor parameter that was left as
  public
- Various small fixes
 2012-10-09 14:30:23 -07:00
Matei Zaharia 4780fee887 Merge pull request #260 from andyk/update-docs-to-use-version-vars 
Updates docs to use the new version num vars and adds Spark version in nav bar
 2012-10-09 09:46:20 -07:00
Andy Konwinski e1a724f39c Updating lots of docs to use the new special version number variables, 
also adding the version to the navbar so it is easy to tell which
version of Spark these docs were compiled for.
 2012-10-08 17:17:17 -07:00
Patrick Wendell 7887e171f4 Adding new download instructions 2012-10-08 17:05:13 -07:00
Andy Konwinski 56b91ab380 Updates README.md with instructions for running jekyll without building 
scaladoc (i.e. run `SKIP_SCALADOC=1 jekyll`).
 2012-10-08 12:15:44 -07:00
Andy Konwinski 89f8e1c2e7 Merge remote-tracking branch 'public-spark/dev' into add-version-vars-to-docs 
Conflicts:
	docs/quick-start.md
 2012-10-08 12:15:38 -07:00
Matei Zaharia 88152e2164 Merge pull request #254 from pwendell/quickstart-fix 
Removing one link in quickstart
 2012-10-08 11:08:29 -07:00
Andy Konwinski 45d03231d0 Adds liquid variables to docs templating system so that they can be used 
throughout the docs: SPARK_VERSION, SCALA_VERSION, and MESOS_VERSION.

To use them, e.g. use {{site.SPARK_VERSION}}.

Also removes uses of {{HOME_PATH}} which were being resolved to ""
by the templating system anyway.
 2012-10-08 10:30:38 -07:00
Andy Konwinski cd710a7718 Fixes the small gap above the nav menu dropdown boxes and the hoverable 
menu items that was causing the dropdowns to go away when the user
moved their mouse down towards them.
 2012-10-08 09:33:40 -07:00
Patrick Wendell 99aac2f6c4 Removing one link in quickstart 2012-10-08 08:53:27 -07:00
Matei Zaharia efc5423210 Made compression configurable separately for shuffle, broadcast and RDDs 2012-10-07 11:30:53 -07:00
Matei Zaharia dc28a3ac0a Modified shuffle to limit the maximum outstanding data size in bytes, 
instead of the maximum number of outstanding fetches. This should make
it faster when there are many small map output files, as well as more
robust to overallocating memory on large map outputs.
 2012-10-06 20:07:10 -07:00
Matei Zaharia e6e27a05d8 Links quick start from nav bar 2012-10-05 17:06:55 -07:00
Patrick Wendell e84c068fab Some additions to the Tuning Guide. 
1. Slight change in organization
2. Added pre-requisites
3. Made a new section about determining memory footprint
   of an RDD
4. Other small changes
 2012-10-03 14:06:34 -07:00
Matei Zaharia 42cd148507 Merge pull request #236 from pwendell/quickstart 
A Spark "Quick Start" example
 2012-10-03 08:31:43 -07:00
Patrick Wendell 35b767f478 Responding to Matei's comments 2012-10-02 23:54:03 -07:00
Shivaram Venkataraman 3d2b900b08 First cut at adding documentation for GC tuning 2012-10-02 20:07:18 -07:00
Patrick Wendell f78edf94cf A Spark "Quick Start" example 
This commit includes a quick start example that covers:
1) Basic usage of the Spark shell
2) A simple Spark job in Scala
3) A simple Spark job in Java
 2012-10-01 17:24:09 -07:00
Matei Zaharia 802aa8aef9 Some bug fixes and logging fixes for broadcast. 2012-10-01 15:20:42 -07:00