ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
16780 commits 6 branches 181 tags 397 MiB
Commit graph

1588 commits

Author SHA1 Message Date
Tathagata Das 9e644402c1 Improved jekyll and scala docs. Made many classes and method private to remove them from scala docs. 2012-12-29 18:31:51 -08:00
Josh Rosen c5cee53f20 Merge remote-tracking branch 'origin/master' into python-api 
Conflicts:
	docs/quick-start.md
 2012-12-29 16:00:51 -08:00
Josh Rosen c2b105af34 Add documentation for Python API. 2012-12-28 22:51:28 -08:00
Josh Rosen 85b8f2c64f Add epydoc API documentation for PySpark. 2012-12-27 18:04:10 -08:00
Tathagata Das 7c33f76291 Merge branch 'mesos' into dev-merge 2012-12-26 19:19:07 -08:00
Reynold Xin c68a076037 Updated Kryo documentation for Kryo version update. 2012-12-21 16:03:17 -08:00
Reynold Xin eac566a7f4 Merge branch 'master' of github.com:mesos/spark into dev 
Conflicts:
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/PairRDDFunctions.scala
	core/src/main/scala/spark/ParallelCollection.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/CartesianRDD.scala
	core/src/main/scala/spark/rdd/CoGroupedRDD.scala
	core/src/main/scala/spark/rdd/CoalescedRDD.scala
	core/src/main/scala/spark/rdd/FilteredRDD.scala
	core/src/main/scala/spark/rdd/FlatMappedRDD.scala
	core/src/main/scala/spark/rdd/GlommedRDD.scala
	core/src/main/scala/spark/rdd/HadoopRDD.scala
	core/src/main/scala/spark/rdd/MapPartitionsRDD.scala
	core/src/main/scala/spark/rdd/MapPartitionsWithSplitRDD.scala
	core/src/main/scala/spark/rdd/MappedRDD.scala
	core/src/main/scala/spark/rdd/PipedRDD.scala
	core/src/main/scala/spark/rdd/SampledRDD.scala
	core/src/main/scala/spark/rdd/ShuffledRDD.scala
	core/src/main/scala/spark/rdd/UnionRDD.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockManagerId.scala
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/main/scala/spark/storage/StorageLevel.scala
	core/src/main/scala/spark/util/MetadataCleaner.scala
	core/src/main/scala/spark/util/TimeStampedHashMap.scala
	core/src/test/scala/spark/storage/BlockManagerSuite.scala
	run
 2012-12-20 14:53:40 -08:00
Josh Rosen 1948f46093 Use spark-env.sh to configure standalone master. See SPARK-638. 
Also fixed a typo in the standalone mode documentation.
 2012-12-14 01:20:00 +00:00
Patrick Wendell 6ceb559994 Adding multi-jar constructor in quickstart 2012-11-27 23:32:24 -08:00
Tathagata Das 0fe2fc4d5e Merged branch mesos/master to branch dev. 2012-11-26 13:16:59 -08:00
Matei Zaharia 6adc7c965f Doc fix 2012-11-16 20:49:02 -08:00
Patrick Wendell d39ac5fbc1 Streaming programming guide. STREAMING-2 #resolve 2012-11-13 21:19:58 -08:00
Matei Zaharia 51477e8874 Merge pull request #294 from JoshRosen/docs/quickstart 
Fix minor typos in quickstart and Scala programming guides
 2012-10-27 16:56:39 -07:00
Josh Rosen 33bea24f8e Fix Spark groupId in Scala Programming Guide. 2012-10-26 15:01:28 -07:00
Josh Rosen c4aa10154e Fix minor typos in quick start guide. 2012-10-23 13:49:52 -07:00
Matei Zaharia 0967e71a00 Bump up version to 0.7.0-SNAPSHOT for master branch 2012-10-22 11:49:42 -07:00
Matei Zaharia 902a608187 Update version to 0.6.1-SNAPSHOT to show this is in development 2012-10-22 11:43:57 -07:00
Matei Zaharia 6999724ce8 Fix a path in the web UI 2012-10-20 23:33:37 -07:00
Patrick Wendell 7a03a0e35d Adding dependency repos in quickstart example 2012-10-14 11:48:24 -07:00
Matei Zaharia 4be12d97ec Some doc fixes, including showing version number in nav bar again 2012-10-13 19:05:11 -07:00
Matei Zaharia 19910c00c3 tweaks 2012-10-13 16:22:39 -07:00
Matei Zaharia 4a3e9cf69c Document how to configure SPARK_MEM & co on a per-job basis 2012-10-13 16:20:25 -07:00
Matei Zaharia 8d7b77bcb5 Some doc and usability improvements: 
- Added a StorageLevels class for easy access to StorageLevel constants
  in Java
- Added doc comments on Function classes in Java
- Updated Accumulator and HadoopWriter docs slightly
 2012-10-12 17:53:20 -07:00
Matei Zaharia 1183b30941 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-12 14:40:07 -07:00
Matei Zaharia 603b419fdf Tweak 2012-10-12 14:40:00 -07:00
Patrick Wendell 5788a92953 Updating Bagel build instructions 2012-10-09 22:39:29 -07:00
Patrick Wendell 0f760a0bd3 Updating programming guide with new link instructions 2012-10-09 22:39:29 -07:00
Patrick Wendell 4de5cc1ad4 Removing reference to publish-local in the quickstart 2012-10-09 22:39:29 -07:00
Patrick Wendell 8321e7f0c2 Fixing YARN instructions 2012-10-09 22:39:28 -07:00
Patrick Wendell 5013c785fd Adding SNAPSHOT to Spark version in doc config 2012-10-09 22:39:28 -07:00
Matei Zaharia bc0bc672d0 Updates to documentation: 
- Edited quick start and tuning guide to simplify them a little
- Simplified top menu bar
- Made private a SparkContext constructor parameter that was left as
  public
- Various small fixes
 2012-10-09 14:30:23 -07:00
Matei Zaharia 4780fee887 Merge pull request #260 from andyk/update-docs-to-use-version-vars 
Updates docs to use the new version num vars and adds Spark version in nav bar
 2012-10-09 09:46:20 -07:00
Andy Konwinski e1a724f39c Updating lots of docs to use the new special version number variables, 
also adding the version to the navbar so it is easy to tell which
version of Spark these docs were compiled for.
 2012-10-08 17:17:17 -07:00
Patrick Wendell 7887e171f4 Adding new download instructions 2012-10-08 17:05:13 -07:00
Andy Konwinski 56b91ab380 Updates README.md with instructions for running jekyll without building 
scaladoc (i.e. run `SKIP_SCALADOC=1 jekyll`).
 2012-10-08 12:15:44 -07:00
Andy Konwinski 89f8e1c2e7 Merge remote-tracking branch 'public-spark/dev' into add-version-vars-to-docs 
Conflicts:
	docs/quick-start.md
 2012-10-08 12:15:38 -07:00
Matei Zaharia 88152e2164 Merge pull request #254 from pwendell/quickstart-fix 
Removing one link in quickstart
 2012-10-08 11:08:29 -07:00
Andy Konwinski 45d03231d0 Adds liquid variables to docs templating system so that they can be used 
throughout the docs: SPARK_VERSION, SCALA_VERSION, and MESOS_VERSION.

To use them, e.g. use {{site.SPARK_VERSION}}.

Also removes uses of {{HOME_PATH}} which were being resolved to ""
by the templating system anyway.
 2012-10-08 10:30:38 -07:00
Andy Konwinski cd710a7718 Fixes the small gap above the nav menu dropdown boxes and the hoverable 
menu items that was causing the dropdowns to go away when the user
moved their mouse down towards them.
 2012-10-08 09:33:40 -07:00
Patrick Wendell 99aac2f6c4 Removing one link in quickstart 2012-10-08 08:53:27 -07:00
Matei Zaharia efc5423210 Made compression configurable separately for shuffle, broadcast and RDDs 2012-10-07 11:30:53 -07:00
Matei Zaharia dc28a3ac0a Modified shuffle to limit the maximum outstanding data size in bytes, 
instead of the maximum number of outstanding fetches. This should make
it faster when there are many small map output files, as well as more
robust to overallocating memory on large map outputs.
 2012-10-06 20:07:10 -07:00
Matei Zaharia e6e27a05d8 Links quick start from nav bar 2012-10-05 17:06:55 -07:00
Patrick Wendell e84c068fab Some additions to the Tuning Guide. 
1. Slight change in organization
2. Added pre-requisites
3. Made a new section about determining memory footprint
   of an RDD
4. Other small changes
 2012-10-03 14:06:34 -07:00
Matei Zaharia 42cd148507 Merge pull request #236 from pwendell/quickstart 
A Spark "Quick Start" example
 2012-10-03 08:31:43 -07:00
Patrick Wendell 35b767f478 Responding to Matei's comments 2012-10-02 23:54:03 -07:00
Shivaram Venkataraman 3d2b900b08 First cut at adding documentation for GC tuning 2012-10-02 20:07:18 -07:00
Patrick Wendell f78edf94cf A Spark "Quick Start" example 
This commit includes a quick start example that covers:
1) Basic usage of the Spark shell
2) A simple Spark job in Scala
3) A simple Spark job in Java
 2012-10-01 17:24:09 -07:00
Matei Zaharia 802aa8aef9 Some bug fixes and logging fixes for broadcast. 2012-10-01 15:20:42 -07:00
Reynold Xin f5812d0354 Added mapPartitionsWithSplit to the programming guide. 2012-09-29 01:31:36 -07:00
Josh Rosen 37c199bbb0 Allow controlling number of splits in distinct(). 2012-09-28 23:44:19 -07:00
Matei Zaharia 009b0e37e7 Added an option to compress blocks in the block store 2012-09-27 18:45:44 -07:00
Matei Zaharia 7bcb08cef5 Renamed storage levels to something cleaner; fixes #223. 2012-09-27 17:50:59 -07:00
Matei Zaharia bf18e0994e Minor typos 2012-09-26 23:53:38 -07:00
Matei Zaharia a4093f7563 Minor doc fixes 2012-09-26 23:22:15 -07:00
Matei Zaharia ea05fc130b Updates to standalone cluster, web UI and deploy docs. 2012-09-26 22:54:39 -07:00
Matei Zaharia 874a9fd407 More updates to docs, including tuning guide 2012-09-26 19:17:58 -07:00
Matei Zaharia 58eb44acbb Doc tweaks 2012-09-26 00:32:59 -07:00
Matei Zaharia d51d5e0582 Doc fixes 2012-09-25 23:59:04 -07:00
Matei Zaharia c5754bb939 Fixes to Java guide 2012-09-25 23:51:04 -07:00
Matei Zaharia f1246cc7c1 Various enhancements to the programming guide and HTML/CSS 2012-09-25 23:26:56 -07:00
Matei Zaharia 56c90485fd More updates to documentation 2012-09-25 19:31:07 -07:00
Matei Zaharia 1821bf1f1f Merge branch 'dev' of github.com:mesos/spark into dev 2012-09-25 15:46:27 -07:00
Matei Zaharia e47e11720f Documentation updates 2012-09-25 15:46:18 -07:00
Andy Konwinski 098351b735 Makes nav menu dropdowns show on hover instead of on click. 2012-09-25 15:42:50 -07:00
Matei Zaharia 30362a21e7 Update license info on deploy scripts 2012-09-25 14:43:47 -07:00
Matei Zaharia 60bce9574f Update Jekyll plugin to look at Scala 2.9.2 docs 2012-09-25 14:31:53 -07:00
Josh Rosen c94e9cc54a Add Java Programming Guide; fix broken doc links. 2012-09-16 20:46:46 -07:00
Andy Konwinski 52c29071a4 - Add docs/api to .gitignore 
- Rework/expand the nav bar with more of the docs site
- Removing parts of docs about EC2 and Mesos that differentiate between
  running 0.5 and before
    - Merged subheadings from running-on-amazon-ec2.html that are still relevant
      (i.e., "Using a newer version of Spark" and "Accessing Data in S3") into
      ec2-scripts.html and deleted running-on-amazon-ec2.html
- Added some TODO comments to a few docs
- Updated the blurb about AMP Camp
- Renamed programming-guide to spark-programming-guide
- Fixing typos/etc. in Standalone Spark doc
 2012-09-16 15:28:52 -07:00
Andy Konwinski 6765d9727e Adds a jekyll plugin (written in Ruby) to the _plugins directory 
which generates scala doc by calling `sbt/sbt doc`, copies it over
to docs, and updates the links from the api webpage to now point to
the copied over scaladoc (making the _site directory easy to just
copy over to a public website).
 2012-09-13 17:17:58 -07:00
Andy Konwinski 462cd8be60 Re-enabling responsive for better looking padding and more sane resizing, 
but removed the collapsable stuff from the nav bar.
 2012-09-13 15:39:35 -07:00
Andy Konwinski 5ec7a6665b More crisp logo created from vector source (ai) and disabled 
responsive css (so nav menu doesn't switch to collapsed version
for narrow viewports).
 2012-09-13 15:27:33 -07:00
Andy Konwinski b0207e2bfd Replaces "Spark" word in nav bar with logo. 2012-09-13 12:08:12 -07:00
Andy Konwinski 130f2f2f41 Merge remote-tracking branch 'public-spark/dev' into doc 2012-09-13 11:29:25 -07:00
Denny 638a8511fa Fixed navbar style problem 2012-09-13 09:56:27 -07:00
Denny 6d53b971b9 Added standalone and YARN docs. Merged standalone cluster into standalone doc 2012-09-13 09:47:54 -07:00
Andy Konwinski ca2c999e0f Making the link to api scaladocs work and migrating other code snippets 
to use pygments syntax highlighting.
 2012-09-12 23:25:07 -07:00
Andy Konwinski c4db09ea76 Adds ec2-scripts.md back (it was mistakenly removed earlier due to 
git weirdness).
 2012-09-12 20:56:32 -07:00
Matei Zaharia d3db46fdef Remove title from content in Bagel 2012-09-12 19:50:38 -07:00
Matei Zaharia 88181bee9e Small tweaks to generated doc pages 2012-09-12 19:47:31 -07:00
Andy Konwinski 1bcd09e093 Merge remote-tracking branch 'public-spark/dev' into doc 2012-09-12 19:28:55 -07:00
Andy Konwinski 35adccd008 Adds syntax highlighting (via pygments), and some style tweaks to make things 
easier to read.
 2012-09-12 19:27:44 -07:00
Reynold Xin 4b8b0d5c08 Added a link to AMPCamp website in Spark docs. 2012-09-12 17:54:16 -07:00
Andy Konwinski 95c5a376b5 Fixing a hanging sentence in docs/ec2-scripts.md 2012-09-12 16:17:24 -07:00
Andy Konwinski 3c7e40fa27 Removed the upper-case version of docs/EC2-Scripts.md. 2012-09-12 16:09:34 -07:00
Andy Konwinski 4d3a17c8d7 Fixing lots of broken links. 2012-09-12 16:06:18 -07:00
Andy Konwinski 49e98500a9 Updated base README to point to documentation site instead of wiki, updated 
docs/README.md to describe use of Jekyll, and renmaed things to make them
more consistent with the lower-case-with-hyphens convention.
 2012-09-12 13:03:43 -07:00
Andy Konwinski 16da942d66 Adding docs directory containing documentation currently on the wiki 
which can be compiled via jekyll, using the command `jekyll`. To compile
and run a local webserver to serve the doc as a website, run
`jekyll --server`.
 2012-09-12 13:03:43 -07:00