ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1231 commits 6 branches 181 tags 397 MiB
Commit graph

1231 commits

This branch
This branch
All branches
Author SHA1 Message Date
Patrick Wendell 629dd2691e Removing credentials line in build. 2012-10-14 19:33:39 -07:00
Matei Zaharia f8768da418 Comment out PGP stuff for publish-local to work 2012-10-14 17:37:21 -07:00
Matei Zaharia 1f06445b03 tweak 2012-10-14 12:04:58 -07:00
Matei Zaharia 4947bd0958 tweak 2012-10-14 12:02:58 -07:00
Matei Zaharia 6c766a9187 tweak 2012-10-14 12:02:32 -07:00
Matei Zaharia 8192fe0325 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-14 12:01:38 -07:00
Matei Zaharia 1c73d8974d Update README 2012-10-14 12:00:25 -07:00
Matei Zaharia 7855bacd26 Merge pull request #278 from pwendell/quickstart-fix 
Adding dependency repos in quickstart example
 2012-10-14 11:52:24 -07:00
Patrick Wendell 7a03a0e35d Adding dependency repos in quickstart example 2012-10-14 11:48:24 -07:00
Matei Zaharia 64dbf8d372 Made ShuffleDependency automatically find a shuffle ID for itself 2012-10-14 10:00:22 -07:00
Matei Zaharia 64b52166ee Changed default Hadoop version back to 0.20.205 2012-10-14 09:51:34 -07:00
Matei Zaharia 4be12d97ec Some doc fixes, including showing version number in nav bar again 2012-10-13 19:05:11 -07:00
Matei Zaharia 19910c00c3 tweaks 2012-10-13 16:22:39 -07:00
Matei Zaharia 4a3e9cf69c Document how to configure SPARK_MEM & co on a per-job basis 2012-10-13 16:20:25 -07:00
Matei Zaharia ce6b5a3ee5 Uncomment Maven publishing stuff and set version to 0.6.0 2012-10-13 15:55:39 -07:00
Matei Zaharia 8815aeba0c Take executor environment vars as an arguemnt to SparkContext 2012-10-13 15:31:11 -07:00
Matei Zaharia 84979499db Merge pull request #273 from dennybritz/executorVars 
Let the user specify environment variables to be passed to the Executors
 2012-10-13 14:52:14 -07:00
Denny 0700d1920a Protect from null env variables in mesos. 2012-10-13 13:57:59 -07:00
Denny 21047d923e Protect from setting null environment variables. 2012-10-13 13:44:24 -07:00
Denny fa41d50f7d Don't use system envs for Mesos. 2012-10-13 13:15:50 -07:00
Denny 67c42a41d0 Let the user specify environment variables to be passed to the Executors. 
Also removed unused variables in the ExecutorRunner.
 2012-10-13 13:08:44 -07:00
Matei Zaharia 5b7ee173e1 Update EC2 scripts for Spark 0.6 2012-10-12 19:53:03 -07:00
Matei Zaharia b4067cbad4 More doc updates, and moved Serializer to a subpackage. 2012-10-12 18:19:21 -07:00
Matei Zaharia 8d7b77bcb5 Some doc and usability improvements: 
- Added a StorageLevels class for easy access to StorageLevel constants
  in Java
- Added doc comments on Function classes in Java
- Updated Accumulator and HadoopWriter docs slightly
 2012-10-12 17:53:20 -07:00
Matei Zaharia 682b2d9329 Added a test for when an RDD only partially fits in memory 2012-10-12 14:58:26 -07:00
Matei Zaharia dca496bb77 Document cartesian() operation 2012-10-12 14:46:41 -07:00
Matei Zaharia 1183b30941 Merge branch 'dev' of github.com:mesos/spark into dev 2012-10-12 14:40:07 -07:00
Matei Zaharia 603b419fdf Tweak 2012-10-12 14:40:00 -07:00
Matei Zaharia 23015ccac0 Merge pull request #271 from shivaram/block-manager-npe-fix 
Change block manager to accept a ArrayBuffer
 2012-10-12 14:36:28 -07:00
Shivaram Venkataraman 8577523f37 Add test to verify if RDD is computed even if block manager has insufficient 
memory
 2012-10-12 14:14:57 -07:00
Matei Zaharia bd78bbb2cf Merge pull request #270 from pwendell/java-javadoc 
Adding Java documentation
 2012-10-11 12:21:47 -07:00
Patrick Wendell dc8adbd359 Adding Java documentation 2012-10-11 00:49:03 -07:00
Shivaram Venkataraman 2cf40c5fd5 Change block manager to accept a ArrayBuffer instead of an iterator to ensure 
that the computation can proceed even if we run out of memory to cache the
block. Update CacheTracker to use this new interface
 2012-10-11 00:42:46 -07:00
Matei Zaharia 4001cbdec1 Merge pull request #268 from pwendell/sonatype 
Adding code for publishing to Sonatype.
 2012-10-10 18:57:32 -07:00
Patrick Wendell 6d328f54d0 Changing tabs to spaces 2012-10-10 18:54:22 -07:00
Matei Zaharia 066e979e9e Merge pull request #267 from dennybritz/dev 
Fixed bug when fetching Jar dependencies.
 2012-10-10 18:32:23 -07:00
Patrick Wendell 3ed172ea59 Adding code for publishing to Sonatype. 
By default - I'm leaving this commented out. This is because
there is a bug in the PGP signing plugin which causes it to active
even duing a publish-local. So we'll just uncomment when we decide
to publish.
 2012-10-10 17:25:29 -07:00
Denny d3f095f904 Fixed bug when fetching Jar dependencies. 
Instead of checking currentFiles check currentJars.
 2012-10-10 16:09:53 -07:00
Matei Zaharia 1ac7bf893b Merge pull request #265 from pwendell/docs-version-config 
Making Spark version configurable in docs and updating Bagel doc
 2012-10-10 11:02:29 -07:00
Patrick Wendell 5788a92953 Updating Bagel build instructions 2012-10-09 22:39:29 -07:00
Patrick Wendell 0f760a0bd3 Updating programming guide with new link instructions 2012-10-09 22:39:29 -07:00
Patrick Wendell 4de5cc1ad4 Removing reference to publish-local in the quickstart 2012-10-09 22:39:29 -07:00
Patrick Wendell 8321e7f0c2 Fixing YARN instructions 2012-10-09 22:39:28 -07:00
Patrick Wendell 5013c785fd Adding SNAPSHOT to Spark version in doc config 2012-10-09 22:39:28 -07:00
Matei Zaharia ee2fcb2ce6 Added documentation to all the *RDDFunction classes, and moved them into 
the spark package to make them more visible. Also documented various
other miscellaneous things in the API.
 2012-10-09 18:38:36 -07:00
Matei Zaharia bc0bc672d0 Updates to documentation: 
- Edited quick start and tuning guide to simplify them a little
- Simplified top menu bar
- Made private a SparkContext constructor parameter that was left as
  public
- Various small fixes
 2012-10-09 14:30:23 -07:00
Matei Zaharia ad28aebb0a Merge pull request #262 from andyk/document-public-apis 
Document RDD api (i.e. RDD.scala)
 2012-10-09 09:47:58 -07:00
Matei Zaharia 4780fee887 Merge pull request #260 from andyk/update-docs-to-use-version-vars 
Updates docs to use the new version num vars and adds Spark version in nav bar
 2012-10-09 09:46:20 -07:00
Matei Zaharia d3b8252050 Merge pull request #263 from RayRacine/m1.medium 
Add m1.medium node option to cluster management script.
 2012-10-09 09:44:58 -07:00
Ray Racine c52bd1b6b1 Add m1.medium node option to cluster management script. 2012-10-09 11:05:13 -04:00