ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4013 commits 6 branches 181 tags 397 MiB
Commit graph

4013 commits

This branch
This branch
All branches
Author SHA1 Message Date
Jey Kottalam 70661246fd Fix YARN assembly generation under Maven 2013-09-06 11:31:16 -07:00
Jey Kottalam 35ed09f1d1 Clarify YARN example 2013-09-06 11:31:16 -07:00
Reynold Xin 1e15feb5a3 Hot fix to resolve the compilation error caused by SPARK-821. 2013-09-06 22:44:05 +08:00
Patrick Wendell ddcb9d310a Merge pull request #895 from ilikerps/821 
SPARK-821: Don't cache results when action run locally on driver
 2013-09-05 23:54:09 -07:00
Aaron Davidson a63d4c7dc2 SPARK-660: Add StorageLevel support in Python 
It uses reflection... I am not proud of that fact, but it at least ensures
compatibility (sans refactoring of the StorageLevel stuff).
 2013-09-05 23:36:27 -07:00
Aaron Davidson 3a04e76c89 Reynold's second round of comments 2013-09-05 21:43:26 -07:00
Ameet Talwalkar d52edfa753 updated content 2013-09-05 21:06:50 -07:00
Matei Zaharia 699c331f2f Merge pull request #891 from xiajunluan/SPARK-864 
[SPARK-864]DAGScheduler Exception if we delete Worker and StandaloneExecutorBackend then add Worker
 2013-09-05 20:21:53 -07:00
Aaron Davidson 4f2236a1c5 Add unit test and address comments 2013-09-05 18:06:30 -07:00
Aaron Davidson 1418d18af4 SPARK-821: Don't cache results when action run locally on driver 
Caching the results of local actions (e.g., rdd.first()) causes the driver to
store entire partitions in its own memory, which may be highly constrained.
This patch simply makes the CacheManager avoid caching the result of all locally-run computations.
 2013-09-05 15:34:42 -07:00
Andrew xia 7c15e3c5de Fix bug SPARK-864 2013-09-05 15:56:11 +08:00
Patrick Wendell 5c7494d7c1 Merge pull request #893 from ilikerps/master 
SPARK-884: Add unit test to validate Spark JSON output
 2013-09-04 22:47:03 -07:00
Aaron Davidson 714e7f9e32 Fix line over 100 chars 2013-09-04 22:40:08 -07:00
Aaron Davidson 37db141aef Address Patrick's comments 2013-09-04 21:34:20 -07:00
Matei Zaharia a54786678f Merge pull request #894 from c0s/master 
Updating assembly README to reflect recent changes in the build.
 2013-09-04 21:11:56 -07:00
Konstantin Boudnik 7c7c7e10ca Updating assembly README to reflect recent changes in the build. 2013-09-04 20:54:35 -07:00
Aaron Davidson 9e6f2b6822 SPARK-884: Add unit test to validate Spark JSON output 
This unit test simply validates that the outputs of
the JsonProtocol methods are syntactically valid JSON.
 2013-09-04 15:26:46 -07:00
Mridul Muralidharan 1e2474b814 Address review comments - rename toHash to nonNegativeHash 2013-09-04 07:46:46 +05:30
Mridul Muralidharan b3a82b7df3 Fix hash bug - caused failure after 35k stages, sigh 2013-09-04 07:02:25 +05:30
Patrick Wendell c592a3c9b9 Minor spacing fix 2013-09-03 14:39:11 -07:00
Patrick Wendell 19f70273d2 Merge pull request #878 from tgravescs/yarnUILink 
Link the Spark UI up to the Yarn UI
 2013-09-03 14:29:10 -07:00
Matei Zaharia 68df2464d1 Merge pull request #889 from alig/master 
Return the port the WebUI is bound to (useful if port 0 was used)
 2013-09-03 13:01:17 -07:00
Y.CORP.YAHOO.COM\tgraves 41c1b5b9a0 Update based on review comments. Change function to prependBaseUri and fix formatting. 2013-09-03 14:46:51 -05:00
Y.CORP.YAHOO.COM\tgraves c8cc276110 Review comment changes and update to org.apache packaging 2013-09-03 10:50:21 -05:00
Y.CORP.YAHOO.COM\tgraves 547fc4a412 Merge remote-tracking branch 'mesos/master' into yarnUILink 
Conflicts:
	core/src/main/scala/org/apache/spark/ui/UIUtils.scala
	core/src/main/scala/org/apache/spark/ui/jobs/PoolTable.scala
	core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
	docs/running-on-yarn.md
 2013-09-03 08:36:59 -05:00
Ali Ghodsi b25918d841 Merge branch 'master' of https://github.com/alig/spark 
Conflicts:
	core/src/main/scala/org/apache/spark/deploy/master/Master.scala
 2013-09-03 00:56:12 -07:00
Ali Ghodsi bd0788505f Using configured akka timeouts 2013-09-03 00:50:35 -07:00
Ali Ghodsi cbfef9b3ff Sort order of imports to match project guidelines 2013-09-02 19:33:55 -07:00
Ali Ghodsi 36d8fca2cc Reynold's comment fixed 2013-09-02 19:31:09 -07:00
Ali Ghodsi e452bd6d77 Brushing the code up slightly 2013-09-02 19:04:08 -07:00
Matei Zaharia 59218bdd49 Add Apache parent POM 2013-09-02 18:34:03 -07:00
Ali Ghodsi cf7b115496 Enabling getting the actual WEBUI port 2013-09-02 18:21:21 -07:00
Matei Zaharia d3dd48f8b3 Merge pull request #887 from mateiz/misc-fixes 
Miscellaneous fixes for 0.8
 2013-09-02 16:44:54 -07:00
Matei Zaharia f586c8ef38 Updated LICENSE with third-party licenses 2013-09-02 16:43:06 -07:00
Matei Zaharia 4cc27d962b Add Apache incubator notice to README 2013-09-02 14:34:09 -07:00
Matei Zaharia 2615cad30b Some doc improvements 
- List higher-level projects that run on Spark
- Tweak CSS
 2013-09-02 13:35:28 -07:00
Matei Zaharia 12b2f1f9c9 Add missing license headers found with RAT 2013-09-02 12:23:03 -07:00
Matei Zaharia 2ba695292a Exclude some private modules in epydoc 2013-09-02 12:22:52 -07:00
Matei Zaharia 636fc0c89e Merge pull request #886 from mateiz/codec 
Fix spark.io.compression.codec and change default codec to LZF
 2013-09-02 11:20:39 -07:00
Matei Zaharia 246bf67f58 Fix test 2013-09-02 10:57:34 -07:00
Matei Zaharia 9329a7d4cd Fix spark.io.compression.codec and change default codec to LZF 2013-09-02 10:15:22 -07:00
Matei Zaharia d9a53b94f8 Merge pull request #885 from mateiz/win-py 
Allow PySpark to run on Windows
 2013-09-01 22:12:30 -07:00
Matei Zaharia 9ee1e9db2e Doc improvements 2013-09-01 22:12:03 -07:00
Matei Zaharia 141f54279e Further fixes to get PySpark to work on Windows 2013-09-02 01:19:29 +00:00
Matei Zaharia 6550e5e60c Allow PySpark to launch worker.py directly on Windows 2013-09-01 18:06:15 -07:00
Matei Zaharia 3c520fea77 Merge pull request #884 from mateiz/win-fixes 
Run script fixes for Windows after package & assembly change
 2013-09-01 17:26:55 -07:00
Matei Zaharia 3db404a43a Run script fixes for Windows after package & assembly change 2013-09-01 23:45:57 +00:00
Matei Zaharia f957c26fa2 Merge pull request #882 from mateiz/package-rename 
Rename spark package to org.apache.spark
 2013-09-01 14:53:57 -07:00
Matei Zaharia 0a8cc30921 Move some classes to more appropriate packages: 
* RDD, *RDDFunctions -> org.apache.spark.rdd
* Utils, ClosureCleaner, SizeEstimator -> org.apache.spark.util
* JavaSerializer, KryoSerializer -> org.apache.spark.serializer
 2013-09-01 14:13:16 -07:00
Matei Zaharia 5b4dea2143 More fixes 2013-09-01 14:13:16 -07:00