ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3913 commits 6 branches 181 tags 397 MiB
Commit graph

3913 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 0a8cc30921 Move some classes to more appropriate packages: 
* RDD, *RDDFunctions -> org.apache.spark.rdd
* Utils, ClosureCleaner, SizeEstimator -> org.apache.spark.util
* JavaSerializer, KryoSerializer -> org.apache.spark.serializer
 2013-09-01 14:13:16 -07:00
Matei Zaharia 5b4dea2143 More fixes 2013-09-01 14:13:16 -07:00
Matei Zaharia 5701eb92c7 Fix some URLs 2013-09-01 14:13:16 -07:00
Matei Zaharia debcf24389 Fix over-zealous find-and-replace in HTML 2013-09-01 14:13:16 -07:00
Matei Zaharia d27cd03f30 Fix more URLs in docs 2013-09-01 14:13:16 -07:00
Matei Zaharia bbaa9d7d6e Add banner to PySpark and make wordcount output nicer 2013-09-01 14:13:16 -07:00
Matei Zaharia 12495ec63a Remove shutdown hook to stop jetty; this is unnecessary for releasing 
ports and creates noisy log messages
 2013-09-01 14:13:15 -07:00
Matei Zaharia 4f422032e5 Update docs for new package 2013-09-01 14:13:15 -07:00
Matei Zaharia 4d1cb59fe1 Small tweak to docs gradient 2013-09-01 14:13:15 -07:00
Matei Zaharia 46eecd110a Initial work to rename package to org.apache.spark 2013-09-01 14:13:13 -07:00
Matei Zaharia a30fac16ca Merge pull request #883 from alig/master 
Don't require the spark home environment variable to be set for standalone mode (change needed by SIMR)
 2013-09-01 12:27:50 -07:00
Matei Zaharia 03cc76506a Merge pull request #881 from pwendell/master 
Extend QuickStart to include next steps
 2013-09-01 10:20:56 -07:00
Patrick Wendell 0e375a3cc2 Add assmebly plug in links 2013-09-01 09:43:42 -07:00
Patrick Wendell 6371febe18 Better docs 2013-08-31 19:09:06 -07:00
Matei Zaharia 0e9565a704 Merge pull request #880 from mateiz/ui-tweaks 
Various UI tweaks
 2013-08-31 18:55:41 -07:00
Matei Zaharia 2c5a4b89ee Small fixes to README 2013-08-31 18:08:05 -07:00
Matei Zaharia 2b29a1d43f Merge pull request #877 from mateiz/docs 
Doc improvements for 0.8
 2013-08-31 17:49:45 -07:00
Matei Zaharia e34bc3a8ee Small tweak 2013-08-31 17:47:15 -07:00
Matei Zaharia 7862c4a3c8 Another fix suggested by Patrick 2013-08-31 17:41:47 -07:00
Matei Zaharia 9ddad0dcb4 Fixes suggested by Patrick 2013-08-31 17:40:33 -07:00
Matei Zaharia 2ee6a7e32a Print output from spark-daemon only when it fails to launch 2013-08-31 17:31:07 -07:00
Ali Ghodsi 250bddc255 Don't require spark home to be set for standalone mode 2013-08-31 17:29:05 -07:00
Matei Zaharia 25ac50668b Various web UI improvements: 
- Use "fluid" layout that can expand to wide browser windows, instead of
  the old one's limit of 1200 px
- Remove unnecessary <hr> elements
- Switch back to Bootstrap's default theme and tweak progress bar colors
- Make headers more consistent between deploy and app UIs
- Replace some inline CSS with stylesheets
 2013-08-31 16:55:40 -07:00
Matei Zaharia 89a20b83e9 Delete some code that was added back in a merge and print less info in 
spark-daemon
 2013-08-31 16:55:25 -07:00
Matei Zaharia 4819baa658 More updates, describing changes to recommended use of environment vars 
and new Python stuff
 2013-08-31 14:21:10 -07:00
Matei Zaharia 6edef9c833 Merge pull request #861 from AndreSchumacher/pyspark_sampling_function 
Pyspark sampling function
 2013-08-31 13:39:24 -07:00
Matei Zaharia fd89835965 Merge pull request #870 from JoshRosen/spark-885 
Don't send SIGINT / ctrl-c to Py4J gateway subprocess
 2013-08-31 13:18:12 -07:00
Matei Zaharia 618f0ecb43 Merge pull request #869 from AndreSchumacher/subtract 
PySpark: implementing subtractByKey(), subtract() and keyBy()
 2013-08-30 18:17:13 -07:00
Matei Zaharia 4293533032 Update docs about HDFS versions 2013-08-30 15:04:43 -07:00
Matei Zaharia f3a964848d More doc improvements + better warnings when you haven't built Spark 2013-08-30 12:41:25 -07:00
Reynold Xin 94bb7fd46e Merge pull request #876 from mbautin/master_hadoop_rdd_conf 
Make HadoopRDD's configuration accessible
 2013-08-30 12:05:13 -07:00
Mikhail Bautin 35090958b3 Also add getConf to NewHadoopRDD 2013-08-30 11:03:57 -07:00
Mikhail Bautin 5e30172f70 Make HadoopRDD's configuration accessible 2013-08-30 11:01:06 -07:00
Matei Zaharia 23762efda2 New hardware provisioning doc, and updates to menus 2013-08-30 10:16:26 -07:00
Matei Zaharia 1b0f69c623 Change docs color theme for 0.8 2013-08-30 10:15:58 -07:00
Reynold Xin 9e17e456d2 Merge pull request #875 from shivaram/build-fix 
Fix broken build by removing addIntercept
 2013-08-30 00:22:53 -07:00
Shivaram Venkataraman adc700582b Fix broken build by removing addIntercept 2013-08-30 00:16:32 -07:00
Evan Sparks 016787de32 Merge pull request #863 from shivaram/etrain-ridge 
Adding linear regression and refactoring Ridge regression to use SGD
 2013-08-29 22:15:14 -07:00
Evan Sparks 852d810787 Merge pull request #819 from shivaram/sgd-cleanup 
Change SVM to use {0,1} labels
 2013-08-29 22:13:15 -07:00
Matei Zaharia ca71620950 Merge pull request #857 from mateiz/assembly 
Change build and run instructions to use assemblies
 2013-08-29 21:51:14 -07:00
Reynold Xin 1528776628 Merge pull request #874 from jerryshao/fix-report-bug 
Fix removed block zero size log reporting
 2013-08-29 21:30:47 -07:00
Matei Zaharia e11bc18294 Update Maven docs 2013-08-29 21:19:07 -07:00
Matei Zaharia d8a4008685 Fix path to assembly in make-distribution.sh 2013-08-29 21:19:07 -07:00
Matei Zaharia 2de756ff19 Update some build instructions because only sbt assembly and mvn package 
are now needed
 2013-08-29 21:19:06 -07:00
Matei Zaharia 666d93c294 Update Maven build to create assemblies expected by new scripts 
This includes the following changes:
- The "assembly" package now builds in Maven by default, and creates an
  assembly containing both hadoop-client and Spark, unlike the old
  BigTop distribution assembly that skipped hadoop-client
- There is now a bigtop-dist package to build the old BigTop assembly
- The repl-bin package is no longer built by default since the scripts
  don't reply on it; instead it can be enabled with -Prepl-bin
- Py4J is now included in the assembly/lib folder as a local Maven repo,
  so that the Maven package can link to it
- run-example now adds the original Spark classpath as well because the
  Maven examples assembly lists spark-core and such as provided
- The various Maven projects add a spark-yarn dependency correctly
 2013-08-29 21:19:06 -07:00
Matei Zaharia d7dec938e5 Don't use SPARK_LAUNCH_WITH_SCALA in pyspark 2013-08-29 21:19:06 -07:00
Matei Zaharia 3ff105f87d Find assembly correctly in pyspark 2013-08-29 21:19:06 -07:00
Matei Zaharia aab345c463 Fix finding of assembly JAR, as well as some pointers to ./run 2013-08-29 21:19:06 -07:00
Matei Zaharia 8d81358a05 Provide more memory for tests 2013-08-29 21:19:06 -07:00
Matei Zaharia ab0e625d9e Fix PySpark for assembly run and include it in dist 2013-08-29 21:19:06 -07:00