ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3608 commits 6 branches 181 tags 397 MiB
Commit graph

162 commits

Author SHA1 Message Date
Evan Sparks ff9ebfabb4 Merge pull request #762 from shivaram/sgd-cleanup 
Refactor SGD options into a new class.
 2013-08-11 10:52:55 -07:00
Alexander Pivovarov 2d97cc46af Fixed path to JavaALS.java and JavaKMeans.java, fixed hadoop2-yarn profile 2013-08-10 23:04:50 -07:00
Matei Zaharia 4c4f769187 Optimize Scala PageRank to use reduceByKey 2013-08-10 18:09:54 -07:00
Matei Zaharia 06e4f2a8f2 Merge pull request #789 from MLnick/master 
Adding Scala version of PageRank example
 2013-08-10 18:06:23 -07:00
Matei Zaharia cd247ba5bb Merge pull request #786 from shivaram/mllib-java 
Java fixes, tests and examples for ALS, KMeans
 2013-08-09 20:41:13 -07:00
Matei Zaharia 06303a62e5 Optimize JavaPageRank to use reduceByKey instead of groupByKey 2013-08-08 18:50:00 -07:00
Shivaram Venkataraman 2812e72200 Add setters for optimizer, gradient in SGD. 
Also remove java-specific constructor for LabeledPoint.
 2013-08-08 16:24:31 -07:00
Shivaram Venkataraman e1a209f791 Remove Java-specific constructor for Rating. 
The scala constructor works for native type java types. Modify examples
to match this.
 2013-08-08 14:36:02 -07:00
Nick Pentreath c4eea875ac Style changes as per Matei's comments 2013-08-08 12:40:37 +02:00
Nick Pentreath cce758b893 Adding Scala version of PageRank example 2013-08-07 16:38:52 +02:00
Shivaram Venkataraman 338b7a7455 Merge branch 'master' of git://github.com/mesos/spark into sgd-cleanup 
Conflicts:
	mllib/src/main/scala/spark/mllib/util/MLUtils.scala
 2013-08-06 21:21:55 -07:00
Shivaram Venkataraman 7db69d56f2 Refactor GLM algorithms and add Java tests 
This change adds Java examples and unit tests for all GLM algorithms
to make sure the MLLib interface works from Java. Changes include
- Introduce LabeledPoint and avoid using Doubles in train arguments
- Rename train to run in class methods
- Make the optimizer a member variable of GLM to make sure the builder
  pattern works
 2013-08-06 17:23:22 -07:00
Shivaram Venkataraman 471fbadd0c Java examples, tests for KMeans and ALS 
- Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it
  easier to call from Java
- Renames class methods from `train` to `run` to enable static methods to be
  called from Java.
- Add unit tests which check if both static / class methods can be called.
- Also add examples which port the main() function in ALS, KMeans to the
  examples project.

Couple of minor changes to existing code:
- Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily
- Workaround a bug where using double[] from Java leads to class cast exception in
  KMeans init
 2013-08-06 15:43:46 -07:00
stayhf 882baee489 Got rid of unnecessary map function 2013-08-06 21:34:39 +00:00
stayhf 326a7a82e0 changes as reviewer requested 2013-08-06 21:03:24 +00:00
stayhf 98fd62605d Updated code with reviewer's suggestions 2013-08-05 00:30:28 +00:00
stayhf a682637301 Simple PageRank algorithm implementation in Java for SPARK-760 2013-08-03 06:01:16 +00:00
Matei Zaharia af3c9d5042 Add Apache license headers and LICENSE and NOTICE files 2013-07-16 17:21:33 -07:00
Mark Hamstra 0b39d66f3f pom cleanup 2013-07-08 16:07:09 -07:00
Mark Hamstra afdaf430bd Explicit dependencies for scala-library and scalap to prevent 2.9.2 vs. 2.9.3 problems 2013-07-08 15:40:50 -07:00
Konstantin Boudnik 6fdbc68f2c Fixing missed hbase dependency in examples hadoop2-yarn profile 2013-07-01 17:45:07 -07:00
Matei Zaharia ccfe953a4d Merge pull request #577 from skumargithub/master 
Example of cumulative counting using updateStateByKey
 2013-06-29 17:57:53 -07:00
Matei Zaharia 1667158544 Merge remote-tracking branch 'mrpotes/master' 2013-06-29 14:36:09 -07:00
James Phillpotts 176193b1e8 Fix usage and parameter extraction 2013-06-25 23:06:15 +01:00
James Phillpotts 366572edca Include a default OAuth implementation, and update examples and JavaStreamingContext 2013-06-25 22:59:34 +01:00
Tathagata Das c89af0a7f9 Merge branch 'master' into streaming 
Conflicts:
	.gitignore
 2013-06-24 23:57:47 -07:00
Matei Zaharia dbfab49d2a Merge remote-tracking branch 'milliondreams/casdemo' 
Conflicts:
	project/SparkBuild.scala
 2013-06-18 14:55:31 +02:00
Rohit Rai b5b12823fa Fixing the style as per feedback 2013-06-13 14:05:46 +05:30
Rohit Rai b104c7f5c7 Example to write the output to cassandra 2013-06-03 15:15:52 +05:30
Rohit Rai 56c64c4033 A better way to read column value if you are sure the column exists in every row. 2013-06-03 12:48:35 +05:30
Rohit Rai 6d8423fd1b Adding deps to examples/pom.xml 
Fixing exclusion in examples deps in SparkBuild.scala
 2013-06-02 13:03:45 +05:30
Rohit Rai 81c2adc15c Removing infix call 2013-06-02 12:51:15 +05:30
Rohit Rai 3be7bdcefd Adding example to make Spark RDD from Cassandra 2013-06-01 19:32:17 +05:30
Ethan Jewett 3217d486f7 Add hBase dependency to examples POM 2013-05-20 19:41:38 -05:00
Ethan Jewett ee6f6aa6cd Add hBase example 2013-05-09 18:33:38 -05:00
Reynold Xin 012c9e5ab0 Revert "Merge pull request #596 from esjewett/master" because the 
dependency on hbase introduces netty-3.2.2 which conflicts with
netty-3.5.3 already in Spark. This caused multiple test failures.

This reverts commit 0f1b7a06e1, reversing
changes made to aacca1b8a8.
 2013-05-09 14:20:01 -07:00
Ethan Jewett a3d5f92210 Switch to using SparkContext method to create RDD 2013-05-07 11:43:06 -05:00
unknown cbf6a5ee1e Removed unused code, clarified intent of the program, batch size to 1 second 2013-05-06 08:05:45 -06:00
Ethan Jewett 7cff7e7897 Fix indents and mention other configuration options 2013-05-04 14:56:55 -05:00
Ethan Jewett 9290f16430 Remove unnecessary column family config 2013-05-04 12:39:14 -05:00
Ethan Jewett 02e8cfa617 HBase example 2013-05-04 12:31:30 -05:00
unknown 1d54401d7e Modified as per TD's suggestions 2013-04-30 23:01:32 -06:00
Mridul Muralidharan dd515ca3ee Attempt at fixing merge conflict 2013-04-24 09:24:17 +05:30
unknown 0dc1e2d60f Examaple of cumulative counting using updateStateByKey 2013-04-22 09:22:45 -06:00
Mridul Muralidharan 7acab3ab45 Fix review comments, add a new api to SparkHadoopUtil to create appropriate Configuration. Modify an example to show how to use SplitInfo 2013-04-22 08:01:13 +05:30
seanm 7e56e99573 Surfacing decoders on KafkaInputDStream 2013-04-16 17:17:16 -06:00
Andrew Ash f1d8871ca1 Uniform whitespace across scala examples 2013-04-09 23:35:13 -04:00
Matei Zaharia 65caa8f711 Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0' 
Conflicts:
	docs/_config.yml
	project/SparkBuild.scala
 2013-04-08 12:43:17 -04:00
Matei Zaharia b362df39ea Merge pull request #552 from MLnick/master 
Bumping version for Twitter Algebird to latest
 2013-04-07 17:17:52 -07:00
Mridul Muralidharan 6798a09df8 Add support for building against hadoop2-yarn : adding new maven profile for it 2013-04-07 17:47:38 +05:30