Matei Zaharia
43b24635ee
Renamed ML package to MLlib and added it to classpath
2013-07-05 11:38:53 -07:00
Matei Zaharia
399bd65ef5
Fixed compile error due to merge
2013-07-05 11:27:06 -07:00
Shivaram Venkataraman
0e33c88cbd
Rename package gradient to optimization
2013-07-05 11:15:19 -07:00
Shivaram Venkataraman
09f187a400
Add top-level methods for regression methods.
...
Also add multiple versions of them to make it easier to call them from java.
2013-07-05 11:15:19 -07:00
Matei Zaharia
9441d3ef09
Use random seeds for K-means and ALS, and increase tolerance in tests
...
Random seeds make more sense by default for a machine learning library
because other libraries behave the same way (people expect to be able to
run the algorithm multiple times and get a better answer), but we can
add configuration later if needed. Tests that depend on specific seed
choices seem brittle.
2013-07-05 11:15:19 -07:00
Matei Zaharia
e7d49388e3
Added unit test for K-means, and fixed some bugs
2013-07-05 11:15:19 -07:00
Matei Zaharia
652ea0f1d8
Allow RDD.takeSample to give samples bigger than the RDD
...
Before, when withReplacement was set to true, we would not get a sample
bigger than the RDD's count().
Conflicts:
core/src/main/scala/spark/RDD.scala
core/src/test/scala/spark/RDDSuite.scala
2013-07-05 11:15:13 -07:00
Matei Zaharia
cffe3340c5
Fix logistic regression test failure and test suite cleanup
2013-07-05 11:13:46 -07:00
Shivaram Venkataraman
496c7548bb
Change test to use fewer iterations
2013-07-05 11:13:46 -07:00
Matei Zaharia
52f491125e
Implementation of k-means and k-means||
2013-07-05 11:13:46 -07:00
Matei Zaharia
39684eafe3
Formatting
2013-07-05 11:13:46 -07:00
Matei Zaharia
6586c5e28b
Added a SparkContext accessor to RDD
2013-07-05 11:13:46 -07:00
Matei Zaharia
43dae967d7
Renamed "als" package to "recommendation"
2013-07-05 11:13:46 -07:00
Matei Zaharia
d3ce898b8e
Scaffolding and model for K-means
2013-07-05 11:13:46 -07:00
Matei Zaharia
3c046a6eca
Some small fixes to ALS.
2013-07-05 11:13:46 -07:00
Matei Zaharia
6f0ebb2db2
Remove unused import
2013-07-05 11:13:46 -07:00
Matei Zaharia
d903b3887f
Initial implementation of Alternating Least Squares.
...
Includes unit tests and sample data to run on.
2013-07-05 11:13:46 -07:00
Matei Zaharia
05be233ce2
Removed dependency on Apache Commons Math
2013-07-05 11:13:46 -07:00
Shivaram Venkataraman
39ed41652b
Move to regression, util and gradient packages
2013-07-05 11:13:46 -07:00
Shivaram Venkataraman
43b398db6a
Fix logistic regression to not center data.
...
Also add a feature to get the intercept correct and test these
using a small unit test.
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
6dd3a816c8
Use a private constructor instead of private vars
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
76acc9fe9d
Make regression arguments private and add method to predict one point
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
3a6924cb8f
Clean up some comments.
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
6aadaf4d71
Move normalization to MLUtils and remove Regression trait.
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
2d0e64900e
Convert regression classes to builder pattern.
...
Remove extraneous methods and classes
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
ed32ec2b3b
Update test based on interface changes
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
fd137bd7c6
Address Reynold's comments. Also use a builder pattern to construct the regression classes.
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
48770419bd
Add random data used for LR testing.
...
Verified that results match with glm in R
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
282c8ed788
Add LogisticRegression using StochasticGradientDescent.
...
Also refactor RidgeRegression and LogisticRegression to re-use code
and update the test as well
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
b9d9b6f981
Add a unit test for Ridge Regression
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
4dc13bf5be
Revert back to closed form CV error
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
c8169c0a33
Add LPSA data set.
...
Data from
http://www-stat.stanford.edu/~tibs/ElemStatLearn/datasets/prostate.data
2013-07-05 11:13:45 -07:00
Shivaram Venkataraman
c070decb8e
Add methods to normalize the data before training
...
Also update model after training based appropriately.
2013-07-05 11:13:45 -07:00
Reynold Xin
6a9a9a364c
Minor clean up of the RidgeRegression code. I am not even sure why I did
...
this :s.
2013-07-05 11:13:45 -07:00
Matei Zaharia
729e463f64
Import RidgeRegression example
...
Conflicts:
run
2013-07-05 11:13:41 -07:00
Matei Zaharia
6ad85d0918
Merge pull request #677 from jerryshao/fix_stage_clean
...
Clean StageToInfos periodically when spark.cleaner.ttl is enabled
2013-07-04 21:32:29 -07:00
jerryshao
e4ff544a8d
Clean StageToInfos periodically when spark.cleaner.ttl is enabled
2013-07-05 10:34:45 +08:00
Konstantin Boudnik
7687ed5292
Use standard ASF published avro module instead of a proprietory built one
2013-07-04 13:48:33 -07:00
Matei Zaharia
2e32fc8d72
Merge pull request #666 from c0s/master
...
hbase dependency is missed in hadoop2-yarn profile of examples module
2013-07-04 12:18:20 -07:00
Lian Cheng
c0c3155c3c
Bug fix: SPARK-789
...
https://spark-project.atlassian.net/browse/SPARK-789
2013-07-05 00:54:10 +08:00
Andrew xia
6ccfb73ca9
Add fair scheduler config template file
2013-07-04 19:19:44 +08:00
Holden Karau
0f06d6217d
s/ActorSystemImpl/ExtendedActorSystem/ as ActorSystemImpl results in a warning
2013-07-04 01:05:39 -07:00
Gavin Li
94238aae57
fix dependencies
2013-07-03 18:08:38 +00:00
Mingfei
04567a1771
update guava version from 11.0.1 to 14.0.1
2013-07-03 17:43:37 +08:00
Gavin Li
96130c30d9
add compression codec trait and snappy compression
2013-07-03 05:49:04 +00:00
Y.CORP.YAHOO.COM\tgraves
923cf92900
Rework from pull request. Removed --user option from Spark on Yarn Client, made the user of JAVA_HOME environment
...
variable conditional on if its set, and created addCredentials in each of the SparkHadoopUtil classes
to only add the credentials when the profile is hadoop2-yarn.
2013-07-02 21:18:59 -05:00
Patrick Wendell
39e2325675
Removing dead code
2013-07-02 16:28:40 -07:00
Patrick Wendell
8ca1cc1786
Adding truncation for log files
2013-07-02 16:10:50 -07:00
Matei Zaharia
6d60fe571a
Merge pull request #666 from c0s/master
...
hbase dependency is missed in hadoop2-yarn profile of examples module
2013-07-01 18:24:03 -07:00
Konstantin Boudnik
6fdbc68f2c
Fixing missed hbase dependency in examples hadoop2-yarn profile
2013-07-01 17:45:07 -07:00