Shivaram Venkataraman
471fbadd0c
Java examples, tests for KMeans and ALS
...
- Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it
easier to call from Java
- Renames class methods from `train` to `run` to enable static methods to be
called from Java.
- Add unit tests which check if both static / class methods can be called.
- Also add examples which port the main() function in ALS, KMeans to the
examples project.
Couple of minor changes to existing code:
- Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily
- Workaround a bug where using double[] from Java leads to class cast exception in
KMeans init
2013-08-06 15:43:46 -07:00
anfeng
dda2ac8b5d
reformat registerFileSystemStat()
2013-08-06 15:22:25 -07:00
Karen Feng
099528b6c4
Pre-sorts stage/env tables, changes text/link of stage summaries
2013-08-06 14:52:12 -07:00
Matei Zaharia
d2b0f0c23d
Merge pull request #770 from stayhf/SPARK-760-Java
...
Simple PageRank algorithm implementation in Java for SPARK-760
2013-08-06 14:49:39 -07:00
stayhf
882baee489
Got rid of unnecessary map function
2013-08-06 21:34:39 +00:00
Karen Feng
254a930730
Reverse sorts StageTable by submitted time
2013-08-06 14:18:38 -07:00
stayhf
326a7a82e0
changes as reviewer requested
2013-08-06 21:03:24 +00:00
Karen Feng
5ed5b73026
Sorts first column of env tables
2013-08-06 13:59:53 -07:00
anfeng
0748c60817
expose HDFS file system stats via Executor metrics
2013-08-06 11:47:06 -07:00
Reynold Xin
d031f73679
Merge pull request #782 from WANdisco/master
...
SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD
2013-08-05 22:33:00 -07:00
Matei Zaharia
1b63dea816
Merge pull request #769 from markhamstra/NegativeCores
...
SPARK-847 + SPARK-845: Zombie workers and negative cores
2013-08-05 22:21:26 -07:00
Alexander Pivovarov
a30866438b
SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD
2013-08-05 21:48:43 -07:00
Matei Zaharia
828aff744d
Merge pull request #776 from gingsmith/master
...
adding matrix factorization data generator
2013-08-05 21:37:33 -07:00
Ginger Smith
bf7033f3eb
fixing formatting, style, and input
2013-08-05 21:26:24 -07:00
Matei Zaharia
8b277892c9
Merge pull request #774 from pwendell/job-description
...
Show user-defined job name in UI
2013-08-05 19:14:52 -07:00
Christopher Nguyen
b1bbbe699c
[HOTFIX] Mark lastSetSparkEnv @volatile in case it gets HotSpot-cached
...
On branch adatao-global-SparkEnv
Changes to be committed:
modified: core/src/main/scala/spark/SparkEnv.scala
2013-08-05 17:22:27 -07:00
Mark Hamstra
35d8f5ee52
Moved handling of timed out workers within the Master actor
2013-08-05 13:13:56 -07:00
Mark Hamstra
37ccf9301a
milliseconds -> seconds in timeOutDeadWorkers logging
2013-08-05 13:13:56 -07:00
Mark Hamstra
cdd1af562e
Timeout zombie workers
2013-08-05 13:13:56 -07:00
Mikhail Bautin
e8bec8365f
Only reduce the number of cores once when removing an executor
2013-08-05 13:13:56 -07:00
Karen Feng
95025afdec
Made most small fixes for SPARK-849 except for table sort, task progress overlay
2013-08-05 13:04:56 -07:00
Patrick Wendell
550b0cf48a
Merge pull request #780 from cybermaster/master
...
SPARK-850
2013-08-05 12:10:32 -07:00
Bill Zhao
33b9b155de
JBoss repository working now
2013-08-05 12:02:36 -07:00
Bill Zhao
9df66cd831
Merge branch 'master' of github.com:cybermaster/spark
2013-08-05 11:56:30 -07:00
Bill Zhao
87134b3648
SPARK-850: give better console message
2013-08-05 11:55:35 -07:00
Ginger Smith
8c8947e2b6
fixing formatting
2013-08-05 11:22:18 -07:00
Bill Zhao
d93d5fcaac
SPARK-850: Give better error message on the console
2013-08-05 10:09:03 -07:00
Christopher Nguyen
39e4fda76f
[HOTFIX] Extend thread safety for SparkEnv.get()
...
A ThreadLocal SparkEnv.env is facing various situations leading to
NullPointerExceptions, where SparkEnv.env set in one thread is not
gettable in another thread, but often assumed to be available.
See, e.g., https://groups.google.com/forum/#!topic/spark-developers/GLx8yunSj0A
This hotfixes SparkEnv.env to return either (a) the ThreadLocal
value if non-null, or (b) the previously set value in any thread.
This approach preserves SparkEnv.set() thread safety needed by
RDD.compute() and possibly other places. A refactoring that
parameterizes SparkEnv should be addressed subsequently.
On branch adatao-global-SparkEnv
Changes to be committed:
modified: core/src/main/scala/spark/SparkEnv.scala
2013-08-05 02:09:54 -07:00
stayhf
98fd62605d
Updated code with reviewer's suggestions
2013-08-05 00:30:28 +00:00
Patrick Wendell
f3660d5ab8
Make output formatting consistent between bash/scala
2013-08-03 21:30:15 -07:00
Patrick Wendell
ad94fbb322
Log the launch command for Spark executors
2013-08-03 09:19:46 -07:00
stayhf
a682637301
Simple PageRank algorithm implementation in Java for SPARK-760
2013-08-03 06:01:16 +00:00
Ginger Smith
4ab4df5edb
adding matrix factorization data generator
2013-08-02 22:22:36 -07:00
Patrick Wendell
b4905c383b
Log the launch command for Spark daemons
...
For debugging and analysis purposes, it's nice to have the exact command
used to launch Spark contained within the logs. This adds the necessary
hooks to make that possible.
2013-08-02 16:58:19 -07:00
Matei Zaharia
22abbc10d6
Merge pull request #772 from karenfeng/ui-843
...
Show app duration
2013-08-02 16:37:59 -07:00
Matei Zaharia
abfa9e6f70
Increase Kryo buffer size in ALS since some arrays become big
2013-08-02 16:17:32 -07:00
Patrick Wendell
5b3784a79c
Show user-defined job name in UI
2013-08-02 15:47:41 -07:00
Karen Feng
b3ae5b25d5
Shows time the app has been running
2013-08-02 13:25:14 -07:00
Patrick Wendell
9d7dfd2d5a
Merge pull request #743 from pwendell/app-metrics
...
Add application metrics to standalone master
2013-08-01 17:41:58 -07:00
Patrick Wendell
6d7afd7ced
Merge pull request #768 from pwendell/pr-695
...
Minor clean-up of fair scheduler UI
2013-08-01 17:13:28 -07:00
Karen Feng
999eaac78c
Merge branch 'master' of https://github.com/mesos/spark
2013-08-01 15:54:56 -07:00
Matei Zaharia
e466a55a6b
Revert Mesos version to 0.9 since the 0.12 artifact has target Java 7
2013-08-01 15:45:21 -07:00
Patrick Wendell
d3c37ff120
Improving documentation in config file example
2013-08-01 15:26:26 -07:00
Patrick Wendell
f1d2ad550e
under_scores --> camelCase for config options
2013-08-01 15:26:26 -07:00
Patrick Wendell
12d9c82c9b
Small style fix
2013-08-01 15:25:52 -07:00
Patrick Wendell
37bc64a205
Adding application-level metrics.
...
This adds metrics for applications in the deploy Master.
2013-08-01 15:25:52 -07:00
Karen Feng
73692f3cb9
Unify, reduce body font size
2013-08-01 15:10:30 -07:00
Patrick Wendell
87fd321a5a
Minor refactoring and code cleanup
2013-08-01 15:02:31 -07:00
Patrick Wendell
b10199413a
Slight refactoring to SparkContext functions
2013-08-01 15:00:42 -07:00
Patrick Wendell
cfcd77b5da
Increasing inter job arrival
2013-08-01 15:00:42 -07:00