ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4123 commits 6 branches 181 tags 397 MiB
Commit graph

2130 commits

Author SHA1 Message Date
Matei Zaharia 246bf67f58 Fix test 2013-09-02 10:57:34 -07:00
Matei Zaharia 9329a7d4cd Fix spark.io.compression.codec and change default codec to LZF 2013-09-02 10:15:22 -07:00
Matei Zaharia 6550e5e60c Allow PySpark to launch worker.py directly on Windows 2013-09-01 18:06:15 -07:00
Matei Zaharia 3db404a43a Run script fixes for Windows after package & assembly change 2013-09-01 23:45:57 +00:00
Matei Zaharia 0a8cc30921 Move some classes to more appropriate packages: 
* RDD, *RDDFunctions -> org.apache.spark.rdd
* Utils, ClosureCleaner, SizeEstimator -> org.apache.spark.util
* JavaSerializer, KryoSerializer -> org.apache.spark.serializer
 2013-09-01 14:13:16 -07:00
Matei Zaharia 5701eb92c7 Fix some URLs 2013-09-01 14:13:16 -07:00
Matei Zaharia 12495ec63a Remove shutdown hook to stop jetty; this is unnecessary for releasing 
ports and creates noisy log messages
 2013-09-01 14:13:15 -07:00
Matei Zaharia 46eecd110a Initial work to rename package to org.apache.spark 2013-09-01 14:13:13 -07:00
Matei Zaharia a30fac16ca Merge pull request #883 from alig/master 
Don't require the spark home environment variable to be set for standalone mode (change needed by SIMR)
 2013-09-01 12:27:50 -07:00
Matei Zaharia e34bc3a8ee Small tweak 2013-08-31 17:47:15 -07:00
Matei Zaharia 2ee6a7e32a Print output from spark-daemon only when it fails to launch 2013-08-31 17:31:07 -07:00
Ali Ghodsi 250bddc255 Don't require spark home to be set for standalone mode 2013-08-31 17:29:05 -07:00
Matei Zaharia 25ac50668b Various web UI improvements: 
- Use "fluid" layout that can expand to wide browser windows, instead of
  the old one's limit of 1200 px
- Remove unnecessary <hr> elements
- Switch back to Bootstrap's default theme and tweak progress bar colors
- Make headers more consistent between deploy and app UIs
- Replace some inline CSS with stylesheets
 2013-08-31 16:55:40 -07:00
Y.CORP.YAHOO.COM\tgraves 96452eea56 fix up minor things 2013-08-30 16:04:31 -05:00
Y.CORP.YAHOO.COM\tgraves bac46266a9 Link the Spark UI to the Yarn UI 2013-08-30 15:55:32 -05:00
Mikhail Bautin 35090958b3 Also add getConf to NewHadoopRDD 2013-08-30 11:03:57 -07:00
Mikhail Bautin 5e30172f70 Make HadoopRDD's configuration accessible 2013-08-30 11:01:06 -07:00
Matei Zaharia ca71620950 Merge pull request #857 from mateiz/assembly 
Change build and run instructions to use assemblies
 2013-08-29 21:51:14 -07:00
Matei Zaharia 666d93c294 Update Maven build to create assemblies expected by new scripts 
This includes the following changes:
- The "assembly" package now builds in Maven by default, and creates an
  assembly containing both hadoop-client and Spark, unlike the old
  BigTop distribution assembly that skipped hadoop-client
- There is now a bigtop-dist package to build the old BigTop assembly
- The repl-bin package is no longer built by default since the scripts
  don't reply on it; instead it can be enabled with -Prepl-bin
- Py4J is now included in the assembly/lib folder as a local Maven repo,
  so that the Maven package can link to it
- run-example now adds the original Spark classpath as well because the
  Maven examples assembly lists spark-core and such as provided
- The various Maven projects add a spark-yarn dependency correctly
 2013-08-29 21:19:06 -07:00
Matei Zaharia aab345c463 Fix finding of assembly JAR, as well as some pointers to ./run 2013-08-29 21:19:06 -07:00
Matei Zaharia ab0e625d9e Fix PySpark for assembly run and include it in dist 2013-08-29 21:19:06 -07:00
Matei Zaharia 53cd50c069 Change build and run instructions to use assemblies 
This commit makes Spark invocation saner by using an assembly JAR to
find all of Spark's dependencies instead of adding all the JARs in
lib_managed. It also packages the examples into an assembly and uses
that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script
with two better-named scripts: "run-examples" for examples, and
"spark-class" for Spark internal classes (e.g. REPL, master, etc). This
is also designed to minimize the confusion people have in trying to use
"run" to run their own classes; it's not meant to do that, but now at
least if they look at it, they can modify run-examples to do a decent
job for them.

As part of this, Bagel's examples are also now properly moved to the
examples package instead of bagel.
 2013-08-29 21:19:04 -07:00
jerryshao f3dbe6b215 Fix removed block zero size log reporting 2013-08-30 09:39:01 +08:00
Patrick Wendell abdbacf252 Merge pull request #871 from pwendell/expose-local 
Expose `isLocal` in SparkContext.
 2013-08-28 21:11:31 -07:00
Patrick Wendell 30d2421112 Make local variable public 2013-08-28 19:53:31 -07:00
Matei Zaharia baa84e7e4c Merge pull request #865 from tgravescs/fixtmpdir 
Spark on Yarn should use yarn approved directories for spark.local.dir and tmp
 2013-08-28 12:44:46 -07:00
Y.CORP.YAHOO.COM\tgraves aac1214ee4 Change Executor to only look at the env variable SPARK_YARN_MODE 2013-08-28 13:26:26 -05:00
Y.CORP.YAHOO.COM\tgraves 3f206bf0b5 Updated based on review comments. 2013-08-27 14:34:27 -05:00
Y.CORP.YAHOO.COM\tgraves cf52a3cba6 Allow for Executors to have different directories then the Spark Master for Yarn 2013-08-27 11:00:21 -05:00
Reynold Xin a77e0abb96 Added worker state to the cluster master JSON ui. 2013-08-26 11:21:03 -07:00
Reynold Xin 9db1e50344 Revert "Merge pull request #841 from rxin/json" 
This reverts commit 1fb1b09928, reversing
changes made to c69c48947d.
 2013-08-26 11:05:14 -07:00
Matei Zaharia 8a36fd09dd Merge pull request #854 from markhamstra/pomUpdate 
Synced sbt and maven builds to use the same dependencies, etc.
 2013-08-22 10:13:35 -07:00
Matei Zaharia c2d00f12e2 Merge pull request #832 from alig/coalesce 
Coalesced RDD with locality
 2013-08-22 10:13:03 -07:00
Mark Hamstra ff6f1b0500 Synced sbt and maven builds 2013-08-21 13:50:24 -07:00
Mark Hamstra 5eea613ec0 Removed meaningless types 2013-08-20 16:49:18 -07:00
Ali Ghodsi f20ed14e87 Merged in from upstream to use TaskLocation instead of strings 2013-08-20 16:21:43 -07:00
Ali Ghodsi 5cd21c4195 added curly braces to make the code more consistent 2013-08-20 16:16:05 -07:00
Ali Ghodsi db4bc55bef indent 2013-08-20 16:16:05 -07:00
Ali Ghodsi c0942a710f Bug in test fixed 2013-08-20 16:16:05 -07:00
Ali Ghodsi 5db41919b5 Added a test to make sure no locality preferences are ignored 2013-08-20 16:16:05 -07:00
Ali Ghodsi 7b123b3126 Simpler code 2013-08-20 16:16:05 -07:00
Ali Ghodsi 9192c358e4 simpler code 2013-08-20 16:16:05 -07:00
Ali Ghodsi a75a64eade Fixed almost all of Matei's feedback 2013-08-20 16:16:05 -07:00
Ali Ghodsi f1c853d76d fixed Matei's comments 2013-08-20 16:16:04 -07:00
Ali Ghodsi 890ea6ba79 making CoalescedRDDPartition public 2013-08-20 16:16:04 -07:00
Ali Ghodsi d6b6c680be comment in the test to make it more understandable 2013-08-20 16:16:04 -07:00
Ali Ghodsi b69e7166ba Coalescer now uses current preferred locations for derived RDDs. Made run() in DAGScheduler thread safe and added a method to be able to ask it for preferred locations. Added a similar method that wraps the former inside SparkContext. 2013-08-20 16:16:04 -07:00
Ali Ghodsi 3b5bb8a4ae added one test that will test a future functionality 2013-08-20 16:13:37 -07:00
Ali Ghodsi 33a0f59354 Added error messages to the tests to make failed tests less cryptic 2013-08-20 16:13:37 -07:00
Ali Ghodsi abcefb3858 fixed matei's comments 2013-08-20 16:13:37 -07:00
Ali Ghodsi 35537e6341 Made a function object that returns the coalesced groups 2013-08-20 16:13:37 -07:00
Ali Ghodsi 339598c080 several of Reynold's suggestions implemented 2013-08-20 16:13:37 -07:00
Ali Ghodsi 02d6464f2f space removed 2013-08-20 16:13:37 -07:00
Ali Ghodsi 4f99be1ffd use count rather than foreach 2013-08-20 16:13:37 -07:00
Ali Ghodsi f67753cdfc made preferredLocation a val of the surrounding case class 2013-08-20 16:13:37 -07:00
Ali Ghodsi f24861b60a Fix bug in tests 2013-08-20 16:13:36 -07:00
Ali Ghodsi f6e47e8b51 Renamed split to partition 2013-08-20 16:13:36 -07:00
Ali Ghodsi 937f72feb8 word wrap before 100 chars per line 2013-08-20 16:13:36 -07:00
Ali Ghodsi c4d59910b1 added goals inline as comment 2013-08-20 16:13:36 -07:00
Ali Ghodsi 7a2a33e32d Large scale load and locality tests for the coalesced partitions added 2013-08-20 16:13:36 -07:00
Ali Ghodsi 66edf854aa Bug, should compute slack wrt parent partition size, not number of bins 2013-08-20 16:13:36 -07:00
Ali Ghodsi 1ede102ba5 load balancing coalescer 2013-08-20 16:13:36 -07:00
Matei Zaharia aa2b89d98d Merge remote-tracking branch 'jey/hadoop-agnostic' 
Conflicts:
	core/src/main/scala/spark/PairRDDFunctions.scala
 2013-08-20 10:14:15 -07:00
Mark Hamstra 1630fbf838 changeGeneration --> changeEpoch renaming 2013-08-20 00:17:16 -07:00
Mark Hamstra ad18410427 Renamed 'priority' to 'jobId' and assorted minor changes 2013-08-20 00:07:04 -07:00
Matei Zaharia 8cae72e94e Merge pull request #828 from mateiz/sched-improvements 
Scheduler fixes and improvements
 2013-08-19 23:40:04 -07:00
Matei Zaharia efeb142981 Merge pull request #849 from mateiz/web-fixes 
Small fixes to web UI
 2013-08-19 19:23:50 -07:00
Matei Zaharia 793a722f8e Allow some wiggle room in UISuite port test and in EC2 ports 2013-08-19 18:51:00 -07:00
Matei Zaharia abdc1f8bbb Merge pull request #847 from rxin/rdd 
Allow subclasses of Product2 in all key-value related classes
 2013-08-19 18:30:56 -07:00
Matei Zaharia 498a26189b Small fixes to web UI: 
- Use SPARK_PUBLIC_DNS environment variable if set (for EC2)
- Use a non-ephemeral port (3030 instead of 33000) by default
- Updated test to use non-ephemeral port too
 2013-08-19 18:17:49 -07:00
Reynold Xin 5054abd41b Code review feedback. (added tests for cogroup and substract; added more documentation on MutablePair) 2013-08-19 12:58:02 -07:00
Reynold Xin acc4aa1f47 Added a test for sorting using MutablePair's. 2013-08-19 11:02:10 -07:00
Reynold Xin 71d705a66e Made PairRDDFunctions taking only Tuple2, but made the rest of the shuffle code path working with general Product2. 2013-08-19 00:40:43 -07:00
Reynold Xin 2a7b99c08b Added the missing RDD files and cleaned up SparkContext. 2013-08-18 20:39:29 -07:00
Reynold Xin 82bf4c0339 Allow subclasses of Product2 in all key-value related classes (ShuffleDependency, PairRDDFunctions, etc). 2013-08-18 20:25:45 -07:00
Matei Zaharia 8ac3d1e263 Added unit tests for ClusterTaskSetManager, and fix a bug found with 
resetting locality level after a non-local launch
 2013-08-18 19:51:07 -07:00
Matei Zaharia 4004cf775d Added some comments on threading in scheduler code 2013-08-18 19:51:07 -07:00
Matei Zaharia 2a4ed10210 Address some review comments: 
- When a resourceOffers() call has multiple offers, force the TaskSets
  to consider them in increasing order of locality levels so that they
  get a chance to launch stuff locally across all offers

- Simplify ClusterScheduler.prioritizeContainers

- Add docs on the new configuration options
 2013-08-18 19:51:07 -07:00
Matei Zaharia 222c897128 Comment cleanup (via Kay) and some debug messages 2013-08-18 19:51:07 -07:00
Matei Zaharia cf39d45d14 More scheduling fixes: 
- Added periodic revival of offers in StandaloneSchedulerBackend

- Replaced task scheduling aggression with multi-level delay scheduling
  in ClusterTaskSetManager

- Fixed ZippedRDD preferred locations because they can't currently be
  process-local

- Fixed some uses of hostPort
 2013-08-18 19:51:07 -07:00
Matei Zaharia 90a04dab8d Initial work towards scheduler refactoring: 
- Replace use of hostPort vs host in Task.preferredLocations with a
  TaskLocation class that contains either an executorId and a host or
  just a host. This is part of a bigger effort to eliminate hostPort
  based data structures and just use executorID, since the hostPort vs
  host stuff is confusing (and not checkable with static typing, leading
  to ugly debug code), and hostPorts are not provided by Mesos.

- Replaced most hostPort-based data structures and fields as above.

- Simplified ClusterTaskSetManager to deal with preferred locations in a
  more concise way and generally be more concise.

- Updated the way ClusterTaskSetManager handles racks: instead of
  enqueueing a task to a separate queue for all the hosts in the rack,
  which would create lots of large queues, have one queue per rack name.

- Removed non-local fallback stuff in ClusterScheduler that tried to
  launch less-local tasks on a node once the local ones were all
  assigned. This change didn't work because many cluster schedulers send
  offers for just one node at a time (even the standalone and YARN ones
  do so as nodes join the cluster one by one). Thus, lots of non-local
  tasks would be assigned even though a node with locality for them
  would be able to receive tasks just a short time later.

- Renamed MapOutputTracker "generations" to "epochs".
 2013-08-18 19:51:06 -07:00
Jey Kottalam bdd861c6c3 Fix Maven build with Hadoop 0.23.9 2013-08-18 18:28:57 -07:00
Matei Zaharia 8fa0747978 Merge pull request #840 from AndreSchumacher/zipegg 
Implementing SPARK-878 for PySpark: adding zip and egg files to context ...
 2013-08-18 17:02:54 -07:00
Reynold Xin 2c00ea3efc Moved shuffle serializer setting from a constructor parameter to a setSerializer method in various RDDs that involve shuffle operations. 2013-08-17 21:43:29 -07:00
Reynold Xin 0e84fee76b Removed the mapSideCombine option in partitionBy. 2013-08-17 21:13:41 -07:00
Reynold Xin 10af952a3d Removed the mapSideCombine option in CoGroupedRDD. 2013-08-17 21:07:34 -07:00
Reynold Xin 5d050a3e1f Removed the unused shuffleId in ShuffleDependency's constructor. 2013-08-16 23:23:16 -07:00
Matei Zaharia e89ffc7b3c Merge pull request #839 from jegonzal/zip_partitions 
Currying RDD.zipPartitions
 2013-08-16 14:02:34 -07:00
Jey Kottalam ad580b94d5 Maven build now also works with YARN 2013-08-16 13:50:12 -07:00
Jey Kottalam 9dd15fe700 Don't mark hadoop-client as 'provided' 2013-08-16 13:50:12 -07:00
Jey Kottalam 11b42a84db Maven build now works with CDH hadoop-2.0.0-mr1 2013-08-16 13:50:12 -07:00
Jey Kottalam 353fab2440 Initial changes to make Maven build agnostic of hadoop version 2013-08-16 13:50:12 -07:00
Joseph E. Gonzalez 53b2639a1e Reversing the argument order in zipPartitions to enable stronger type inference. 2013-08-16 12:38:59 -07:00
Andre Schumacher c7e348faec Implementing SPARK-878 for PySpark: adding zip and egg files to context and passing it down to workers which add these to their sys.path 2013-08-16 11:58:20 -07:00
Reynold Xin c961c19b7b Use the JSON formatter from Scala library and removed dependency on lift-json. 
It made the JSON creation slightly more complicated, but reduces one external dependency. The scala library also properly escape "/" (which lift-json doesn't).
 2013-08-15 18:23:01 -07:00
Reynold Xin eddbf43b54 Revert "Merge pull request #834 from Daemoen/master" 
This reverts commit 230ab2722e, reversing
changes made to 659553b21d.
 2013-08-15 17:49:37 -07:00
Reynold Xin 230ab2722e Merge pull request #834 from Daemoen/master 
Updated json output to allow for display of worker state
 2013-08-15 17:45:17 -07:00
Patrick Wendell 659553b21d Merge pull request #836 from pwendell/rename 
Rename `memoryBytesToString` and `memoryMegabytesToString`
 2013-08-15 16:56:31 -07:00
Jey Kottalam a06a9d5c5f Rename HadoopWriter to SparkHadoopWriter since it's outside of our package 2013-08-15 16:50:37 -07:00
Jey Kottalam 8f979edef5 Fix newTaskAttemptID to work under YARN 2013-08-15 16:50:37 -07:00
Jey Kottalam e2d7656ca3 re-enable YARN support 2013-08-15 16:50:37 -07:00
Jey Kottalam bd0bab47c9 SparkEnv isn't available this early, and not needed anyway 2013-08-15 16:50:37 -07:00
Jey Kottalam 4f43fd791a make SparkHadoopUtil a member of SparkEnv 2013-08-15 16:50:37 -07:00
Jey Kottalam 43ebcb8484 rename HadoopMapRedUtil => SparkHadoopMapRedUtil, HadoopMapReduceUtil => SparkHadoopMapReduceUtil 2013-08-15 16:50:37 -07:00
Jey Kottalam 8b1c1520fc add comment 2013-08-15 16:50:37 -07:00
Jey Kottalam 69c3bbf688 dynamically detect hadoop version 2013-08-15 16:50:37 -07:00
Jey Kottalam f67b94ad4f remove core/src/hadoop{1,2} dirs 2013-08-15 16:50:36 -07:00
Jey Kottalam b877e20a33 move yarn to its own directory 2013-08-15 16:50:36 -07:00
Patrick Wendell 4c6ade1ad5 Rename memoryBytesToString and memoryMegabytesToString 
These are used all over the place now and they are not specific to memory at all.

memoryBytesToString --> bytesToString
memoryMegabytesToString --> megabytesToString
 2013-08-15 15:58:07 -07:00
Reynold Xin 1a51deae8a More minor UI changes including code review feedback. 2013-08-15 14:34:07 -07:00
Daemoen ad2e8b5126 Updated json output to allow for display of worker state 
Ops teams need to ensure that the cluster is functional and performant.  Having to scrape the html source for worker state won't work reliably, and will be slow.  By exposing the state in the json output, ops teams are able to ensure a fully functional environment by querying for the json output and parsing for dead nodes.
 2013-08-15 12:19:14 -07:00
Reynold Xin 2d2a556bdf Various UI improvements. 2013-08-14 23:23:09 -07:00
Reynold Xin 290e3e6e65 Renamed setCurrentJobDescription to setJobDescription. 2013-08-14 18:40:53 -07:00
Reynold Xin 3886b54933 A few small scheduler / job description changes. 
1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property.

2. Renamed SparkContext.setDescription to setCurrentJobDescription.

3. Throw an exception if the fair scheduler allocation file is invalid.
 2013-08-14 17:19:42 -07:00
Matei Zaharia 839f2d4f3f Merge pull request #822 from pwendell/ui-features 
Adding GC Stats to TaskMetrics (and three small fixes)
 2013-08-14 16:17:23 -07:00
Patrick Wendell 04ad78b09d Style cleanup based on Matei feedback 2013-08-14 14:57:21 -07:00
Kay Ousterhout a88aa5e6ed Fixed 2 bugs in executor UI. 
1) UI crashed if the executor UI was loaded before any tasks started.
2) The total tasks was incorrectly reported due to using string (rather
than int) arithmetic.
 2013-08-13 23:44:58 -07:00
Patrick Wendell c223176388 Small style clean-up 2013-08-13 16:56:37 -07:00
Patrick Wendell fab5cee111 Correcting terminology in RDD page 2013-08-13 16:25:55 -07:00
Patrick Wendell 024e5c5ce1 Correct sorting order for stages 2013-08-13 16:25:55 -07:00
Patrick Wendell 4e9f0c2df6 Capturing GC detials in TaskMetrics 2013-08-13 16:25:55 -07:00
Patrick Wendell f0382007dc Bug fix for display of shuffle read/write metrics. 
This fixes an error where empty cells are missing if a given task
has no shuffle read/write.
 2013-08-13 16:25:55 -07:00
Matei Zaharia d316af9c84 Merge pull request #821 from pwendell/print-launch-command 
Print run command to stderr rather than stdout
 2013-08-13 15:31:01 -07:00
Patrick Wendell a7feb69ae8 Print run command to stderr rather than stdout 2013-08-13 15:07:03 -07:00
Kay Ousterhout 1beb843a6f Reuse the set of failed states rather than creating a new object each time 2013-08-13 14:27:40 -07:00
Kay Ousterhout c92dd627ca Properly account for killed tasks. 
The TaskState class's isFinished() method didn't return true for
KILLED tasks, which means some resources are never reclaimed
for tasks that are killed. This also made it inconsistent with the
isFinished() method used by CoarseMesosSchedulerBackend.
 2013-08-13 12:40:15 -07:00
Patrick Wendell ed6a1646e6 Slight change to pr-784 2013-08-13 09:29:40 -07:00
Patrick Wendell a0133bfbad Merge pull request #784 from jerryshao/dev-metrics-servlet 
Add MetricsServlet for Spark metrics system
 2013-08-13 09:28:18 -07:00
Matei Zaharia 65d0d91fba Merge pull request #807 from JoshRosen/guava-optional 
Change scala.Option to Guava Optional in Java APIs
 2013-08-12 19:00:57 -07:00
Josh Rosen cf08bb7a3e Fix import organization. 2013-08-12 18:55:02 -07:00
jerryshao 09c7179e81 MetricsServlet code refactor according to comments 2013-08-12 13:23:23 +08:00
jerryshao 320e87e7ab Add MetricsServlet for Spark metrics system 2013-08-12 13:23:23 +08:00
Reynold Xin e5b9ed2833 Merge pull request #808 from pwendell/ui_compressed_bytes 
Report compressed bytes read when calculating TaskMetrics
 2013-08-11 17:22:47 -07:00
Patrick Wendell 3d8f281604 Report compressed bytes read when calculating TaskMetrics 2013-08-11 16:25:57 -07:00
Matei Zaharia 379648630b Merge pull request #805 from woggle/hadoop-rdd-jobconf 
Use new Configuration() instead of slower new JobConf() in SerializableWritable
 2013-08-11 14:51:47 -07:00
Josh Rosen d7f78b443b Change scala.Option to Guava Optional in Java APIs. 2013-08-11 12:05:09 -07:00
Charles Reiss 6402b539d0 Use new Configuration() instead of new JobConf() for ObjectWritable. 
JobConf's constructor loads default config files in some verisons of
Hadoop, which is quite slow, and we only need the Configuration object
to pass the correct ClassLoader.
 2013-08-10 21:31:05 -07:00
Matei Zaharia 71c63de22f Merge pull request #795 from mridulm/master 
Fix bug reported in PR 791 : a race condition in ConnectionManager and Connection
 2013-08-10 10:21:20 -07:00
Matei Zaharia d3277a0daf Merge remote-tracking branch 'origin/pr/792' 
Conflicts:
	core/src/main/scala/spark/ui/jobs/IndexPage.scala
	core/src/main/scala/spark/ui/jobs/StagePage.scala
 2013-08-10 10:18:50 -07:00
Patrick Wendell d17eeb997d Merge pull request #785 from anfeng/master 
expose HDFS file system stats via Executor metrics
 2013-08-10 09:02:27 -07:00
Kay Ousterhout 14d14f451a Shortened names, as per Matei's suggestion 2013-08-10 07:50:27 -07:00
Matei Zaharia cd247ba5bb Merge pull request #786 from shivaram/mllib-java 
Java fixes, tests and examples for ALS, KMeans
 2013-08-09 20:41:13 -07:00
Kay Ousterhout 7810a76512 Only print event queue full error message once 2013-08-09 18:20:48 -07:00
Kay Ousterhout 44ca8629d8 Style fix: removing unnecessary return type 2013-08-09 17:22:50 -07:00
Kay Ousterhout 29b79714f9 Style fixes based on code review 2013-08-09 16:46:34 -07:00
Kay Ousterhout 81e1d4a7d1 Refactored SparkListener to process all events asynchronously. 
This commit fixes issues where SparkListeners that take a while to
process events slow the DAGScheduler.

This commit also fixes a bug in the UI where if a user goes to a
web page of a stage that does not exist, they can create a memory
leak (granted, this is not an issue at small scale -- probably only
an issue if someone actively tried to DOS the UI).
 2013-08-09 13:27:41 -07:00
Matei Zaharia b09d4b79e8 Merge pull request #799 from woggle/sync-fix 
Remove extra synchronization in ResultTask
 2013-08-09 13:17:08 -07:00
Patrick Wendell cc6b92e80e Merge pull request #775 from pwendell/print-launch-command 
Log the launch command for Spark daemons
 2013-08-09 13:00:33 -07:00
Patrick Wendell 3970b580c2 Using quotes when printing out command 2013-08-09 11:53:32 -07:00
Charles Reiss 9dfc280f74 Remove extra synchronization in ResultTask 2013-08-09 11:09:02 -07:00