Matei Zaharia
9970926ede
formatting
2013-02-01 14:07:34 -08:00
Matei Zaharia
79c24abe4c
Merge pull request #432 from stephenh/moreprivacy
...
Add more private declarations.
2013-02-01 14:06:55 -08:00
Matei Zaharia
de340ddf0b
Merge pull request #437 from stephenh/cancelmetacleaner
...
Stop BlockManagers metadataCleaner.
2013-02-01 12:59:25 -08:00
Matei Zaharia
0455650713
Merge pull request #439 from JoshRosen/spark-580
...
Use spark.local.dir for PySpark temp files (SPARK-580).
2013-02-01 12:07:42 -08:00
Josh Rosen
e211f405bc
Use spark.local.dir for PySpark temp files (SPARK-580).
2013-02-01 11:50:27 -08:00
Matei Zaharia
b6a6092177
Merge pull request #438 from JoshRosen/spark-674
...
Do not launch JavaGateways on workers (SPARK-674).
2013-02-01 11:29:47 -08:00
Josh Rosen
9cc6ff9c4e
Do not launch JavaGateways on workers (SPARK-674).
...
The problem was that the gateway was being initialized whenever the
pyspark.context module was loaded. The fix uses lazy initialization
that occurs only when SparkContext instances are actually constructed.
I also made the gateway and jvm variables private.
This change results in ~3-4x performance improvement when running the
PySpark unit tests.
2013-02-01 11:13:10 -08:00
Imran Rashid
c6190067ae
remove unneeded (and unused) filter on block info
2013-02-01 09:55:25 -08:00
Stephen Haberman
59c57e48df
Stop BlockManagers metadataCleaner.
2013-02-01 10:34:02 -06:00
Matei Zaharia
571af31304
Merge pull request #433 from rxin/master
...
Changed PartitionPruningRDD's split to make sure it returns the correct split index.
2013-02-01 00:32:41 -08:00
Matei Zaharia
5ce5efec10
Merge pull request #435 from JoshRosen/pyspark_stdout_fix
...
Fix stdout redirection in PySpark.
2013-02-01 00:32:07 -08:00
Josh Rosen
57b64d0d19
Fix stdout redirection in PySpark.
2013-02-01 00:25:19 -08:00
Imran Rashid
8a0a5ed533
track total partitions, in addition to cached partitions; use scala string formatting
2013-02-01 00:23:38 -08:00
Imran Rashid
f127f2ae76
fixup merge (master -> driver renaming)
2013-02-01 00:20:49 -08:00
Reynold Xin
f9af9cee6f
Moved PruneDependency into PartitionPruningRDD.scala.
2013-02-01 00:02:46 -08:00
haitao.yao
b57570fd12
Merge branch 'mesos'
2013-02-01 14:06:45 +08:00
Matei Zaharia
7e2e046e37
Merge pull request #434 from pwendell/python-exceptions
...
SPARK-673: Capture and re-throw Python exceptions
2013-01-31 21:58:26 -08:00
Patrick Wendell
39ab83e957
Small fix from last commit
2013-01-31 21:52:52 -08:00
Patrick Wendell
c33f0ef41a
Some style cleanup
2013-01-31 21:50:02 -08:00
Shivaram Venkataraman
bf675ab4f9
Turn on ganglia by default
2013-01-31 21:43:45 -08:00
Matei Zaharia
95e14fbc38
Merge pull request #431 from mbautin/revert_default_profile
...
Remove activation of profiles by default
2013-01-31 21:34:59 -08:00
Patrick Wendell
3446d5c8d6
SPARK-673: Capture and re-throw Python exceptions
...
This patch alters the Python <-> executor protocol to pass on
exception data when they occur in user Python code.
2013-01-31 18:06:11 -08:00
Reynold Xin
6289d9654e
Removed the TODO comment from PartitionPruningRDD.
2013-01-31 17:49:36 -08:00
Reynold Xin
5b0fc265c2
Changed PartitionPruningRDD's split to make sure it returns the correct
...
split index.
2013-01-31 17:48:39 -08:00
Stephen Haberman
782187c210
Once we find a split with no block, we don't have to look for more.
2013-01-31 18:27:25 -06:00
Stephen Haberman
418e36caa8
Add more private declarations.
2013-01-31 17:18:33 -06:00
Mikhail Bautin
fe3eceab57
Remove activation of profiles by default
...
See the discussion at https://github.com/mesos/spark/pull/355 for why
default profile activation is a problem.
2013-01-31 13:30:41 -08:00
haitao.yao
3190483b98
bug fix for javadoc
2013-01-31 14:23:51 +08:00
Imran Rashid
02a6761589
Merge branch 'master' into blockmanager_info
...
Conflicts:
core/src/main/scala/spark/storage/BlockManagerMaster.scala
2013-01-30 18:52:35 -08:00
Imran Rashid
c1df24d085
rename Slaves --> Executor
2013-01-30 18:51:14 -08:00
haitao.yao
babbb521d5
Merge branch 'mesos'
2013-01-31 10:08:32 +08:00
Matei Zaharia
55327a283e
Merge pull request #430 from pwendell/pyspark-guide
...
Minor improvements to PySpark docs
2013-01-30 15:35:29 -08:00
Patrick Wendell
3f945e3b83
Make module help available in python shell.
...
Also, adds a line in doc explaining how to use.
2013-01-30 15:04:06 -08:00
Patrick Wendell
58a7d320d7
Inclue packaging and launching pyspark in guide.
...
It's nicer if all the commands you need are made explicit.
2013-01-30 15:04:02 -08:00
Matei Zaharia
d12330bd2c
Merge pull request #426 from woggling/conn-manager-ips
...
Remember ConnectionManagerId used to initiate SendingConnections
2013-01-30 15:02:53 -08:00
Matei Zaharia
612a9fee71
Merge pull request #428 from woggling/mesos-exec-id
...
Make ExecutorIDs include SlaveIDs when running Mesos
2013-01-30 15:01:46 -08:00
Matei Zaharia
dfb721b970
Merge pull request #429 from stephenh/includemessage
...
Include message and exitStatus if availalbe.
2013-01-30 15:01:24 -08:00
Stephen Haberman
871476d506
Include message and exitStatus if availalbe.
2013-01-30 16:56:46 -06:00
Charles Reiss
252845d304
Remove remants of attempt to use slaveId-executorId in MesosExecutorBackend
2013-01-30 10:38:06 -08:00
Charles Reiss
f7de6978c1
Use Mesos ExecutorIDs to hold SlaveIDs. Then we can safely use
...
the Mesos ExecutorID as a Spark ExecutorID.
2013-01-30 09:38:57 -08:00
Charles Reiss
7f51458774
Comment at top of DAGSchedulerSuite
2013-01-30 09:34:53 -08:00
Charles Reiss
9c0bae75ad
Change DAGSchedulerSuite to run DAGScheduler in the same Thread.
2013-01-30 09:22:07 -08:00
Charles Reiss
178b89204c
Refactor DAGScheduler more to allow testing without a separate thread.
2013-01-30 09:19:55 -08:00
haitao.yao
764d78d72f
Merge branch 'mesos'
2013-01-30 17:57:51 +08:00
Charles Reiss
4bf3d7ea12
Clear spark.master.port to cleanup for other tests
2013-01-29 19:05:58 -08:00
Charles Reiss
9eac7d01f0
Add DAGScheduler tests.
2013-01-29 18:55:43 -08:00
Charles Reiss
a3d14c0404
Refactoring to DAGScheduler to aid testing
2013-01-29 18:55:42 -08:00
Charles Reiss
0f81025eca
Add easymock to SBT configuration.
2013-01-29 18:55:42 -08:00
Charles Reiss
16a0789e10
Remember ConnectionManagerId used to initiate SendingConnections.
...
This prevents ConnectionManager from getting confused if a machine
has multiple host names and the one getHostName() finds happens
not to be the one that was passed from, e.g., the BlockManagerMaster.
2013-01-29 18:13:59 -08:00
Matei Zaharia
d54b10b6ad
Merge remote-tracking branch 'stephenh/removefailedjob'
...
Conflicts:
core/src/main/scala/spark/deploy/master/Master.scala
2013-01-29 18:12:29 -08:00