Luca Rosellini
4689ce29fd
Added license header and removed @author tag
2014-01-07 09:44:24 +01:00
Luca Rosellini
0b6db8c186
Added ‘-i’ command line option to spark REPL.
...
We had to create a new implementation of both scala.tools.nsc.CompilerCommand and scala.tools.nsc.Settings, because using scala.tools.nsc.GenericRunnerSettings would bring in other options (-howtorun, -save and -execute) which don’t make sense in Spark.
Any new Spark specific command line option could now be added to org.apache.spark.repl.SparkRunnerSettings class.
Since the behavior of loading a script from the command line should be the same as loading it using the “:load” command inside the shell, the script should be loaded when the SparkContext is available, that’s why we had to move the call to ‘loadfiles(settings)’ _after_ the call to postInitialization(). This still doesn’t work if ‘isAsync = true’.
2014-01-03 12:57:06 +01:00
Prashant Sharma
94f2fffa23
fixed review comments
2014-01-03 14:43:37 +05:30
Matei Zaharia
e2c68642c6
Miscellaneous fixes from code review.
...
Also replaced SparkConf.getOrElse with just a "get" that takes a default
value, and added getInt, getLong, etc to make code that uses this
simpler later on.
2014-01-01 22:03:39 -05:00
Matei Zaharia
642029e7f4
Various fixes to configuration code
...
- Got rid of global SparkContext.globalConf
- Pass SparkConf to serializers and compression codecs
- Made SparkConf public instead of private[spark]
- Improved API of SparkContext and SparkConf
- Switched executor environment vars to be passed through SparkConf
- Fixed some places that were still using system properties
- Fixed some tests, though others are still failing
This still fails several tests in core, repl and streaming, likely due
to properties not being set or cleared correctly (some of the tests run
fine in isolation).
2013-12-28 17:13:15 -05:00
Prashant Sharma
2573add94c
spark-544, introducing SparkConf and related configuration overhaul.
2013-12-25 00:09:36 +05:30
Prashant Sharma
a854cc536d
Review comments on the PR for scala 2.10 migration.
2013-12-13 15:19:51 +05:30
Prashant Sharma
d092a8cc6a
Fixed compile time warnings and formatting post merge.
2013-11-26 15:21:50 +05:30
Aaron Davidson
f629ba95b6
Various merge corrections
...
I've diff'd this patch against my own -- since they were both created
independently, this means that two sets of eyes have gone over all the
merge conflicts that were created, so I'm feeling significantly more
confident in the resulting PR.
@rxin has looked at the changes to the repl and is resoundingly
confident that they are correct.
2013-11-14 22:13:09 -08:00
Raymond Liu
a60620b76a
Merge branch 'master' into scala-2.10
2013-11-14 12:44:19 +08:00
Raymond Liu
0f2e3c6e31
Merge branch 'master' into scala-2.10
2013-11-13 16:55:11 +08:00
Reynold Xin
319299941d
Propagate the SparkContext local property from the thread that calls the spark-repl to the actual execution thread.
2013-11-09 00:32:14 -08:00
Ali Ghodsi
05a0df2b9e
Makes Spark SIMR ready.
2013-10-24 11:59:51 -07:00
Aaron Davidson
74737264c4
Spark shell exits if it cannot create SparkContext
...
Mainly, this occurs if you provide a messed up MASTER url (one that doesn't match one
of our regexes). Previously, we would default to Mesos, fail, and then start the shell
anyway, except that any Spark command would fail.
2013-10-17 18:51:19 -07:00
Prashant Sharma
5829692885
Merge branch 'master' into scala-2.10
...
Conflicts:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala
docs/_config.yml
project/SparkBuild.scala
repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
2013-10-01 11:57:24 +05:30
Patrick Wendell
6079721fa1
Update build version in master
2013-09-24 11:41:51 -07:00
Prashant Sharma
69fd42aee3
ported repl improvements from master
2013-09-15 15:51:02 +05:30
Prashant Sharma
20c65bc334
Fixed repl suite
2013-09-15 10:43:06 +05:30
Prashant Sharma
6fcfefcb27
Few more fixes to tests broken during merge
2013-09-10 10:57:47 +05:30
Prashant Sharma
4106ae9fbf
Merged with master
2013-09-06 17:53:01 +05:30
Matei Zaharia
f586c8ef38
Updated LICENSE with third-party licenses
2013-09-02 16:43:06 -07:00
Matei Zaharia
0a8cc30921
Move some classes to more appropriate packages:
...
* RDD, *RDDFunctions -> org.apache.spark.rdd
* Utils, ClosureCleaner, SizeEstimator -> org.apache.spark.util
* JavaSerializer, KryoSerializer -> org.apache.spark.serializer
2013-09-01 14:13:16 -07:00
Matei Zaharia
46eecd110a
Initial work to rename package to org.apache.spark
2013-09-01 14:13:13 -07:00
Benjamin Hindman
7bdafa918a
Format cleanup.
2013-07-30 17:01:00 -07:00
Benjamin Hindman
f6f46455eb
Added property 'spark.executor.uri' for launching on Mesos without
...
requiring Spark to be installed. Using 'make_distribution.sh' a user
can put a Spark distribution at a URI supported by Mesos (e.g.,
'hdfs://...') and then set that when launching their job. Also added
SPARK_EXECUTOR_URI for the REPL.
2013-07-29 23:32:52 -07:00
Matei Zaharia
af3c9d5042
Add Apache license headers and LICENSE and NOTICE files
2013-07-16 17:21:33 -07:00
Prashant Sharma
6e6d94ffdf
Added add jars functionality to new repl, which was dropped while merging with old.
2013-07-12 11:55:16 +05:30
Prashant Sharma
ca249eea50
Removed an unnecessary test case
2013-07-11 18:31:36 +05:30
Prashant Sharma
a5f1f6a907
Merge branch 'master' into master-merge
...
Conflicts:
core/pom.xml
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/RDDCheckpointData.scala
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/Utils.scala
core/src/main/scala/spark/api/python/PythonRDD.scala
core/src/main/scala/spark/deploy/client/Client.scala
core/src/main/scala/spark/deploy/master/MasterWebUI.scala
core/src/main/scala/spark/deploy/worker/Worker.scala
core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/ZippedRDD.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockManagerMaster.scala
core/src/main/scala/spark/storage/BlockManagerMasterActor.scala
core/src/main/scala/spark/storage/BlockManagerUI.scala
core/src/main/scala/spark/util/AkkaUtils.scala
core/src/test/scala/spark/SizeEstimatorSuite.scala
pom.xml
project/SparkBuild.scala
repl/src/main/scala/spark/repl/SparkILoop.scala
repl/src/test/scala/spark/repl/ReplSuite.scala
streaming/src/main/scala/spark/streaming/StreamingContext.scala
streaming/src/main/scala/spark/streaming/api/java/JavaStreamingContext.scala
streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
streaming/src/main/scala/spark/streaming/util/MasterFailureTest.scala
2013-07-03 11:43:26 +05:30
Matei Zaharia
2bd04c3513
Formatting
2013-06-25 18:37:14 -04:00
Matei Zaharia
f2263350ed
Added a local-cluster mode test to ReplSuite
2013-06-25 18:35:35 -04:00
Matei Zaharia
0e0f9d3069
Fix search path for REPL class loader to really find added JARs
2013-06-22 17:44:04 -07:00
Matei Zaharia
b5df1cd668
ADD_JARS environment variable for spark-shell
2013-06-22 17:14:44 -07:00
Matei Zaharia
7902baddc7
Update ASM to version 4.0
2013-06-19 13:34:30 +02:00
Matei Zaharia
96c895f653
Fix StandaloneClusterReplSuite to allow running multiple tests
2013-06-08 14:21:41 -07:00
Prashant Sharma
24bbf318b3
Fixied other warnings
2013-04-29 19:56:28 +05:30
Mridul Muralidharan
afee902443
Attempt to fix streaming test failures after yarn branch merge
2013-04-28 22:26:45 +05:30
Prashant Sharma
ad88f083a6
scala 2.10 and master merge
2013-04-24 18:08:26 +05:30
Prashant Sharma
185bb9525a
Manually merged scala-2.10 and master
2013-04-22 14:14:03 +05:30
Prashant Sharma
bf5fc07379
Added more tests
2013-04-19 13:51:16 +05:30
Prashant Sharma
36ccb35371
Changed spark context as lazy val to become just val.
2013-04-16 11:13:51 +05:30
Prashant Sharma
19b0256ae4
Added standalone cluster repl suite
2013-04-15 19:49:40 +05:30
Prashant Sharma
f31e41c270
Added class wrappers instead of object and incorporated most of matei comments
2013-04-10 15:08:01 +05:30
Prashant Sharma
b67c638b85
Made shell loading synchronous as async loading confuses with out of order status messages
2013-04-10 14:48:49 +05:30
Matei Zaharia
65caa8f711
Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0'
...
Conflicts:
docs/_config.yml
project/SparkBuild.scala
2013-04-08 12:43:17 -04:00
Jey Kottalam
bc8ba222ff
Bump development version to 0.8.0
2013-03-28 15:42:01 -07:00
Prashant Sharma
5a080acdb8
Fixed broken tests by last commit for repl.
2013-03-25 10:43:45 +05:30
Prashant Sharma
d9f34e505d
Ctrl-D hang bug fixed!
2013-03-20 00:18:04 +05:30
prashant
432a227320
fixed autocompletion apparent hang due to logging
2013-03-19 12:29:08 +05:30
Prashant Sharma
15530c2b23
porting of repl to scala-2.10
2013-03-17 10:47:17 +05:30
Matei Zaharia
db9b90fdbd
Change version to 0.7.1-SNAPSHOT for development branch
2013-02-27 09:15:26 -08:00
Stephen Haberman
7dfb82a992
Replace old 'master' term with 'driver'.
2013-01-25 11:03:00 -06:00
Tathagata Das
4719e6d8fe
Changed locations for unit test logs.
2013-01-07 16:06:07 -08:00
Thomas Dudziak
4af6cad37a
Fixed repl maven build to produce artifacts with the appropriate hadoop classifier and extracted repl fat-jar and debian packaging into a separate project to make Maven happy
2012-12-18 12:08:19 -08:00
Thomas Dudziak
811a32257b
Added maven and debian build files
2012-11-20 16:19:51 -08:00
Matei Zaharia
0967e71a00
Bump up version to 0.7.0-SNAPSHOT for master branch
2012-10-22 11:49:42 -07:00
Matei Zaharia
902a608187
Update version to 0.6.1-SNAPSHOT to show this is in development
2012-10-22 11:43:57 -07:00
Matei Zaharia
ce6b5a3ee5
Uncomment Maven publishing stuff and set version to 0.6.0
2012-10-13 15:55:39 -07:00
Matei Zaharia
eca570f66a
Removed the need to sleep in tests due to waiting for Akka to shut down
2012-10-07 00:17:59 -07:00
Matei Zaharia
74a9244255
Write all unit test output to a file
2012-10-01 15:07:42 -07:00
Matei Zaharia
83143f9a5f
Fixed several bugs that caused weird behavior with files in spark-shell:
...
- SizeEstimator was following through a ClassLoader field of Hadoop
JobConfs, which referenced the whole interpreter, Scala compiler, etc.
Chaos ensued, giving an estimated size in the tens of gigabytes.
- Broadcast variables in local mode were only stored as MEMORY_ONLY and
never made accessible over a server, so they fell out of the cache when
they were deemed too large and couldn't be reloaded.
2012-09-30 21:19:39 -07:00
Matei Zaharia
2c16ae36d7
Set log level in tests to WARN
2012-08-23 20:38:14 -07:00
Matei Zaharia
e72afdb817
Some refactoring to make cluster scheduler pluggable.
2012-07-06 15:23:26 -07:00
Matei Zaharia
08579ffa11
Update version number for dev branch
2012-06-15 23:55:43 -07:00
Matei Zaharia
4449eb9783
Changed version in master branch to 0.5.1-SNAPSHOT for further
...
development.
2012-06-13 22:26:14 -04:00
Matei Zaharia
4971e0f547
Updated version number to 0.5.0
2012-06-12 13:41:57 -04:00
Matei Zaharia
dbc3c86ae3
Merge branch 'master' into mesos-0.9
...
Conflicts:
core/src/main/scala/spark/Executor.scala
2012-06-03 17:44:04 -07:00
Reynold Xin
d176422586
Make spark.repl.Main.interp_ publicly accessible (so Shark can get rid
...
of a weird file dediated to accessing this variable).
2012-05-30 18:40:10 -07:00
Matei Zaharia
08cda89e8a
Further fixes to how Mesos is found and used
2012-03-17 13:39:14 -07:00
Matei Zaharia
63da22c025
Update REPL code to use our own version of JLineReader, which fixes #89 .
...
I'm not entirely sure why this broke in the jump from Scala 2.9.0.1 to
2.9.1 -- maybe something about name resolution changed?
2011-11-07 20:16:25 -08:00
Ismael Juma
483f724d62
Upgrade to Scala 2.9.1.
...
Interestingly, the version in Maven is 2.9.1, but SBT outputs file to the 2.9.1.final
directory inside target.
A couple of small changes in SparkIMain were also required.
All tests pass and ./spark-shell launches successfully.
2011-08-31 10:43:05 +01:00
Ismael Juma
065043a14f
Use process instead of main as the latter is deprecated.
2011-08-02 10:26:03 +01:00
Matei Zaharia
cf8f5de61b
Merge branch 'master' into scala-2.9
...
Conflicts:
project/build.properties
repl/src/main/scala/spark/repl/SparkInterpreterLoop.scala
2011-07-14 17:48:56 -04:00
Matei Zaharia
02678724a4
Update version number to 0.4-SNAPSHOT
2011-07-14 17:47:39 -04:00
Matei Zaharia
b187675b68
Print version number 0.3 in REPL
2011-06-26 18:27:01 -07:00
Matei Zaharia
b49d1be65b
Ensure logging is initialized before any Spark threads run in the REPL
2011-05-31 23:54:48 -07:00
Matei Zaharia
90f924202b
Another fix ported forward for the REPL
2011-05-31 23:11:49 -07:00
Matei Zaharia
73975d7491
Further fixes to interpreter (adding in some code generation changes I
...
missed before and setting SparkEnv properly on the threads that execute
each line in the 2.9 interpreter).
2011-05-31 22:05:24 -07:00
Matei Zaharia
d52660c969
Ported code generation changes from 2.8 interpreter (to use a class for
...
each line's object rather than a singleton object so that we can ship
these classes to worker nodes). This is pretty hairy stuff, which would
be nice to avoid in the future by integrating with the interpreter some
other way.
2011-05-31 19:23:15 -07:00
Matei Zaharia
bcce6e8d01
Various work to use the 2.9 interpreter
2011-05-31 17:31:51 -07:00
Ismael Juma
1396678baa
Move REPL classes to separate module.
2011-05-27 11:22:50 +01:00