Matei Zaharia
d52660c969
Ported code generation changes from 2.8 interpreter (to use a class for
...
each line's object rather than a singleton object so that we can ship
these classes to worker nodes). This is pretty hairy stuff, which would
be nice to avoid in the future by integrating with the interpreter some
other way.
2011-05-31 19:23:15 -07:00
Matei Zaharia
beb9c117f0
Merge branch 'master' into scala-2.9
...
Conflicts:
project/build/SparkProject.scala
2011-05-31 19:23:07 -07:00
Matei Zaharia
bcce6e8d01
Various work to use the 2.9 interpreter
2011-05-31 17:31:51 -07:00
Matei Zaharia
8b0390d344
Instantiate NullWritable properly in HadoopFile
2011-05-30 23:54:14 -07:00
Matei Zaharia
4096c2287e
Various fixes
2011-05-29 18:46:01 -07:00
Matei Zaharia
ef706ae959
Merge branch 'master' into new-rdds-protobuf
...
Conflicts:
run
2011-05-29 16:20:23 -07:00
Matei Zaharia
c501cff924
Executor was looking for the wrong constructor for ExecutorClassLoader
2011-05-29 16:15:59 -07:00
Matei Zaharia
50ac1d2a40
Merge remote-tracking branch 'ijuma/issue51'
2011-05-29 15:41:12 -07:00
Ismael Juma
0c62ee4321
Depend on jetty-server in compile scope and upgrade to 7.4.2.
...
As Matei described: "We're using Jetty to run an HTTP server, not to embed Spark
in a webapp"
2011-05-29 20:12:50 +01:00
Matei Zaharia
22c8b84d8b
Merge pull request #53 from ijuma/master
...
Use explicit asInstanceOf instead of misleading unchecked pattern matching.
2011-05-27 10:20:54 -07:00
Ismael Juma
2c691295bf
SCALA_VERSION in run should be 2.9.0-1
2011-05-27 14:59:23 +01:00
Ismael Juma
1d75c6060a
Update to Scala 2.9.0-1 and disable repl module for now.
...
The repl module requires more complex work.
2011-05-27 14:59:23 +01:00
Ismael Juma
e3b323321d
Use ManagedStyle.Maven.
2011-05-27 14:56:01 +01:00
Ismael Juma
3a6b0b8a57
Publish javadoc and sources.
2011-05-27 14:55:51 +01:00
Ismael Juma
59f1f42a9a
Update run to work with SBT managed dependencies and the newly introduced repl module.
2011-05-27 11:22:59 +01:00
Ismael Juma
3af6003c87
Update sbt to 0.7.7.
2011-05-27 11:22:59 +01:00
Ismael Juma
1396678baa
Move REPL classes to separate module.
2011-05-27 11:22:50 +01:00
Ismael Juma
3e8114ddbd
Change project.organization to org.spark-project to fit Maven convention.
2011-05-27 11:22:10 +01:00
Ismael Juma
7b7dfdb085
Set project.version to 0.3-SNAPSHOT.
2011-05-27 11:22:10 +01:00
Ismael Juma
051da8b4ad
Delete liblzf from lib as it's no longer used.
2011-05-27 11:22:10 +01:00
Ismael Juma
ae1a1f91f1
Remove several dependencies from git and configure them as SBT managed dependencies.
...
Upgrade some of the dependencies while at it.
2011-05-27 11:22:01 +01:00
Ismael Juma
164ef4c751
Use explicit asInstanceOf instead of misleading unchecked pattern matching.
...
Also enable -unchecked warnings in SBT build file.
2011-05-27 07:57:10 +01:00
Matei Zaharia
cfbe2da1a6
Merge pull request #52 from ijuma/master
...
Fix deprecations when compiled with Scala 2.8.1
2011-05-26 23:53:10 -07:00
Ismael Juma
89c8ea2bb2
Replace deprecated -
and --
with suggested filterNot (which is uglier).
2011-05-26 22:22:37 +01:00
Ismael Juma
94f05683bd
Replace deprecated first
with head
.
2011-05-26 22:13:41 +01:00
Ismael Juma
26000af4fa
Replace deprecated fromFunction with either tabulate or fill.
...
tabulate used if indexed used by function and fill otherwise.
2011-05-26 22:12:11 +01:00
Ismael Juma
0b6a862b68
Use math instead of Math as the latter is deprecated.
2011-05-26 22:06:36 +01:00
Ismael Juma
1f27d94c48
Use Array.iterator instead of Iterator.fromArray as the latter is deprecated.
2011-05-26 22:04:42 +01:00
Ismael Juma
1993a8e556
Use += instead of + for mutable sequences as the latter is deprecated.
2011-05-26 21:59:48 +01:00
Ismael Juma
222729171e
Upgrade sbt-idea to 0.4.0.
2011-05-26 21:48:08 +01:00
Matei Zaharia
f707856522
Removed java-opts.template
2011-05-24 15:59:01 -07:00
root
5ef938615f
Initial work on making stuff compile with protobuf Mesos
2011-05-24 22:27:08 +00:00
Matei Zaharia
cec427e777
Fixed a bug with preferred locations having changed meaning in new RDDs
2011-05-22 17:12:29 -07:00
Matei Zaharia
4c888b2933
Fix queue type for executor
2011-05-22 16:42:05 -07:00
Matei Zaharia
bea3a33012
doc tweak
2011-05-22 16:03:41 -07:00
Matei Zaharia
9bde5a54cb
class loader fix
2011-05-22 16:00:41 -07:00
Matei Zaharia
91c07a33d9
Various fixes to serialization
2011-05-21 22:50:08 -07:00
Matei Zaharia
f61b61c4ac
Merge branch 'master' into new-rdds
2011-05-21 21:25:58 -07:00
Matei Zaharia
24a1e7f838
Scheduler can now recover from lost map outputs
2011-05-20 00:19:53 -07:00
Matei Zaharia
82329b0b28
Updated scheduler to support running on just some partitions of final RDD
2011-05-19 12:47:09 -07:00
Matei Zaharia
328e51b693
Various minor fixes
2011-05-19 11:19:25 -07:00
Matei Zaharia
fd1d255821
Stop objectifying various trackers, caches, etc.
2011-05-17 12:41:13 -07:00
Matei Zaharia
4db50e26c7
Fixed unit tests by making them clean up the SparkContext after use and
...
thus clean up the various singletons (RDDCache, MapOutputTracker, etc).
This isn't perfect yet (ideally we shouldn't use singleton objects at
all) but we can fix that later.
2011-05-13 12:03:58 -07:00
Matei Zaharia
aca8150c52
Ensure that AddedToCache messages make it home before tasks finish
2011-05-13 11:43:52 -07:00
Matei Zaharia
16c886a581
Optimization for count()
2011-05-13 10:41:34 -07:00
Matei Zaharia
4b1f0f1ce4
Merge pull request #48 from ankurdave/bagel-new
...
Bagel: Large-scale graph processing on Spark
2011-05-12 21:34:38 -07:00
Ankur Dave
f40a0898a7
Rename bagel to spark.bagel and Pregel to Bagel
2011-05-09 15:23:21 -07:00
Matei Zaharia
7e20648914
Upgraded to SBT 0.7.5
2011-05-09 14:48:39 -07:00
Matei Zaharia
4bedf5b13a
Merge pull request #47 from ankurdave/cache-to-disk
...
Merging in Ankur's code for a cache that spills to disk
2011-05-09 14:22:56 -07:00
Ankur Dave
c1104058c6
Move shortest path and PageRank to bagel.examples
2011-05-03 18:53:58 -07:00