Matei Zaharia
095dd9c444
Merge pull request #62 from ogrisel/cogroup-test
...
Add missing test for RDD.groupWith
2011-06-23 10:12:24 -07:00
Matei Zaharia
e8e35d5fb5
Merge pull request #61 from ogrisel/better-readme
...
Better readme
2011-06-23 10:11:14 -07:00
Olivier Grisel
7ef48a4df0
typo
2011-06-23 02:28:17 +02:00
Olivier Grisel
5b9e0a126d
format
2011-06-23 02:27:14 +02:00
Olivier Grisel
236bcd0d9b
Markdown rendering for the toplevel README.md to improve readability on github
2011-06-23 02:24:04 +02:00
Olivier Grisel
005d1605a4
add missing test for RDD.groupWith
2011-06-23 02:10:52 +02:00
Matei Zaharia
214250016a
Added simple version of lookup
2011-06-20 11:59:16 -07:00
Matei Zaharia
23b1c309fb
Added pipe() operation on RDDs for mapping through a shell command.
2011-06-19 23:05:19 -07:00
Matei Zaharia
a413b8e59d
Merge pull request #59 from ijuma/master
...
Move managedStyle to SparkProject
2011-06-07 00:41:50 -07:00
Ismael Juma
1ad4dcd3de
Move managedStyle to SparkProject.
...
I had added it to DepJar by mistake.
2011-06-02 14:06:54 +01:00
Matei Zaharia
9bb448a151
Catch Throwable instead of Exception in LocalScheduler and Executor. Fixes #57 .
2011-06-01 11:45:47 -07:00
Matei Zaharia
850fe3274e
Make the runJob API public. Fixes #56 .
2011-06-01 11:38:44 -07:00
Matei Zaharia
5166d76843
Ensure logging is initialized before spawning any threads to fix issue #45
2011-05-31 23:47:32 -07:00
Matei Zaharia
96daa31a01
Pass quoted arguments properly to run
2011-05-31 23:34:16 -07:00
Matei Zaharia
7862995569
Give SBT a bit more memory so it can do a update / compile / test in one JVM
2011-05-31 23:33:47 -07:00
Matei Zaharia
8b0390d344
Instantiate NullWritable properly in HadoopFile
2011-05-30 23:54:14 -07:00
Matei Zaharia
c501cff924
Executor was looking for the wrong constructor for ExecutorClassLoader
2011-05-29 16:15:59 -07:00
Matei Zaharia
50ac1d2a40
Merge remote-tracking branch 'ijuma/issue51'
2011-05-29 15:41:12 -07:00
Ismael Juma
0c62ee4321
Depend on jetty-server in compile scope and upgrade to 7.4.2.
...
As Matei described: "We're using Jetty to run an HTTP server, not to embed Spark
in a webapp"
2011-05-29 20:12:50 +01:00
Matei Zaharia
22c8b84d8b
Merge pull request #53 from ijuma/master
...
Use explicit asInstanceOf instead of misleading unchecked pattern matching.
2011-05-27 10:20:54 -07:00
Ismael Juma
e3b323321d
Use ManagedStyle.Maven.
2011-05-27 14:56:01 +01:00
Ismael Juma
3a6b0b8a57
Publish javadoc and sources.
2011-05-27 14:55:51 +01:00
Ismael Juma
59f1f42a9a
Update run to work with SBT managed dependencies and the newly introduced repl module.
2011-05-27 11:22:59 +01:00
Ismael Juma
3af6003c87
Update sbt to 0.7.7.
2011-05-27 11:22:59 +01:00
Ismael Juma
1396678baa
Move REPL classes to separate module.
2011-05-27 11:22:50 +01:00
Ismael Juma
3e8114ddbd
Change project.organization to org.spark-project to fit Maven convention.
2011-05-27 11:22:10 +01:00
Ismael Juma
7b7dfdb085
Set project.version to 0.3-SNAPSHOT.
2011-05-27 11:22:10 +01:00
Ismael Juma
051da8b4ad
Delete liblzf from lib as it's no longer used.
2011-05-27 11:22:10 +01:00
Ismael Juma
ae1a1f91f1
Remove several dependencies from git and configure them as SBT managed dependencies.
...
Upgrade some of the dependencies while at it.
2011-05-27 11:22:01 +01:00
Ismael Juma
164ef4c751
Use explicit asInstanceOf instead of misleading unchecked pattern matching.
...
Also enable -unchecked warnings in SBT build file.
2011-05-27 07:57:10 +01:00
Matei Zaharia
cfbe2da1a6
Merge pull request #52 from ijuma/master
...
Fix deprecations when compiled with Scala 2.8.1
2011-05-26 23:53:10 -07:00
Ismael Juma
89c8ea2bb2
Replace deprecated -
and --
with suggested filterNot (which is uglier).
2011-05-26 22:22:37 +01:00
Ismael Juma
94f05683bd
Replace deprecated first
with head
.
2011-05-26 22:13:41 +01:00
Ismael Juma
26000af4fa
Replace deprecated fromFunction with either tabulate or fill.
...
tabulate used if indexed used by function and fill otherwise.
2011-05-26 22:12:11 +01:00
Ismael Juma
0b6a862b68
Use math instead of Math as the latter is deprecated.
2011-05-26 22:06:36 +01:00
Ismael Juma
1f27d94c48
Use Array.iterator instead of Iterator.fromArray as the latter is deprecated.
2011-05-26 22:04:42 +01:00
Ismael Juma
1993a8e556
Use += instead of + for mutable sequences as the latter is deprecated.
2011-05-26 21:59:48 +01:00
Ismael Juma
222729171e
Upgrade sbt-idea to 0.4.0.
2011-05-26 21:48:08 +01:00
Matei Zaharia
f707856522
Removed java-opts.template
2011-05-24 15:59:01 -07:00
Matei Zaharia
cec427e777
Fixed a bug with preferred locations having changed meaning in new RDDs
2011-05-22 17:12:29 -07:00
Matei Zaharia
4c888b2933
Fix queue type for executor
2011-05-22 16:42:05 -07:00
Matei Zaharia
bea3a33012
doc tweak
2011-05-22 16:03:41 -07:00
Matei Zaharia
9bde5a54cb
class loader fix
2011-05-22 16:00:41 -07:00
Matei Zaharia
91c07a33d9
Various fixes to serialization
2011-05-21 22:50:08 -07:00
Matei Zaharia
f61b61c4ac
Merge branch 'master' into new-rdds
2011-05-21 21:25:58 -07:00
Matei Zaharia
24a1e7f838
Scheduler can now recover from lost map outputs
2011-05-20 00:19:53 -07:00
Matei Zaharia
82329b0b28
Updated scheduler to support running on just some partitions of final RDD
2011-05-19 12:47:09 -07:00
Matei Zaharia
328e51b693
Various minor fixes
2011-05-19 11:19:25 -07:00
Matei Zaharia
fd1d255821
Stop objectifying various trackers, caches, etc.
2011-05-17 12:41:13 -07:00
Matei Zaharia
4db50e26c7
Fixed unit tests by making them clean up the SparkContext after use and
...
thus clean up the various singletons (RDDCache, MapOutputTracker, etc).
This isn't perfect yet (ideally we shouldn't use singleton objects at
all) but we can fix that later.
2011-05-13 12:03:58 -07:00