Matei Zaharia
04c2d6a60c
stuff
2011-03-06 19:27:03 -08:00
Matei Zaharia
0fb691dd28
Various fixes to get MesosScheduler working with new RDDs
2011-03-06 16:16:38 -08:00
Matei Zaharia
1df5a65a01
Pass cache locations correctly to DAGScheduler.
2011-03-06 12:16:38 -08:00
Matei Zaharia
e1436f1eaa
Merge remote branch 'origin/master' into new-rdds
2011-03-06 11:11:47 -08:00
Matei Zaharia
370b95816f
Added sampling for large arrays in SizeEstimator
2011-03-06 11:11:20 -08:00
Matei Zaharia
a789e9aaea
Merge remote branch 'origin/master' into new-rdds
2011-03-01 10:33:37 -08:00
Matei Zaharia
021c50a8d4
Remove unnecessary lock which was there to work around a bug in
...
Configuration in Hadoop 0.20.0
2011-03-01 10:28:38 -08:00
Matei Zaharia
adaba4d550
Removed old slf4j jars that came with Hadoop
2011-03-01 10:28:21 -08:00
Matei Zaharia
447debb771
Updated Hadoop to 0.20.2 to include some bug fixes
2011-03-01 10:27:48 -08:00
Matei Zaharia
9e59afd710
More work on new RDD design
2011-02-27 19:15:52 -08:00
Matei Zaharia
f38f86d59e
More stuff
2011-02-27 14:27:12 -08:00
Matei Zaharia
2e6023f2bf
stuff
2011-02-26 23:41:44 -08:00
Matei Zaharia
309367c477
Initial work towards new RDD design
2011-02-26 23:15:33 -08:00
Matei Zaharia
dc24aecd8f
Close record readers in HadoopFile after finishing a split
2011-02-10 12:07:48 -08:00
Matei Zaharia
62f1c6f5a8
Remove build.properties from version control
2011-02-09 11:52:56 -08:00
Matei Zaharia
d3df963a13
Brought in some reorganization of build file from Hive branch
2011-02-08 21:27:36 -08:00
Matei Zaharia
e8df4bbd40
Added more SBT stuff to gitignore
2011-02-08 17:06:07 -08:00
Matei Zaharia
26b77aece9
Increased SBT mem to 700 MB so that unit tests run more nicely
2011-02-08 17:03:28 -08:00
Matei Zaharia
99f3f23efa
Changed default shuffle to LocalFileShuffle because it's way faster for small files
2011-02-08 17:03:03 -08:00
Matei Zaharia
f4f7aa2ab2
formatting
2011-02-08 16:39:17 -08:00
Matei Zaharia
ee60aaa0f5
Added a pointer to wiki in readme
2011-02-08 16:38:10 -08:00
Matei Zaharia
c1c766a93c
Updated readme
2011-02-02 19:21:49 -08:00
Matei Zaharia
50df43bf7b
Added SBT target for building a single JAR with Spark Core and its
...
dependencies
2011-02-02 19:08:14 -08:00
Matei Zaharia
a11fe23017
Moved examples to spark.examples package
2011-02-02 16:30:27 -08:00
Matei Zaharia
82170608b1
Added IntelliJ's build directory to gitignore
2011-02-02 00:30:29 -08:00
Matei Zaharia
ec28b607fd
Merge branch 'master' into sbt
...
Conflicts:
Makefile
core/src/main/java/spark/compress/lzf/LZF.java
core/src/main/java/spark/compress/lzf/LZFInputStream.java
core/src/main/java/spark/compress/lzf/LZFOutputStream.java
core/src/main/native/spark_compress_lzf_LZF.c
run
2011-02-02 00:25:54 -08:00
Matei Zaharia
7f74ee99f6
Added support for IntelliJ IDEA
2011-02-02 00:08:13 -08:00
Matei Zaharia
e5c4cd8a5e
Made examples and core subprojects
2011-02-01 15:11:08 -08:00
Matei Zaharia
817e722321
Merge branch 'master' of github.com:mesos/spark
2010-12-15 19:40:35 -08:00
Matei Zaharia
14c29c1b14
Fixed import
2010-12-15 19:40:27 -08:00
Mosharaf Chowdhury
5cafdd7ba2
Removed some unused imports from Broadcast.scala
2010-12-15 19:11:23 -08:00
Mosharaf Chowdhury
be0ce57de2
- Fixed an compilation error due to wrong 'import' of legacy lzf libraries in DfsBroadcast.scala
...
- Updated to use ning libraries.
- Passes all unit tests
2010-12-15 18:34:27 -08:00
Matei Zaharia
5c222dbe28
Merge branch 'master' into mos-bt
...
Conflicts:
src/scala/spark/Broadcast.scala
2010-12-15 10:57:39 -08:00
Mosharaf Chowdhury
0a5c24ae3d
- Default broadcast mechanism is set to DfsBroadcast
...
- Configuration parameters are renamed to follow our convention
- Master now automatically supplies its hostAddress instead of reading from config file
- sendBroadcast has been removed from the Broadcast trait
2010-12-13 14:36:39 -08:00
Timothy Hunter
34395730db
Someone forgot to pass the parameters: fixes SPARK_MEM set from main script but not passed to executor.
2010-12-12 13:30:49 -08:00
Matei Zaharia
0d895ba636
Added BSD license
2010-12-07 10:32:17 -08:00
Mosharaf Chowdhury
06dc4a5148
- Removed config files from git's control.
...
- Changed DfsShuffle to default in RDD.scala.
2010-12-07 10:17:47 -08:00
Joshua Hartman
799c1b19f5
Adding license file for compress-lzf
2010-12-07 08:30:29 -08:00
Joshua Hartman
2fb849502f
Replacing the native lzf compression code with the ning open-source compress-lzf library. (Apache 2.0 liscense)
2010-12-05 21:20:15 -08:00
Mosharaf Chowdhury
1013a01469
Made the root Broadcast object initilization method 'synchronized'.
2010-12-02 02:36:21 -08:00
Mosharaf Chowdhury
a69638d98d
Removed deprecated repl file ClassServer.scala
2010-11-30 18:45:04 -08:00
Mosharaf Chowdhury
b7dda4c5bc
Merge branch 'multi-tracker' into mos-bt
...
Conflicts:
conf/java-opts
src/scala/spark/Broadcast.scala
src/scala/spark/DfsBroadcast.scala
src/scala/spark/SparkContext.scala
2010-11-30 18:29:38 -08:00
Mosharaf Chowdhury
815ecd349a
Made Broadcast Pluggable. Finally!
2010-11-30 18:08:49 -08:00
Mosharaf Chowdhury
ea28a30e36
Makes transfer faster for small swarms (with size < MaxRxPeers)
2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury
191af65fb7
- Using the new Cache implementation.
2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury
c9cad03c31
- Using the new Cache implementation.
...
- Removed unused code related to dualMode (deprecated).
2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury
73714da568
Made LocalFileShuffle to be the default.
2010-11-29 19:19:21 -08:00
Mosharaf Chowdhury
c62f2fefa5
Made LocalFileShuffle to be the default.
2010-11-29 19:17:01 -08:00
Mosharaf Chowdhury
15f1f6865d
Variable renaming.
2010-11-29 14:15:36 -08:00
Mosharaf Chowdhury
bb3e7fbf61
Variable renaming.
2010-11-29 14:13:35 -08:00