Mosharaf Chowdhury
119e50c7b9
Conflict fixed
2012-10-02 22:25:39 -07:00
Mosharaf Chowdhury
ff813e4380
Merge remote-tracking branch 'upstream/dev' into dev
2012-10-02 22:17:17 -07:00
Matei Zaharia
87f4451f20
Simplify README even further
2012-10-02 22:14:40 -07:00
Matei Zaharia
a17e66a689
Merge pull request #242 from pwendell/readme-update
...
Changing version of Scala in README
2012-10-02 22:12:08 -07:00
Patrick Wendell
89be3c3e76
Changing version of Scala in README
2012-10-02 22:10:23 -07:00
Matei Zaharia
a3e54de6ca
Merge pull request #241 from shivaram/tuning-doc
...
First cut at adding documentation for GC tuning
2012-10-02 21:26:54 -07:00
Matei Zaharia
626f701931
Merge pull request #240 from dennybritz/private_classes
...
Package-Private Classes
2012-10-02 21:24:32 -07:00
Shivaram Venkataraman
3d2b900b08
First cut at adding documentation for GC tuning
2012-10-02 20:07:18 -07:00
Denny
0361353a70
Make Java API abstract wrapped functions private
2012-10-02 20:02:53 -07:00
Denny
b9badcd5bd
accidentially removed trait
2012-10-02 19:35:07 -07:00
Denny
18a1faedf6
Stylistic changes and Public Accumulable and Broadcast
2012-10-02 19:28:37 -07:00
Denny
b7a913e1fa
Make dependency classes public - used by spark
2012-10-02 19:04:23 -07:00
Denny
4d9f4b01af
Make classes package private
2012-10-02 19:00:19 -07:00
Matei Zaharia
97cbd699d7
Merge branch 'dev' of github.com:mesos/spark into dev
2012-10-02 17:31:01 -07:00
Matei Zaharia
5fda59ab99
Added a test for overly large blocks in memory store
2012-10-02 17:30:40 -07:00
Matei Zaharia
6098f7e87a
Fixed cache replacement behavior of BlockManager:
...
- Partitions that get dropped to disk will now be loaded back into RAM
after they're accessed again
- Same-RDD rule for cache replacement is now implemented (don't drop
partitions from an RDD to make room for other partitions from itself)
- Items stored as MEMORY_AND_DISK go into memory only first, instead of
being eagerly written out to disk
- MemoryStore.ensureFreeSpace is called within a lock on the writer
thread to prevent race conditions (this can still be optimized to
allow multiple concurrent calls to it but it's a start)
- MemoryStore does not accept blocks larger than its limit
2012-10-02 17:25:38 -07:00
Matei Zaharia
6112b1a83c
Don't build an assembly for the REPL
2012-10-02 17:08:16 -07:00
Matei Zaharia
c8ca6bc59b
Merge pull request #238 from rxin/dev
...
Allow whitespaces in cluster URL configuration for local cluster.
2012-10-02 16:30:40 -07:00
Reynold Xin
7997585616
Added a check to make sure SPARK_MEM <= memoryPerSlave for local cluster
...
mode.
2012-10-02 15:45:25 -07:00
Reynold Xin
0898a21b95
Merge branch 'dev' of https://github.com/mesos/spark into dev
2012-10-02 13:08:01 -07:00
Matei Zaharia
a925754675
Place Spray in front of Cloudera in Maven search path
2012-10-02 12:02:00 -07:00
Matei Zaharia
22684653a5
Revert "Place Spray repo ahead of Cloudera in Maven search path"
...
This reverts commit 42e0a68082
2012-10-02 12:01:32 -07:00
Reynold Xin
b8cd681169
Allow whitespaces in cluster URL configuration for local cluster.
2012-10-02 11:52:12 -07:00
Matei Zaharia
42e0a68082
Place Spray repo ahead of Cloudera in Maven search path
2012-10-02 11:37:19 -07:00
Matei Zaharia
dde273f43a
Merge branch 'dev' of github.com:mesos/spark into dev
2012-10-02 11:35:51 -07:00
Mosharaf Chowdhury
288e1c99ab
Merge remote-tracking branch 'upstream/dev' into dev
2012-10-01 22:47:05 -07:00
Matei Zaharia
b9fb8d6463
Include date in folder name for Spark local dir.
2012-10-01 15:55:16 -07:00
Matei Zaharia
c06b0c7537
Merge pull request #235 from pwendell/publish-local-maven
...
publish-local should go to maven + ivy by default
2012-10-01 15:49:16 -07:00
Patrick Wendell
6fee76d6d5
publish-local should go to maven + ivy by default
2012-10-01 15:34:47 -07:00
Matei Zaharia
bc881e4798
Merge branch 'dev' of github.com:mesos/spark into dev
2012-10-01 15:21:56 -07:00
Matei Zaharia
802aa8aef9
Some bug fixes and logging fixes for broadcast.
2012-10-01 15:20:42 -07:00
Matei Zaharia
c1db5a849b
Ignore file spark-tests.log in git
2012-10-01 15:08:20 -07:00
Matei Zaharia
74a9244255
Write all unit test output to a file
2012-10-01 15:07:42 -07:00
Matei Zaharia
8981804c71
Merge pull request #233 from rxin/dev
...
Fixed #232 : DirectBuffer's cleaner was empty and Spark tried to invoke clean on it.
2012-10-01 14:30:39 -07:00
Reynold Xin
f264153162
Fixed #232 : DirectBuffer's cleaner was empty and Spark tried to invoke
...
clean on it.
2012-10-01 14:07:34 -07:00
Matei Zaharia
3b348f909d
Improve log messages from BlockManager
2012-10-01 12:01:38 -07:00
Matei Zaharia
22b6b16e56
Merge branch 'dev' of github.com:mesos/spark into dev
2012-10-01 10:57:32 -07:00
Matei Zaharia
0b84871dbc
Remove some printlns in tests
2012-10-01 10:57:26 -07:00
Matei Zaharia
53f90d0f0e
Use underscores instead of colons in RDD IDs
2012-10-01 10:48:53 -07:00
Matei Zaharia
39363b5185
Merge pull request #231 from rxin/dev
...
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 10:45:03 -07:00
Reynold Xin
5783236ae6
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 00:17:13 -07:00
Matei Zaharia
2314132d57
Added a (failing) test for LRU with MEMORY_AND_DISK.
2012-09-30 22:52:16 -07:00
Matei Zaharia
3128c57f90
Simplified Class / ClassLoader test
2012-09-30 21:48:27 -07:00
Matei Zaharia
83143f9a5f
Fixed several bugs that caused weird behavior with files in spark-shell:
...
- SizeEstimator was following through a ClassLoader field of Hadoop
JobConfs, which referenced the whole interpreter, Scala compiler, etc.
Chaos ensued, giving an estimated size in the tens of gigabytes.
- Broadcast variables in local mode were only stored as MEMORY_ONLY and
never made accessible over a server, so they fell out of the cache when
they were deemed too large and couldn't be reloaded.
2012-09-30 21:19:39 -07:00
Matei Zaharia
fd0374b9de
Comment
2012-09-29 21:43:06 -07:00
Matei Zaharia
5718cef2a4
Removed Logging trait from CoalescedRDD since we don't log anything
2012-09-29 21:40:43 -07:00
Matei Zaharia
4a74e8635c
Merge pull request #228 from rxin/dev
...
Added mapPartitionsWithSplit to the programming guide.
2012-09-29 21:33:38 -07:00
Matei Zaharia
143ef4f90d
Added a CoalescedRDD class for reducing the number of partitions in an RDD.
2012-09-29 21:30:52 -07:00
Matei Zaharia
c45758ddde
Comment
2012-09-29 20:27:54 -07:00
Matei Zaharia
ebd52347b5
Merge branch 'dev' of github.com:mesos/spark into dev
2012-09-29 20:22:31 -07:00
