Shivaram Venkataraman
2cf40c5fd5
Change block manager to accept a ArrayBuffer instead of an iterator to ensure
...
that the computation can proceed even if we run out of memory to cache the
block. Update CacheTracker to use this new interface
2012-10-11 00:42:46 -07:00
Denny
d3f095f904
Fixed bug when fetching Jar dependencies.
...
Instead of checking currentFiles check currentJars.
2012-10-10 16:09:53 -07:00
Matei Zaharia
ee2fcb2ce6
Added documentation to all the *RDDFunction classes, and moved them into
...
the spark package to make them more visible. Also documented various
other miscellaneous things in the API.
2012-10-09 18:38:36 -07:00
Matei Zaharia
bc0bc672d0
Updates to documentation:
...
- Edited quick start and tuning guide to simplify them a little
- Simplified top menu bar
- Made private a SparkContext constructor parameter that was left as
public
- Various small fixes
2012-10-09 14:30:23 -07:00
Andy Konwinski
1d79ff6028
Fixes a typo, adds scaladoc comments to SparkContext constructors.
2012-10-08 22:49:17 -07:00
Patrick Wendell
ac310098ef
More docs in RDD class
2012-10-08 22:25:11 -07:00
Andy Konwinski
bd688940a1
A start on scaladoc for the public APIs.
2012-10-08 21:13:29 -07:00
Mosharaf Chowdhury
edc67bfba8
Merge branch 'dev' into bc-fix-dev
2012-10-08 16:19:13 -07:00
Matei Zaharia
efc5423210
Made compression configurable separately for shuffle, broadcast and RDDs
2012-10-07 11:30:53 -07:00
Matei Zaharia
039cc6228e
Merge pull request #251 from JoshRosen/docs/internals
...
Document Dependency classes and make minor interface improvements
2012-10-07 09:56:53 -07:00
Reynold Xin
f66c0e9561
Changed the println to logInfo in Utils.fetchFile.
2012-10-07 01:53:24 -07:00
Matei Zaharia
d72db3d7dc
Merge pull request #250 from rxin/dev
...
Fixed a bug in addFile that if the file is specified as "file:///", the symlink is created incorrectly for local mode.
2012-10-07 00:56:53 -07:00
Reynold Xin
80f59e17e2
Fixed a bug in addFile that if the file is specified as "file:///", the
...
symlink is created wrong for local mode.
2012-10-07 00:54:38 -07:00
Josh Rosen
e10308f5a0
Make ShuffleDependency.aggregator explicitly optional.
...
It was confusing to be using
new Aggregator[K, V, V](null, null, null, false)
to represent the absence of an aggregator.
2012-10-07 00:36:04 -07:00
Matei Zaharia
f930fe5d81
Improve error message
2012-10-07 07:34:36 +00:00
Matei Zaharia
a3bf0ce57f
Don't crash on ask timeout exceptions in deploy.Client.stop() (fixes a crash in tests)
2012-10-07 07:25:41 +00:00
Matei Zaharia
eca570f66a
Removed the need to sleep in tests due to waiting for Akka to shut down
2012-10-07 00:17:59 -07:00
Josh Rosen
4f72066a9a
Document the Dependency classes.
2012-10-07 00:05:37 -07:00
Josh Rosen
3f2571fe98
Remove unused isShuffle field from Dependency.
2012-10-07 00:03:55 -07:00
Matei Zaharia
b2fc3dd902
Log message
2012-10-07 06:43:52 +00:00
Matei Zaharia
ea096f7cd5
More logging
2012-10-07 06:35:48 +00:00
root
554b42cb24
Log more info in MapOutputTracker
2012-10-07 05:02:18 +00:00
root
a73b25826b
Made Akka thread pool and message batch sizes configurable
2012-10-07 04:19:54 +00:00
root
ce915cadee
Made run script add test-classes onto the classpath only if SPARK_TESTING is set; fixes #216
2012-10-07 04:19:16 +00:00
root
975009d688
Avoid acquiring locks in BlockManager when fetching shuffle outputs
2012-10-07 04:02:10 +00:00
root
0bc63f7ef1
Log initial number of fetches in reducer
2012-10-07 03:51:04 +00:00
Matei Zaharia
dc28a3ac0a
Modified shuffle to limit the maximum outstanding data size in bytes,
...
instead of the maximum number of outstanding fetches. This should make
it faster when there are many small map output files, as well as more
robust to overallocating memory on large map outputs.
2012-10-06 20:07:10 -07:00
Matei Zaharia
9a3b3f32a3
Pass sizes of map outputs back to MapOutputTracker
2012-10-06 18:46:04 -07:00
Matei Zaharia
0e42832e6a
Made block store return the size of each block put in
2012-10-06 18:00:53 -07:00
Matei Zaharia
b0110de5b6
Warn about user programs that try to set spark.cache.class
2012-10-06 17:27:14 -07:00
Matei Zaharia
65113b7e1b
Only group elements ten at a time into SequenceFile records in
...
saveAsObjectFile
2012-10-06 17:14:41 -07:00
Matei Zaharia
716e10ca32
Minor formatting fixes
2012-10-05 22:03:06 -07:00
Matei Zaharia
70f02fa912
Merge branch 'dev' of github.com:mesos/spark into dev
2012-10-05 22:00:22 -07:00
Andy Konwinski
a242cdd0a6
Factor subclasses of RDD out of RDD.scala into their own classes
...
in the rdd package.
2012-10-05 19:53:54 -07:00
Andy Konwinski
d7363a6b8a
Moves all files in core/src/main/scala/ that have RDD in their name
...
from that directory to a new core/src/main/scala/rdd directory.
2012-10-05 19:23:45 -07:00
Andy Konwinski
e0067da082
Moves all files in core/src/main/scala/ that have RDD in them from
...
package spark to package spark.rdd and updates all references to them.
2012-10-05 19:23:45 -07:00
Matei Zaharia
69588baf65
Cleaning up code slightly
2012-10-05 19:16:09 -07:00
root
f52bc09a34
Reduce some overly aggressive logging in connection manager
2012-10-06 01:54:39 +00:00
Matei Zaharia
e3ae98b54e
Merge pull request #247 from squito/dev
...
Dev
2012-10-05 10:27:18 -07:00
Imran Rashid
e0698f8f26
change tests to show utility of localValue
2012-10-04 23:05:42 -07:00
Imran Rashid
82a3327862
make accumulator.localValue public, add tests
...
Conflicts:
core/src/test/scala/spark/AccumulatorSuite.scala
2012-10-04 23:05:01 -07:00
Matei Zaharia
8c82f43db3
Scaladoc documentation for some core Spark functionality
2012-10-04 22:59:36 -07:00
Reynold Xin
45f4b7cc7e
Made Serializer and JavaSerializer non private.
2012-10-03 10:20:59 -07:00
Matei Zaharia
833f1d0c86
Made StorageLevel public
2012-10-03 08:27:25 -07:00
Matei Zaharia
6cf5dffc72
Make more stuff private[spark]
2012-10-02 22:28:55 -07:00
Mosharaf Chowdhury
119e50c7b9
Conflict fixed
2012-10-02 22:25:39 -07:00
Matei Zaharia
626f701931
Merge pull request #240 from dennybritz/private_classes
...
Package-Private Classes
2012-10-02 21:24:32 -07:00
Denny
0361353a70
Make Java API abstract wrapped functions private
2012-10-02 20:02:53 -07:00
Denny
b9badcd5bd
accidentially removed trait
2012-10-02 19:35:07 -07:00
Denny
18a1faedf6
Stylistic changes and Public Accumulable and Broadcast
2012-10-02 19:28:37 -07:00
Denny
b7a913e1fa
Make dependency classes public - used by spark
2012-10-02 19:04:23 -07:00
Denny
4d9f4b01af
Make classes package private
2012-10-02 19:00:19 -07:00
Matei Zaharia
97cbd699d7
Merge branch 'dev' of github.com:mesos/spark into dev
2012-10-02 17:31:01 -07:00
Matei Zaharia
6098f7e87a
Fixed cache replacement behavior of BlockManager:
...
- Partitions that get dropped to disk will now be loaded back into RAM
after they're accessed again
- Same-RDD rule for cache replacement is now implemented (don't drop
partitions from an RDD to make room for other partitions from itself)
- Items stored as MEMORY_AND_DISK go into memory only first, instead of
being eagerly written out to disk
- MemoryStore.ensureFreeSpace is called within a lock on the writer
thread to prevent race conditions (this can still be optimized to
allow multiple concurrent calls to it but it's a start)
- MemoryStore does not accept blocks larger than its limit
2012-10-02 17:25:38 -07:00
Reynold Xin
7997585616
Added a check to make sure SPARK_MEM <= memoryPerSlave for local cluster
...
mode.
2012-10-02 15:45:25 -07:00
Reynold Xin
0898a21b95
Merge branch 'dev' of https://github.com/mesos/spark into dev
2012-10-02 13:08:01 -07:00
Matei Zaharia
22684653a5
Revert "Place Spray repo ahead of Cloudera in Maven search path"
...
This reverts commit 42e0a68082
.
2012-10-02 12:01:32 -07:00
Reynold Xin
b8cd681169
Allow whitespaces in cluster URL configuration for local cluster.
2012-10-02 11:52:12 -07:00
Matei Zaharia
42e0a68082
Place Spray repo ahead of Cloudera in Maven search path
2012-10-02 11:37:19 -07:00
Matei Zaharia
b9fb8d6463
Include date in folder name for Spark local dir.
2012-10-01 15:55:16 -07:00
Matei Zaharia
bc881e4798
Merge branch 'dev' of github.com:mesos/spark into dev
2012-10-01 15:21:56 -07:00
Matei Zaharia
802aa8aef9
Some bug fixes and logging fixes for broadcast.
2012-10-01 15:20:42 -07:00
Reynold Xin
f264153162
Fixed #232 : DirectBuffer's cleaner was empty and Spark tried to invoke
...
clean on it.
2012-10-01 14:07:34 -07:00
Matei Zaharia
3b348f909d
Improve log messages from BlockManager
2012-10-01 12:01:38 -07:00
Matei Zaharia
53f90d0f0e
Use underscores instead of colons in RDD IDs
2012-10-01 10:48:53 -07:00
Matei Zaharia
2314132d57
Added a (failing) test for LRU with MEMORY_AND_DISK.
2012-09-30 22:52:16 -07:00
Matei Zaharia
3128c57f90
Simplified Class / ClassLoader test
2012-09-30 21:48:27 -07:00
Matei Zaharia
83143f9a5f
Fixed several bugs that caused weird behavior with files in spark-shell:
...
- SizeEstimator was following through a ClassLoader field of Hadoop
JobConfs, which referenced the whole interpreter, Scala compiler, etc.
Chaos ensued, giving an estimated size in the tens of gigabytes.
- Broadcast variables in local mode were only stored as MEMORY_ONLY and
never made accessible over a server, so they fell out of the cache when
they were deemed too large and couldn't be reloaded.
2012-09-30 21:19:39 -07:00
Matei Zaharia
fd0374b9de
Comment
2012-09-29 21:43:06 -07:00
Matei Zaharia
5718cef2a4
Removed Logging trait from CoalescedRDD since we don't log anything
2012-09-29 21:40:43 -07:00
Matei Zaharia
143ef4f90d
Added a CoalescedRDD class for reducing the number of partitions in an RDD.
2012-09-29 21:30:52 -07:00
Matei Zaharia
ebd52347b5
Merge branch 'dev' of github.com:mesos/spark into dev
2012-09-29 20:22:31 -07:00
Matei Zaharia
9b326d01e9
Made BlockManager unmap memory-mapped files when necessary to reduce the
...
number of open files. Also optimized sending of disk-based blocks.
2012-09-29 20:21:54 -07:00
Matei Zaharia
2f11e3c285
Merge pull request #227 from JoshRosen/fix/distinct_numsplits
...
Allow controlling number of splits in distinct().
2012-09-28 23:57:24 -07:00
Josh Rosen
8654165e69
Use null as dummy value in distinct().
2012-09-28 23:55:17 -07:00
Josh Rosen
37c199bbb0
Allow controlling number of splits in distinct().
2012-09-28 23:44:19 -07:00
Matei Zaharia
56dcad5936
Don't create a Cache in SparkEnv because we don't use it
2012-09-28 23:40:56 -07:00
Matei Zaharia
1d44644f4f
Logging tweaks
2012-09-28 23:28:16 -07:00
Matei Zaharia
815d6bd69a
Renamed subdirs option
2012-09-28 19:02:41 -07:00
Matei Zaharia
e54e1d7043
Made subdirs per local dir configurable, and reduced lock usage a bit
2012-09-28 19:00:50 -07:00
Matei Zaharia
ae8c7d6cfa
Made disk store use multiple directories, deleted ShuffleManager
2012-09-28 18:28:13 -07:00
Matei Zaharia
3d7267999d
Print and track user call sites in more places in Spark
2012-09-28 17:42:00 -07:00
Matei Zaharia
9f6efbf06a
Merge pull request #225 from pwendell/dev
...
Log message which records RDD origin
2012-09-28 16:28:07 -07:00
Matei Zaharia
0121a26bd1
Changed the way tasks' dependency files are sent to workers so that
...
custom serializers or Kryo registrators can be loaded.
2012-09-28 16:14:05 -07:00
Patrick Wendell
9fc78f8f29
Fixing some whitespace issues
2012-09-28 16:05:50 -07:00
Patrick Wendell
bc909c2903
Changes based on Matei's comments
2012-09-28 16:04:36 -07:00
Patrick Wendell
c387e40fb1
Log message which records RDD origin
...
This adds tracking to determine the "origin" of an RDD. Origin is defined by
the boundary between the user's code and the spark code, during an RDD's
instantiation. It is meant to help users understand where a Spark RDD is
coming from in their code.
This patch also logs origin data when stages are submitted to the scheduler.
Finally, it adds a new log message to fix an inconsitency in the way that
dependent stages (those missing parents) and independent stages (those
without) are logged during submission.
2012-09-28 15:51:46 -07:00
Matei Zaharia
2a8bfbca00
Fixed a bug where isLocal was set to false when using local[K]
2012-09-28 14:50:54 -07:00
Matei Zaharia
4a138403ef
Fix a bug in JAR fetcher that made it always fetch the JAR
2012-09-27 21:32:06 -07:00
Matei Zaharia
009b0e37e7
Added an option to compress blocks in the block store
2012-09-27 18:45:44 -07:00
Matei Zaharia
7bcb08cef5
Renamed storage levels to something cleaner; fixes #223 .
2012-09-27 17:50:59 -07:00
Matei Zaharia
920fab23c3
Merge pull request #222 from rxin/dev
...
Added MapPartitionsWithSplitRDD.
2012-09-26 23:16:45 -07:00
Matei Zaharia
ea05fc130b
Updates to standalone cluster, web UI and deploy docs.
2012-09-26 22:54:39 -07:00
Matei Zaharia
1ef4f0fbd2
Allow controlling number of splits in sortByKey.
2012-09-26 19:18:47 -07:00
Reynold Xin
1ad1331a34
Added MapPartitionsWithSplitRDD.
2012-09-26 17:11:28 -07:00
Matei Zaharia
ee71fa49c1
Look for Kryo registrator using context class loader
2012-09-26 14:15:16 -07:00
Matei Zaharia
d71a358c46
Fixed a test that was getting extremely lucky before, and increased the
...
number of samples used for sorting
2012-09-26 00:25:34 -07:00
Matei Zaharia
051785c7e6
Several fixes to sampling issues pointed out by Henry Milner:
...
- takeSample was biased towards earlier partitions
- There were some range errors in takeSample
- SampledRDDs with replacement didn't produce appropriate counts
across partitions (we took exactly frac of each one)
2012-09-25 21:46:58 -07:00
Matei Zaharia
4d3339a3ec
Merge pull request #217 from rxin/dev
...
Added a method to RDD to expose the ClassManifest.
2012-09-24 23:52:32 -07:00
Reynold Xin
7a4cd92861
Renamed RDD.manifest to RDD.elementClassManifest
2012-09-24 23:42:33 -07:00