jerryshao
c23cd72b4b
Upgrade Kafka 0.7.2 to Kafka 0.8.0-beta1 for Spark Streaming
2013-10-12 20:00:42 +08:00
Shivaram Venkataraman
c441904bce
Add a comment and exclude tools
2013-10-11 18:23:15 -07:00
Matei Zaharia
c71499b779
Merge pull request #19 from aarondav/master-zk
...
Standalone Scheduler fault tolerance using ZooKeeper
This patch implements full distributed fault tolerance for standalone scheduler Masters.
There is only one master Leader at a time, which is actively serving scheduling
requests. If this Leader crashes, another master will eventually be elected, reconstruct
the state from the first Master, and continue serving scheduling requests.
Leader election is performed using the ZooKeeper leader election pattern. We try to minimize
the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of
retries and session monitoring on top of the ZooKeeper client.
Master failover follows directly from the single-node Master recovery via the file
system (patch d5a96fe
), save that the Master state is stored in ZooKeeper instead.
Configuration:
By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE).
By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url
to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled.
By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory
to an appropriate directory accessible by the Master, we will keep the behavior of from d5a96fe
.
Additionally, places where a Master could be specificied by a spark:// url can now take
comma-delimited lists to specify backup masters. Note that this is only used for registration
of NEW Workers and application Clients. Once a Worker or Client has registered with the
Master Leader, it is "in the system" and will never need to register again.
2013-10-10 17:16:42 -07:00
Prashant Sharma
26860639c5
Merge branch 'scala-2.10' of github.com:ScrapCodes/spark into scala-2.10
...
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
project/SparkBuild.scala
2013-10-10 09:42:23 +05:30
Shivaram Venkataraman
484166d520
Add new SBT target for dependency assembly
2013-10-09 04:24:34 -07:00
Prashant Sharma
7be75682b9
Merge branch 'master' into wip-merge-master
...
Conflicts:
bagel/pom.xml
core/pom.xml
core/src/test/scala/org/apache/spark/ui/UISuite.scala
examples/pom.xml
mllib/pom.xml
pom.xml
project/SparkBuild.scala
repl/pom.xml
streaming/pom.xml
tools/pom.xml
In scala 2.10, a shorter representation is used for naming artifacts
so changed to shorter scala version for artifacts and made it a property in pom.
2013-10-08 11:29:40 +05:30
Reynold Xin
213b70a2db
Merge pull request #31 from sundeepn/branch-0.8
...
Resolving package conflicts with hadoop 0.23.9
Hadoop 0.23.9 is having a package conflict with easymock's dependencies.
(cherry picked from commit 023e3fdf00
)
Signed-off-by: Reynold Xin <rxin@apache.org>
2013-10-07 10:54:22 -07:00
Martin Weindel
9b0c9c893d
scala 2.10 requires Java 1.6,
...
using Scala 2.10.3,
resolved maven-scala-plugin warning
2013-10-05 21:41:09 +02:00
Prashant Sharma
c810ee0690
Merge branch 'master' into scala-2.10
...
Conflicts:
core/src/test/scala/org/apache/spark/DistributedSuite.scala
project/SparkBuild.scala
2013-10-05 15:52:57 +05:30
Du Li
9fd6bba60d
ask ivy/sbt to check local maven repo under ~/.m2
2013-10-01 15:46:51 -07:00
Prashant Sharma
5829692885
Merge branch 'master' into scala-2.10
...
Conflicts:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala
docs/_config.yml
project/SparkBuild.scala
repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
2013-10-01 11:57:24 +05:30
Aaron Davidson
f549ea33d3
Standalone Scheduler fault tolerance using ZooKeeper
...
This patch implements full distributed fault tolerance for standalone scheduler Masters.
There is only one master Leader at a time, which is actively serving scheduling
requests. If this Leader crashes, another master will eventually be elected, reconstruct
the state from the first Master, and continue serving scheduling requests.
Leader election is performed using the ZooKeeper leader election pattern. We try to minimize
the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of
retries and session monitoring on top of the ZooKeeper client.
Master failover follows directly from the single-node Master recovery via the file
system (patch 194ba4b8), save that the Master state is stored in ZooKeeper instead.
Configuration:
By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE).
By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url
to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled.
By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory
to an appropriate directory accessible by the Master, we will keep the behavior of from 194ba4b8.
Additionally, places where a Master could be specificied by a spark:// url can now take
comma-delimited lists to specify backup masters. Note that this is only used for registration
of NEW Workers and application Clients. Once a Worker or Client has registered with the
Master Leader, it is "in the system" and will never need to register again.
Forthcoming:
Documentation, tests (! - only ad hoc testing has been performed so far)
I do not intend for this commit to be merged until tests are added, but this patch should
still be mostly reviewable until then.
2013-09-26 15:04:23 -07:00
Reynold Xin
3f283278b0
Removed scala -optimize flag.
2013-09-26 13:58:10 -07:00
Reynold Xin
c514cd1587
Merge pull request #930 from holdenk/master
...
Add mapPartitionsWithIndex
2013-09-26 13:48:20 -07:00
Prashant Sharma
604dc40996
Sync with master and some build fixes
2013-09-26 11:40:02 +05:30
Prashant Sharma
7ff4c2d399
fixed maven build for scala 2.10
2013-09-26 10:48:24 +05:30
Patrick Wendell
6079721fa1
Update build version in master
2013-09-24 11:41:51 -07:00
Prashant Sharma
276c37a51c
Akka 2.2 migration
2013-09-22 08:20:12 +05:30
Joseph E. Gonzalez
55696e2584
GraphX now builds with all merged changes.
2013-09-17 22:42:12 -07:00
Joseph E. Gonzalez
8b59fb72c4
Merging latest changes from spark main branch
2013-09-17 20:56:12 -07:00
Patrick Wendell
c856860c5b
Bumping Mesos version to 0.13.0
2013-09-15 12:46:26 -07:00
Prashant Sharma
383e151fd7
Merge branch 'master' of git://github.com/mesos/spark into scala-2.10
...
Conflicts:
core/src/main/scala/org/apache/spark/SparkContext.scala
project/SparkBuild.scala
2013-09-15 10:55:12 +05:30
Prashant Sharma
20c65bc334
Fixed repl suite
2013-09-15 10:43:06 +05:30
Holden Karau
68068977b8
Fix build on ubuntu
2013-09-14 20:51:11 -07:00
Patrick Wendell
91a59e6b10
Merge pull request #919 from mateiz/jets3t
...
Add explicit jets3t dependency, which is excluded in hadoop-client
2013-09-11 10:21:48 -07:00
Patrick Wendell
0c1985b153
Fix HDFS access bug with assembly build.
...
Due to this change in HDFS:
https://issues.apache.org/jira/browse/HADOOP-7549
there is a bug when using the new assembly builds. The symptom is that any HDFS access
results in an exception saying "No filesystem for scheme 'hdfs'". This adds a merge
strategy in the assembly build which fixes the problem.
2013-09-10 22:05:13 -07:00
Matei Zaharia
f117dc6d0d
Add explicit jets3t dependency, which is excluded in hadoop-client
2013-09-10 06:39:25 +00:00
Patrick Wendell
f68848d95d
Merge pull request #906 from pwendell/ganglia-sink
...
Clean-up of Metrics Code/Docs and Add Ganglia Sink
2013-09-08 18:32:16 -07:00
Matei Zaharia
0b957997ad
Merge pull request #908 from pwendell/master
...
Fix target JVM version in scala build
2013-09-08 15:30:16 -07:00
Patrick Wendell
27bd74c8ad
Fix target JVM version in scala build
2013-09-08 14:37:45 -07:00
Patrick Wendell
8de8ee5d3c
Ganglia sink
2013-09-08 10:08:18 -07:00
Patrick Wendell
a8e376ec0f
Merge pull request #904 from pwendell/master
...
Adding Apache license to two files
2013-09-07 21:16:01 -07:00
Patrick Wendell
6d2198643c
Adding Apache license to two files
2013-09-07 20:46:58 -07:00
Jey Kottalam
30a32c8335
Minor YARN build cleanups
2013-09-06 11:31:16 -07:00
Prashant Sharma
4106ae9fbf
Merged with master
2013-09-06 17:53:01 +05:30
Matei Zaharia
59218bdd49
Add Apache parent POM
2013-09-02 18:34:03 -07:00
Matei Zaharia
5701eb92c7
Fix some URLs
2013-09-01 14:13:16 -07:00
Matei Zaharia
46eecd110a
Initial work to rename package to org.apache.spark
2013-09-01 14:13:13 -07:00
Matei Zaharia
666d93c294
Update Maven build to create assemblies expected by new scripts
...
This includes the following changes:
- The "assembly" package now builds in Maven by default, and creates an
assembly containing both hadoop-client and Spark, unlike the old
BigTop distribution assembly that skipped hadoop-client
- There is now a bigtop-dist package to build the old BigTop assembly
- The repl-bin package is no longer built by default since the scripts
don't reply on it; instead it can be enabled with -Prepl-bin
- Py4J is now included in the assembly/lib folder as a local Maven repo,
so that the Maven package can link to it
- run-example now adds the original Spark classpath as well because the
Maven examples assembly lists spark-core and such as provided
- The various Maven projects add a spark-yarn dependency correctly
2013-08-29 21:19:06 -07:00
Matei Zaharia
8d81358a05
Provide more memory for tests
2013-08-29 21:19:06 -07:00
Matei Zaharia
53cd50c069
Change build and run instructions to use assemblies
...
This commit makes Spark invocation saner by using an assembly JAR to
find all of Spark's dependencies instead of adding all the JARs in
lib_managed. It also packages the examples into an assembly and uses
that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script
with two better-named scripts: "run-examples" for examples, and
"spark-class" for Spark internal classes (e.g. REPL, master, etc). This
is also designed to minimize the confusion people have in trying to use
"run" to run their own classes; it's not meant to do that, but now at
least if they look at it, they can modify run-examples to do a decent
job for them.
As part of this, Bagel's examples are also now properly moved to the
examples package instead of bagel.
2013-08-29 21:19:04 -07:00
Reynold Xin
9db1e50344
Revert "Merge pull request #841 from rxin/json"
...
This reverts commit 1fb1b09928
, reversing
changes made to c69c48947d
.
2013-08-26 11:05:14 -07:00
Jey Kottalam
a9db1b7b6e
Upgrade SBT IDE project generators
2013-08-23 10:27:18 -07:00
Jey Kottalam
b7f9e6374a
Fix SBT generation of IDE project files
2013-08-23 10:26:37 -07:00
Jey Kottalam
281b6c5f28
Re-add removed dependency on 'commons-daemon'
...
Fixes SBT build under Hadoop 0.23.9 and 2.0.4
2013-08-22 15:45:45 -07:00
Matei Zaharia
ae8ba83ef2
Merge pull request #855 from jey/update-build-docs
...
Update build docs
2013-08-22 10:14:54 -07:00
Matei Zaharia
8a36fd09dd
Merge pull request #854 from markhamstra/pomUpdate
...
Synced sbt and maven builds to use the same dependencies, etc.
2013-08-22 10:13:35 -07:00
Jey Kottalam
f9cc1fbf27
Remove references to unsupported Hadoop versions
2013-08-21 17:14:36 -07:00
Mark Hamstra
ff6f1b0500
Synced sbt and maven builds
2013-08-21 13:50:24 -07:00
Reynold Xin
af602ba9d3
Downgraded default build hadoop version to 1.0.4.
2013-08-21 11:38:24 -07:00
Matei Zaharia
aa2b89d98d
Merge remote-tracking branch 'jey/hadoop-agnostic'
...
Conflicts:
core/src/main/scala/spark/PairRDDFunctions.scala
2013-08-20 10:14:15 -07:00
Jey Kottalam
6f6944c807
Update SBT build to use simpler fix for Hadoop 0.23.9
2013-08-19 12:33:13 -07:00
Jey Kottalam
67b593607c
Rename YARN build flag to SPARK_WITH_YARN
2013-08-16 14:00:05 -07:00
Jey Kottalam
b1d99744a8
Fix SBT build under Hadoop 0.23.x
2013-08-16 13:50:12 -07:00
Jey Kottalam
8add2d7a59
Fix repl/assembly when YARN enabled
2013-08-16 13:50:12 -07:00
Jey Kottalam
3f98eff63a
Allow make-distribution.sh to specify Hadoop version used
2013-08-16 13:50:09 -07:00
Reynold Xin
c961c19b7b
Use the JSON formatter from Scala library and removed dependency on lift-json.
...
It made the JSON creation slightly more complicated, but reduces one external dependency. The scala library also properly escape "/" (which lift-json doesn't).
2013-08-15 18:23:01 -07:00
Jey Kottalam
a0f0848463
Update default version of Hadoop to 1.2.1
2013-08-15 16:50:37 -07:00
Jey Kottalam
cb4ef19214
yarn support
2013-08-15 16:50:37 -07:00
Jey Kottalam
273b499b9a
yarn sbt
2013-08-15 16:50:37 -07:00
Jey Kottalam
69c3bbf688
dynamically detect hadoop version
2013-08-15 16:50:37 -07:00
Matei Zaharia
d9588183fa
Update to Mesos 0.12.1
2013-08-13 18:51:35 -07:00
jerryshao
320e87e7ab
Add MetricsServlet for Spark metrics system
2013-08-12 13:23:23 +08:00
Matei Zaharia
dce5e47435
Merge pull request #800 from dlyubimov/HBASE_VERSION
...
Pull HBASE_VERSION in the head of sbt build
2013-08-09 21:53:45 -07:00
Matei Zaharia
cd247ba5bb
Merge pull request #786 from shivaram/mllib-java
...
Java fixes, tests and examples for ALS, KMeans
2013-08-09 20:41:13 -07:00
Dmitriy Lyubimov
27f674f82b
fewer words
2013-08-09 13:54:41 -07:00
Dmitriy Lyubimov
ae95b57469
Pull HBASE_VERSION in the head of sbt build
2013-08-09 12:45:18 -07:00
Matei Zaharia
5a4003c1ac
Update to Chill 0.3.1
2013-08-08 13:30:27 -07:00
Shivaram Venkataraman
471fbadd0c
Java examples, tests for KMeans and ALS
...
- Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it
easier to call from Java
- Renames class methods from `train` to `run` to enable static methods to be
called from Java.
- Add unit tests which check if both static / class methods can be called.
- Also add examples which port the main() function in ALS, KMeans to the
examples project.
Couple of minor changes to existing code:
- Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily
- Workaround a bug where using double[] from Java leads to class cast exception in
KMeans init
2013-08-06 15:43:46 -07:00
Joseph E. Gonzalez
499a0d8383
Merged graphx from @rxin into master
2013-08-06 12:28:29 -07:00
Matei Zaharia
e466a55a6b
Revert Mesos version to 0.9 since the 0.12 artifact has target Java 7
2013-08-01 15:45:21 -07:00
Matei Zaharia
b2b86c2575
Merge pull request #753 from shivaram/glm-refactor
...
Build changes for ML lib
2013-07-31 15:51:39 -07:00
Matei Zaharia
14bf2fe039
Merge pull request #749 from benh/spark-executor-uri
...
Added property 'spark.executor.uri' for launching on Mesos.
2013-07-31 14:18:16 -07:00
Shivaram Venkataraman
15fd0d619d
Add mllib, bagel to repl dependencies
...
Also don't build an assembly jar for them
2013-07-30 18:31:11 -07:00
Reynold Xin
3b1ced83fb
Exclude older version of Snappy in streaming and examples.
2013-07-30 17:25:36 -07:00
Reynold Xin
368c58eac5
Merge branch 'lazy_file_open' of github.com:lyogavin/spark into compression
...
Conflicts:
project/SparkBuild.scala
2013-07-30 16:04:18 -07:00
Shivaram Venkataraman
48851d4dd9
Add bagel, mllib to SBT assembly.
...
Also add jblas dependency to mllib pom.xml
2013-07-30 14:03:15 -07:00
Benjamin Hindman
f6f46455eb
Added property 'spark.executor.uri' for launching on Mesos without
...
requiring Spark to be installed. Using 'make_distribution.sh' a user
can put a Spark distribution at a URI supported by Mesos (e.g.,
'hdfs://...') and then set that when launching their job. Also added
SPARK_EXECUTOR_URI for the REPL.
2013-07-29 23:32:52 -07:00
ryanlecompte
8e0939f5a9
refactor Kryo serializer support to use chill/chill-java
2013-07-24 20:43:57 -07:00
jerryshao
5730193e0c
Fix some typos
2013-07-24 14:57:47 +08:00
jerryshao
576528f0f9
Add dependency of Codahale's metrics library
2013-07-24 14:57:46 +08:00
Josh Rosen
c83680434b
Add JavaAPICompletenessChecker.
...
This is used to find methods in the Scala API that
need to be ported to the Java API. To use it:
./run spark.tools.JavaAPICompletenessChecker
Conflicts:
project/SparkBuild.scala
run
run2.cmd
2013-07-22 16:11:49 -07:00
Liang-Chi Hsieh
d1738d72ba
also exclude asm for hadoop2. hadoop1 looks like no need to do that too.
2013-07-20 00:37:24 +08:00
Liang-Chi Hsieh
3aad452653
fix a bug in build process that pulls in two versionf of ASM.
2013-07-19 02:29:46 +08:00
Matei Zaharia
cad48edb70
Merge pull request #708 from ScrapCodes/dependencies-upgrade
...
Dependency upgrade Akka 2.0.3 -> 2.0.5
2013-07-16 21:41:28 -07:00
Matei Zaharia
af3c9d5042
Add Apache license headers and LICENSE and NOTICE files
2013-07-16 17:21:33 -07:00
Prashant Sharma
2748e73eb9
Dependency upgrade Akka 2.0.3 -> 2.0.5
2013-07-16 16:08:46 +05:30
Prashant Sharma
9d7781c4e1
Adding commons io as dependency
2013-07-15 12:03:48 +05:30
Prashant Sharma
a3494d405d
Merge branch 'master' of github.com:mesos/spark into scala-2.10
...
Conflicts:
core/src/main/scala/spark/Utils.scala
core/src/test/scala/spark/ui/UISuite.scala
project/SparkBuild.scala
run
2013-07-15 11:15:55 +05:30
Matei Zaharia
668b0dc6a7
Merge branch 'master' of github.com:mesos/spark
2013-07-13 19:10:46 -07:00
Matei Zaharia
cd28d9c147
Merge remote-tracking branch 'origin/pr/662'
...
Conflicts:
bin/compute-classpath.sh
2013-07-13 19:10:00 -07:00
seanm
c4d5b01e44
changing com.google.code.findbugs maven coordinates
2013-07-13 14:56:23 -06:00
Prashant Sharma
e86d5dbaad
Merge branch 'master' into master-merge
...
Conflicts:
README.md
core/pom.xml
core/src/main/scala/spark/deploy/JsonProtocol.scala
core/src/main/scala/spark/deploy/LocalSparkCluster.scala
core/src/main/scala/spark/deploy/master/Master.scala
core/src/main/scala/spark/deploy/master/MasterWebUI.scala
core/src/main/scala/spark/deploy/worker/Worker.scala
core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
core/src/main/scala/spark/storage/BlockManagerUI.scala
core/src/main/scala/spark/util/AkkaUtils.scala
pom.xml
project/SparkBuild.scala
streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
2013-07-12 14:49:16 +05:30
Prashant Sharma
69ae7ea227
Removed some unnecessary code and fixed dependencies
2013-07-11 18:30:18 +05:30
Matei Zaharia
3cc6818f13
Merge pull request #668 from shimingfei/guava-14.0.1
...
update guava version from 11.0.1 to 14.0.1
2013-07-06 19:51:20 -07:00
Matei Zaharia
1ffadb2d9e
Merge remote-tracking branch 'pwendell/ui-updates'
...
Conflicts:
core/src/main/scala/spark/scheduler/DAGScheduler.scala
core/src/main/scala/spark/util/AkkaUtils.scala
pom.xml
2013-07-06 15:51:41 -07:00
Matei Zaharia
43b24635ee
Renamed ML package to MLlib and added it to classpath
2013-07-05 11:38:53 -07:00
Matei Zaharia
05be233ce2
Removed dependency on Apache Commons Math
2013-07-05 11:13:46 -07:00
Reynold Xin
6a9a9a364c
Minor clean up of the RidgeRegression code. I am not even sure why I did
...
this :s.
2013-07-05 11:13:45 -07:00
Matei Zaharia
729e463f64
Import RidgeRegression example
...
Conflicts:
run
2013-07-05 11:13:41 -07:00
Gavin Li
94238aae57
fix dependencies
2013-07-03 18:08:38 +00:00
Mingfei
04567a1771
update guava version from 11.0.1 to 14.0.1
2013-07-03 17:43:37 +08:00
Prashant Sharma
a5f1f6a907
Merge branch 'master' into master-merge
...
Conflicts:
core/pom.xml
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/RDDCheckpointData.scala
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/Utils.scala
core/src/main/scala/spark/api/python/PythonRDD.scala
core/src/main/scala/spark/deploy/client/Client.scala
core/src/main/scala/spark/deploy/master/MasterWebUI.scala
core/src/main/scala/spark/deploy/worker/Worker.scala
core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/ZippedRDD.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockManagerMaster.scala
core/src/main/scala/spark/storage/BlockManagerMasterActor.scala
core/src/main/scala/spark/storage/BlockManagerUI.scala
core/src/main/scala/spark/util/AkkaUtils.scala
core/src/test/scala/spark/SizeEstimatorSuite.scala
pom.xml
project/SparkBuild.scala
repl/src/main/scala/spark/repl/SparkILoop.scala
repl/src/test/scala/spark/repl/ReplSuite.scala
streaming/src/main/scala/spark/streaming/StreamingContext.scala
streaming/src/main/scala/spark/streaming/api/java/JavaStreamingContext.scala
streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
streaming/src/main/scala/spark/streaming/util/MasterFailureTest.scala
2013-07-03 11:43:26 +05:30
Matei Zaharia
5cfcd3c336
Remove Twitter4J specific repo since it's in Maven central
2013-06-29 15:37:27 -07:00
Reynold Xin
564d902d79
Merge branch 'master' of github.com:mesos/spark into graph
...
Conflicts:
run
run2.cmd
2013-06-29 15:30:21 -07:00
Evan Chan
1107b4d55b
Merge branch 'master' into 2013-06/assembly-jar-deploy
...
Conflicts:
run
Previous changes that I made to run and set-dev-classpath.sh instead
have been folded into compute-classpath.sh
2013-06-28 17:18:35 -07:00
Matei Zaharia
32370da4e4
Don't use forward slash in exclusion for JAR signature files
2013-06-25 22:08:19 -04:00
Evan Chan
d2f46ac680
Merge branch 'master' into 2013-06/assembly-jar-deploy
...
Conflicts:
run
2013-06-25 14:50:16 -07:00
Tathagata Das
c89af0a7f9
Merge branch 'master' into streaming
...
Conflicts:
.gitignore
2013-06-24 23:57:47 -07:00
Patrick Wendell
91ec5a1a04
Changing JSON protocol and removing spray code
2013-06-22 10:31:36 -07:00
Matei Zaharia
b350f34703
Increase memory for tests to prevent a crash on JDK 7
2013-06-22 07:48:20 -07:00
Evan Chan
071ff7efa1
Enable building a fat jar for the Spark REPL
2013-06-20 17:53:23 -07:00
Matei Zaharia
ae7a5da6b3
Fix some dependency issues in SBT build (same will be needed for Maven):
...
- Exclude a version of ASM 3.x that comes from HBase
- Don't use a special ASF repo for HBase
- Update SLF4J version
- Add sbt-dependency-graph plugin so we can easily find dependency trees
2013-06-20 18:44:46 +02:00
Matei Zaharia
7902baddc7
Update ASM to version 4.0
2013-06-19 13:34:30 +02:00
Matei Zaharia
dbfab49d2a
Merge remote-tracking branch 'milliondreams/casdemo'
...
Conflicts:
project/SparkBuild.scala
2013-06-18 14:55:31 +02:00
Matei Zaharia
73f4c7d2d1
Merge pull request #605 from esjewett/SPARK-699
...
Add hBase example (retry of pull request #596 )
2013-06-18 04:21:17 -07:00
Matei Zaharia
2ab311f4ce
Removed second version of junit test plugin from plugins.sbt
2013-06-18 00:40:25 +02:00
Christopher Nguyen
479442a9b9
Add zeroLengthPartitions() test to make sure, e.g., StatCounter.scala can handle empty partitions without incorrectly returning NaN
2013-06-15 17:35:55 -07:00
Matei Zaharia
5b5b5aedbf
Fixed a few test issues due to Akka 2.1, as well as SBT memory.
...
Unfortunately, in Akka 2.1, ActorSystem.awaitTermination hangs for
remote actors, and Akka also leaves a non-daemon Netty thread even when
run in daemon mode. Thus I had to comment out some of the calls to
awaitTermination, and we still have one failing test.
2013-06-08 01:09:24 -07:00
Rohit Rai
6d8423fd1b
Adding deps to examples/pom.xml
...
Fixing exclusion in examples deps in SparkBuild.scala
2013-06-02 13:03:45 +05:30
Rohit Rai
3be7bdcefd
Adding example to make Spark RDD from Cassandra
2013-06-01 19:32:17 +05:30
Reynold Xin
b0403d3f2b
Merge branch 'master' of github.com:mesos/spark into graph
...
Conflicts:
run
2013-06-01 00:48:27 -07:00
Reynold Xin
f742435f18
Removed the duplicated netty dependency in SBT build file.
2013-05-16 14:31:03 -07:00
Reynold Xin
f3491cb89b
Merge branch 'master' of github.com:mesos/spark into shufflemerge
...
Conflicts:
core/src/main/scala/spark/storage/BlockManager.scala
core/src/test/scala/spark/DistributedSuite.scala
project/SparkBuild.scala
2013-05-15 00:31:52 -07:00
Reynold Xin
81ad2fa331
Merge branch 'jdbc' of github.com:koeninger/spark
...
Conflicts:
project/SparkBuild.scala
2013-05-14 23:12:00 -07:00
Cody Koeninger
b16c4896f6
add test for JdbcRDD using embedded derby, per rxin suggestion
2013-05-14 23:44:04 -05:00
Ethan Jewett
ee6f6aa6cd
Add hBase example
2013-05-09 18:33:38 -05:00
Reynold Xin
012c9e5ab0
Revert "Merge pull request #596 from esjewett/master" because the
...
dependency on hbase introduces netty-3.2.2 which conflicts with
netty-3.5.3 already in Spark. This caused multiple test failures.
This reverts commit 0f1b7a06e1
, reversing
changes made to aacca1b8a8
.
2013-05-09 14:20:01 -07:00
Reynold Xin
90577ada69
Merge branch 'shuffle-performance-fix-0.7' of github.com:shane-huang/spark into shufflemerge
...
Conflicts:
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/DiskStore.scala
project/SparkBuild.scala
2013-05-07 15:56:19 -07:00
Ethan Jewett
02e8cfa617
HBase example
2013-05-04 12:31:30 -05:00
Reynold Xin
f54bc544c5
Merge branch 'master' of github.com:mesos/spark into graph
2013-05-02 17:25:09 -07:00
Jey Kottalam
207afe4088
Remove spark-repl's extraneous dependency on spark-streaming
2013-05-01 16:57:31 -07:00
Prashant Sharma
4041a2689e
Updated to latest stable scala 2.10.1 and akka 2.1.2
2013-05-01 11:35:35 +05:30
Matei Zaharia
f1f92c88eb
Build against Hadoop 1 by default
2013-04-29 17:08:45 -07:00
Prashant Sharma
4b4a36ea7d
Fixed pom.xml with updated dependencies.
2013-04-29 12:55:43 +05:30
Matei Zaharia
1b169f190c
Exclude old versions of Netty, which had a different Maven organization
2013-04-25 19:52:12 -07:00
Matei Zaharia
eef9ea1993
Update unit test memory to 2 GB
2013-04-25 00:42:29 -07:00
Matei Zaharia
01d9ba5038
Add back line removed during YARN merge
2013-04-25 00:11:27 -07:00
Prashant Sharma
ad88f083a6
scala 2.10 and master merge
2013-04-24 18:08:26 +05:30
Mridul Muralidharan
3b594a4e3b
Do not add signature files - results in validation errors when using assembled file
2013-04-24 10:18:25 +05:30
Mridul Muralidharan
dd515ca3ee
Attempt at fixing merge conflict
2013-04-24 09:24:17 +05:30
Mridul Muralidharan
adcda84f96
Pull latest SparkBuild.scala from master and merge conflicts
2013-04-24 08:57:25 +05:30
Mridul Muralidharan
5b85c715c8
Revert back to 2.0.2-alpha : 0.23.7 has protocol changes which break against cloudera
2013-04-24 02:57:51 +05:30
Mridul Muralidharan
8faf5c51c3
Patch from Thomas Graves to improve the YARN Client, and move to more production ready hadoop yarn branch
2013-04-24 02:31:57 +05:30
Prashant Sharma
185bb9525a
Manually merged scala-2.10 and master
2013-04-22 14:14:03 +05:30
Mridul Muralidharan
7acab3ab45
Fix review comments, add a new api to SparkHadoopUtil to create appropriate Configuration. Modify an example to show how to use SplitInfo
2013-04-22 08:01:13 +05:30
Matei Zaharia
17e076de80
Turn on forking in test JVMs to reduce the pressure on perm gen and code
...
cache sizes due to having 2 instances of the Scala compiler and a bunch
of classloaders.
2013-04-18 22:25:57 -07:00
Mridul Muralidharan
5d891534fd
Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet. Also, add netty dependency explicitly to prevent resolving to older 2.3x version. Additionally, comment out retrievePattern to ensure correct netty is picked up
2013-04-17 05:54:43 +05:30
Matei Zaharia
ec5e553b41
Merge pull request #558 from ash211/patch-jackson-conflict
...
Don't pull in old versions of Jackson via hadoop-core
2013-04-14 08:20:13 -07:00
Matei Zaharia
ed336e0d44
Fix tests from different projects running in parallel in SBT 0.12
2013-04-11 22:29:37 -04:00
Prashant Sharma
9f26318bbd
Fixed previously removed dependencies
2013-04-10 14:46:42 +05:30
Andrew Ash
18bd41d1a3
Don't pull in old versions of Jackson via hadoop-core
2013-04-09 14:44:47 -04:00
Matei Zaharia
65caa8f711
Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0'
...
Conflicts:
docs/_config.yml
project/SparkBuild.scala
2013-04-08 12:43:17 -04:00
Matei Zaharia
1cb3eb9762
Merge remote-tracking branch 'kalpit/master'
...
Conflicts:
project/SparkBuild.scala
2013-04-07 20:54:18 -04:00
Matei Zaharia
b362df39ea
Merge pull request #552 from MLnick/master
...
Bumping version for Twitter Algebird to latest
2013-04-07 17:17:52 -07:00
Mridul Muralidharan
6798a09df8
Add support for building against hadoop2-yarn : adding new maven profile for it
2013-04-07 17:47:38 +05:30
Reynold Xin
3728e1bc40
Code to run bagel vs graph experiments.
2013-04-07 15:05:46 +08:00
shane-huang
df47b40b76
Shuffle Performance fix: Use netty embeded OIO file server instead of ConnectionManager
...
Shuffle Performance Optimization: do not send 0-byte block requests to reduce network messages
change reference from io.Source to scala.io.Source to avoid looking into io.netty package
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
2013-04-07 14:37:12 +08:00
Andy Konwinski
5555811bd5
Update build to Scala 2.9.3
2013-04-04 13:26:45 -07:00
Nick Pentreath
0f54344fd8
Bumping Algebird version in examples now that it supports JDK 1.6
2013-04-03 13:15:34 +02:00
Reynold Xin
f130eb624c
Merge branch 'master' of github.com:mesos/spark into graph
2013-04-01 20:06:30 +08:00
Jey Kottalam
bc8ba222ff
Bump development version to 0.8.0
2013-03-28 15:42:01 -07:00
kalpit
f0164e5047
upgraded sbt version, sbt plugins and some library dependencies to latest stable version
2013-03-26 17:49:29 -07:00
Holden Karau
8456d673e2
Re-enable deprecation warnings since there are only two
2013-03-24 17:30:23 -07:00
Holden Karau
e104a76016
Makes the syntax highlighting on the build file not broken in emacs.
2013-03-24 16:16:05 -07:00
Reynold Xin
ba9d00c44a
Merge branch 'master' into graph
...
Conflicts:
run2.cmd
2013-03-18 18:30:14 +08:00
Prashant Sharma
15530c2b23
porting of repl to scala-2.10
2013-03-17 10:47:17 +05:30
seanm
42822cf95d
changing streaming resolver for akka
2013-03-13 11:40:42 -06:00
seanm
4aa1205202
adding typesafe repo to streaming resolvers so that akka-zeromq is found
2013-03-11 12:37:29 -06:00
Hiral Patel
664e5fd24b
Fix reference bug in Kryo serializer, add test, update version
2013-03-07 22:16:11 -08:00
Matei Zaharia
db9b90fdbd
Change version to 0.7.1-SNAPSHOT for development branch
2013-02-27 09:15:26 -08:00
Matei Zaharia
7e67c626ee
Change version number to 0.7.0
2013-02-25 20:30:47 -08:00
Matei Zaharia
6494cab19d
Update Hadoop dependency to 1.0.4
2013-02-25 15:38:21 -08:00
Prashant Sharma
254acb1666
Moving akka dependency resolver to shared.
2013-02-25 13:37:07 +05:30
Tathagata Das
5ab37be983
Fixed class paths and dependencies based on Matei's comments.
2013-02-24 16:24:52 -08:00
Tathagata Das
f282bc4960
Changed Algebird from 0.1.9 to 0.1.8
2013-02-24 12:44:12 -08:00
Tathagata Das
24c0cd6168
Fixed resolver for akka-zeromq
2013-02-22 18:23:29 -08:00
Tathagata Das
cfa65ebff1
Merge pull request #480 from MLnick/streaming-eg-algebird
...
[Streaming] Examples using Twitter's Algebird library
2013-02-22 12:29:04 -08:00
Nick Pentreath
718474b9c6
Bumping Algebird to 0.1.9
2013-02-21 12:11:31 +02:00
Prashant Sharma
4e5b09664c
fixes corresponding to review feedback at pull request #479
2013-02-20 19:14:52 +05:30
Reynold Xin
19d3b059e3
Merge branch 'master' into graph
2013-02-19 12:44:05 -08:00
Reynold Xin
81c4d19c61
Maven and sbt build changes for SparkGraph.
2013-02-19 12:43:13 -08:00
Prashant Sharma
f7d3e309cb
ZeroMQ stream as receiver
2013-02-19 19:32:52 +05:30
Nick Pentreath
315ea069e8
Merge remote-tracking branch 'upstream/streaming' into streaming-eg-algebird
...
Conflicts:
project/SparkBuild.scala
2013-02-19 13:58:05 +02:00
Nick Pentreath
015893f0e8
Adding streaming HyperLogLog example using Algebird
2013-02-19 13:21:33 +02:00
Tathagata Das
def8126d77
Added TwitterInputDStream from example to StreamingContext. Renamed example TwitterBasic to TwitterPopularTags.
2013-02-14 17:49:43 -08:00
Charles Reiss
0f81025eca
Add easymock to SBT configuration.
2013-01-29 18:55:42 -08:00
Matei Zaharia
6e3754bf47
Add Maven build file for streaming, and fix some issues in SBT file
...
As part of this, changed our Scala 2.9.2 Kafka library to be available
as a local Maven repository, following the example in
(http://blog.dub.podval.org/2010/01/maven-in-project-repository.html )
2013-01-20 19:22:24 -08:00
Tathagata Das
cd1521cfdb
Merge branch 'master' into streaming
...
Conflicts:
core/src/main/scala/spark/rdd/CoGroupedRDD.scala
core/src/main/scala/spark/rdd/FilteredRDD.scala
docs/_layouts/global.html
docs/index.md
run
2013-01-15 12:08:51 -08:00
folone
25c0739bad
Moved to scala 2.10.0. Notable changes are:
...
- akka 2.0.3 → 2.1.0
- spray 1.0-M1 → 1.1-M7
For now the repl subproject is commented out, as scala reflection api changed very much since the introduction of macros.
2013-01-14 09:52:11 +01:00
Matei Zaharia
6d1c230281
Merge pull request #357 from tysonjh/master
...
JSON support added to WebUI
2013-01-10 19:06:07 -08:00
Tyson
549ee388a1
Removed io.spray spray-json dependency as it is not needed.
2013-01-09 15:12:23 -05:00
Tyson
6e8c8f61c4
Added the spray implicit marshaller library
...
Added the io.spray JSON library
2013-01-09 10:40:33 -05:00
Stephen Haberman
c3f1675f9c
Retrieve jars to a flat directory so * can be used for the classpath.
2013-01-08 14:44:33 -06:00
Tathagata Das
64dceec293
Merge branch 'streaming-merge' into dev-merge
2013-01-07 16:54:35 -08:00
Shivaram Venkataraman
aed368a970
Update Hadoop dependency to 1.0.3 as 0.20 has Sun specific dependencies. Also
...
fix SequenceFileRDDFunctions to pick the right type conversion across Hadoop
versions
2013-01-07 15:57:33 -08:00
Tathagata Das
af8738dfb5
Moved Spark Streaming examples to examples sub-project.
2013-01-06 19:31:54 -08:00
Patrick Wendell
518111573f
Merge pull request #8 from radlab/twitter-example
...
Adding a Twitter InputDStream with an example
2012-12-29 14:23:01 -08:00
Tathagata Das
7c33f76291
Merge branch 'mesos' into dev-merge
2012-12-26 19:19:07 -08:00
Patrick Wendell
9ac4cb1c5f
Adding a Twitter InputDStream with an example
2012-12-21 17:18:19 -08:00
Matei Zaharia
3334b7c6b5
Merge pull request #341 from rxin/4a3fb06ac2d11125feb08acbbd4df76d1e91b677
...
Kryo2 update against Spark master
2012-12-21 15:31:23 -08:00
Reynold Xin
eac566a7f4
Merge branch 'master' of github.com:mesos/spark into dev
...
Conflicts:
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/PairRDDFunctions.scala
core/src/main/scala/spark/ParallelCollection.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/CartesianRDD.scala
core/src/main/scala/spark/rdd/CoGroupedRDD.scala
core/src/main/scala/spark/rdd/CoalescedRDD.scala
core/src/main/scala/spark/rdd/FilteredRDD.scala
core/src/main/scala/spark/rdd/FlatMappedRDD.scala
core/src/main/scala/spark/rdd/GlommedRDD.scala
core/src/main/scala/spark/rdd/HadoopRDD.scala
core/src/main/scala/spark/rdd/MapPartitionsRDD.scala
core/src/main/scala/spark/rdd/MapPartitionsWithSplitRDD.scala
core/src/main/scala/spark/rdd/MappedRDD.scala
core/src/main/scala/spark/rdd/PipedRDD.scala
core/src/main/scala/spark/rdd/SampledRDD.scala
core/src/main/scala/spark/rdd/ShuffledRDD.scala
core/src/main/scala/spark/rdd/UnionRDD.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockManagerId.scala
core/src/main/scala/spark/storage/BlockManagerMaster.scala
core/src/main/scala/spark/storage/StorageLevel.scala
core/src/main/scala/spark/util/MetadataCleaner.scala
core/src/main/scala/spark/util/TimeStampedHashMap.scala
core/src/test/scala/spark/storage/BlockManagerSuite.scala
run
2012-12-20 14:53:40 -08:00
Reynold Xin
9397c5014e
Let the slave notify the master block removal.
2012-12-20 01:37:09 -08:00
Patrick Wendell
3ff9710265
Adding Flume InputDStream
2012-12-07 16:42:39 -08:00
Denny
556c38ed91
Added kafka JAR
2012-12-05 11:54:42 -08:00
Denny
0c1de43fc7
Working on kafka.
2012-11-06 09:41:42 -08:00
Matei Zaharia
863a55ae42
Merge remote-tracking branch 'public/master' into dev
...
Conflicts:
core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
core/src/main/scala/spark/KryoSerializer.scala
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/executor/Executor.scala
core/src/main/scala/spark/network/Connection.scala
core/src/main/scala/spark/network/ConnectionManagerTest.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/NewHadoopRDD.scala
core/src/main/scala/spark/scheduler/ShuffleMapTask.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockMessage.scala
core/src/main/scala/spark/storage/BlockStore.scala
core/src/main/scala/spark/storage/StorageLevel.scala
core/src/main/scala/spark/util/AkkaUtils.scala
project/SparkBuild.scala
run
2012-10-24 23:21:00 -07:00
Matei Zaharia
0967e71a00
Bump up version to 0.7.0-SNAPSHOT for master branch
2012-10-22 11:49:42 -07:00
Matei Zaharia
902a608187
Update version to 0.6.1-SNAPSHOT to show this is in development
2012-10-22 11:43:57 -07:00
Thomas Dudziak
d9c2a89c57
Support for Hadoop 2 distributions such as cdh4
2012-10-18 16:08:54 -07:00
Reynold Xin
4a3fb06ac2
Updated Kryo to 2.20.
2012-10-16 01:10:01 -07:00
Patrick Wendell
629dd2691e
Removing credentials line in build.
2012-10-14 19:33:39 -07:00
Matei Zaharia
f8768da418
Comment out PGP stuff for publish-local to work
2012-10-14 17:37:21 -07:00
Matei Zaharia
64b52166ee
Changed default Hadoop version back to 0.20.205
2012-10-14 09:51:34 -07:00
Matei Zaharia
ce6b5a3ee5
Uncomment Maven publishing stuff and set version to 0.6.0
2012-10-13 15:55:39 -07:00
Patrick Wendell
6d328f54d0
Changing tabs to spaces
2012-10-10 18:54:22 -07:00
Patrick Wendell
3ed172ea59
Adding code for publishing to Sonatype.
...
By default - I'm leaving this commented out. This is because
there is a bug in the PGP signing plugin which causes it to active
even duing a publish-local. So we'll just uncomment when we decide
to publish.
2012-10-10 17:25:29 -07:00
Andy Konwinski
5897567679
Removes the included mesos-0.9.0.jar and adds a libraryDependency to
...
the build file so that mesos-0.9.0-incubating.jar (which contains the
same class files, but has a silightly different name) will be pulled
down from Maven Central instead.
2012-10-03 08:58:05 -07:00
Matei Zaharia
6112b1a83c
Don't build an assembly for the REPL
2012-10-02 17:08:16 -07:00
Matei Zaharia
a925754675
Place Spray in front of Cloudera in Maven search path
2012-10-02 12:02:00 -07:00
Matei Zaharia
22684653a5
Revert "Place Spray repo ahead of Cloudera in Maven search path"
...
This reverts commit 42e0a68082
.
2012-10-02 12:01:32 -07:00
Matei Zaharia
42e0a68082
Place Spray repo ahead of Cloudera in Maven search path
2012-10-02 11:37:19 -07:00
Patrick Wendell
6fee76d6d5
publish-local should go to maven + ivy by default
2012-10-01 15:34:47 -07:00
Reynold Xin
5783236ae6
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 00:17:13 -07:00
Matei Zaharia
35cc9f13e9
Update Akka to 2.0.3
2012-09-24 14:17:10 -07:00
Matei Zaharia
1f539aa473
Update Scala version dependency to 2.9.2
2012-09-24 14:12:48 -07:00
Tathagata Das
7419d2c7ea
Added transformRDD DStream operation and TransformedDStream. Added sbt assembly option for streaming project.
2012-09-02 02:35:17 -07:00
Matei Zaharia
5a8015d2db
Merge remote-tracking branch 'public/dev' into dev
2012-08-24 16:11:44 -07:00
Denny
0008994044
merged dev branch
2012-08-02 16:00:33 -07:00
Matei Zaharia
71a958b0b7
Merge branch 'dev' of github.com:mesos/spark into dev
...
Conflicts:
project/SparkBuild.scala
2012-08-02 17:23:13 -04:00
Denny
ba7e30fb5e
Mostly stlyistic changes.
2012-08-02 13:55:09 -07:00
Josh Rosen
039b41cb54
Use sbt mergeStrategy for reference.conf files.
...
Cleans up #158 / 509b721
.
2012-08-02 10:21:50 -07:00
Denny
0ee44c225e
Spark standalone mode cluster scripts.
...
Heavily inspired by Hadoop cluster scripts ;-)
2012-08-01 20:38:52 -07:00
Denny
1b29e90a79
merge dev branch
2012-08-01 14:06:09 -07:00
Denny
7a295fee96
Spark WebUI Implementation.
2012-08-01 11:01:09 -07:00
Josh Rosen
509b721d12
Fix Akka configuration in assembly jar.
...
This resolves an issue where running Spark from
the assembly jar would cause a "No configuration
setting found for key 'akka.version'" exception.
This solution is from the Akka Team Blog:
http://letitcrash.com/post/21025950392/
2012-07-30 18:05:13 -07:00
Matei Zaharia
47b7ebad12
Added the Spark Streaing code, ported to Akka 2
2012-07-28 20:03:26 -07:00
Josh Rosen
01dce3f569
Add Java API
...
Add distinct() method to RDD.
Fix bug in DoubleRDDFunctions.
2012-07-18 17:34:29 -07:00
Matei Zaharia
2fb6e7d71e
Initial framework to get a master and web UI up.
2012-06-30 14:45:55 -07:00
Matei Zaharia
c53670b9bf
Various code style fixes, mostly from IntelliJ IDEA
2012-06-29 18:47:12 -07:00
rrmckinley
697b0bee2c
Scalacheck groupId has changed https://github.com/rickynils/scalacheck/issues/24 . Necessary to build with scalaVersion 2.9.2. Works with 2.9.1 too.
2012-06-29 16:42:05 -07:00
Matei Zaharia
3920189932
Upgraded to Akka 2 and fixed test execution (which was still parallel
...
across projects).
2012-06-28 23:51:28 -07:00
Tathagata Das
ede615d719
Fixed issues duplicate class issues in sbt assembly.
2012-06-22 15:03:09 -07:00
Matei Zaharia
3e0396c953
Update SBT and SBT-Eclipse version
2012-06-17 14:37:18 -07:00
Matei Zaharia
08579ffa11
Update version number for dev branch
2012-06-15 23:55:43 -07:00
Matei Zaharia
f58da6164e
Merge branch 'master' into dev
2012-06-15 23:47:11 -07:00
Matei Zaharia
4449eb9783
Changed version in master branch to 0.5.1-SNAPSHOT for further
...
development.
2012-06-13 22:26:14 -04:00
Matei Zaharia
0472cf8e0d
Update version in SBT
2012-06-12 14:30:49 -04:00
Matei Zaharia
63051dd2bc
Merge in engine improvements from the Spark Streaming project, developed
...
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
2012-06-07 12:45:38 -07:00
Matei Zaharia
95fb1a16b8
Use Mesos 0.9 RC3 JAR and protobuf 2.4.1
2012-03-30 11:38:49 -04:00
Matei Zaharia
4d52cc6738
Merge branch 'master' into mesos-0.9
2012-03-29 21:29:39 -04:00
Matei Zaharia
ca5c19c1ba
Remove dependency on Akka
2012-03-29 01:03:34 -04:00
Reynold Xin
90418b70ff
Added sbt-assembly for spark-repl project so we can generate an
...
assembled jar for Shark.
2012-03-22 18:46:31 -07:00
Matei Zaharia
a099a63a8a
Initial work to make Spark compile with Mesos 0.9 and Hadoop 1.0
2012-03-17 12:31:34 -07:00
Matei Zaharia
d6ec664b48
Add dependency on fastutil and update Guava
2012-02-06 15:37:27 -08:00
Matei Zaharia
5fd101d79e
Add dependency on Akka and Netty
2011-12-15 13:21:14 +01:00
Matei Zaharia
c7d6f1a65c
Really upgrade to SBT 0.11.1 (through build.properties and plugin changes)
2011-11-08 21:45:29 -08:00
root
49505a0b0b
Switched Jetty to version 7.5 because 8.0 was causing a conflict with the log4j and Jetty libraries in Hadoop.
2011-10-17 18:06:41 +00:00
Ismael Juma
d76c0fc781
Upgrade to sbt-idea 0.11.0 final.
2011-09-27 23:13:38 +01:00
Ismael Juma
7e92ef9d19
Add workaround for bug in SBT (issue #206 ).
2011-09-27 00:04:59 +01:00
Ismael Juma
3562db6374
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 22:41:07 +01:00
Ismael Juma
28b5d5a2af
Upgrade compress-lzf to 0.8.4.
2011-09-26 22:32:05 +01:00
Ismael Juma
315e55fde3
Upgrade Jetty to 8.0.1.
2011-09-26 22:32:05 +01:00
Ismael Juma
ee980439e2
Use scalatest and scalacheck compiled against Scala 2.9.1.
2011-09-26 22:32:05 +01:00
Ismael Juma
bd774eb274
Use new layout for plugins definitions (recommended for SBT 0.11)
2011-09-26 22:32:05 +01:00
Ismael Juma
e39edcce60
Upgrade to SBT 0.11.0.
2011-09-26 22:24:29 +01:00
Ismael Juma
483f724d62
Upgrade to Scala 2.9.1.
...
Interestingly, the version in Maven is 2.9.1, but SBT outputs file to the 2.9.1.final
directory inside target.
A couple of small changes in SparkIMain were also required.
All tests pass and ./spark-shell launches successfully.
2011-08-31 10:43:05 +01:00
Matei Zaharia
9b7215d74a
Only include JAR files in lib in the unmanaged classpath.
2011-08-29 22:59:18 -07:00
Matei Zaharia
4cc5e51e28
Use the new SBT 0.10 assembly plugin instead of the DepJar code we had.
2011-08-29 16:13:26 -07:00
Ismael Juma
2b6fd3198d
Fix issue #69 : Enable -optimize in the build
2011-08-02 10:34:41 +01:00
Ismael Juma
59c7131dff
Add note about why we can't enable -deprecation switch.
2011-08-02 10:26:17 +01:00
Matei Zaharia
2e57338896
Merge branch 'scala-2.9'
2011-08-01 15:27:08 -07:00
Matei Zaharia
711575391d
Merge branch 'scala-2.9'
...
Conflicts:
project/build/SparkProject.scala
2011-08-01 15:25:26 -07:00
Ismael Juma
3f5d7b5d11
Add publishTo configuration that publishes to target directory.
...
Until Spark is available in a Maven repository, this makes it easy to deploy to
e.g. GitHub pages by copying target/maven to the repository.
2011-07-31 20:17:58 +01:00
Ismael Juma
7565cc2e33
Add type annotation to result of depJarSettings to workaround scalac bug.
2011-07-31 20:17:58 +01:00
Matei Zaharia
d12122502b
Various improvements to Kryo serializer:
...
- Replaced modified Kryo version with the standard one augmented with
the kryo-serializers package, which includes support for classes with
no-arg constructors (that was why we had a modified Kryo before)
- The kryo-serializers version also fixes issue #72 .
- Added a bunch of tests.
- Serialize maps and a few other common types properly by default.
2011-07-21 22:09:33 -07:00
Matei Zaharia
2bfd7931e8
Merge branch 'new-rdds-protobuf'
...
Conflicts:
core/src/main/scala/spark/Executor.scala
core/src/main/scala/spark/RDD.scala
2011-07-21 16:08:39 -07:00
Ismael Juma
fc0a2c8db8
Add and configure junit_xml_listener as a replacement for XmlTestReport.
2011-07-21 01:04:29 +01:00
Ismael Juma
51673ca62e
Introduce DepJarPlugin based on AssemblyPlugin and use it in SparkBuild.
2011-07-18 10:34:51 +01:00
Ismael Juma
c71fc27c74
Update sbt-idea to non-snapshot version and uncomment sbt-assembly dependency.
2011-07-18 00:17:21 +01:00
Ismael Juma
8531c2a079
Update test dependencies.
2011-07-18 00:16:08 +01:00
Ismael Juma
635f501492
Fix copy & paste error in version.
2011-07-18 00:13:37 +01:00
Ismael Juma
f686e3dacb
Initial work on converting build to SBT 0.10.1
2011-07-15 03:38:30 +01:00
Matei Zaharia
cf8f5de61b
Merge branch 'master' into scala-2.9
...
Conflicts:
project/build.properties
repl/src/main/scala/spark/repl/SparkInterpreterLoop.scala
2011-07-14 17:48:56 -04:00
Matei Zaharia
02678724a4
Update version number to 0.4-SNAPSHOT
2011-07-14 17:47:39 -04:00
Matei Zaharia
7c77b2fa6a
Merge branch 'master' into scala-2.9
...
Conflicts:
project/build.properties
2011-07-14 17:39:34 -04:00
Matei Zaharia
c86af80022
Change version to 0.3
2011-07-14 17:38:43 -04:00
Matei Zaharia
d05fea24f3
Simplified parallel shuffle fetcher to use URLConnection
2011-07-11 22:12:36 -04:00
Matei Zaharia
aea5cb4413
Added parallel shuffle fetcher
2011-07-09 17:25:56 -04:00
Matei Zaharia
c62bb4091b
Merge remote-tracking branch 'origin/master' into scala-2.9
2011-06-07 00:42:23 -07:00
Ismael Juma
1ad4dcd3de
Move managedStyle to SparkProject.
...
I had added it to DepJar by mistake.
2011-06-02 14:06:54 +01:00
Ismael Juma
3def9fdb96
Upgrade to scalacheck 1.9.
2011-05-31 22:11:33 -07:00
Matei Zaharia
beb9c117f0
Merge branch 'master' into scala-2.9
...
Conflicts:
project/build/SparkProject.scala
2011-05-31 19:23:07 -07:00
Matei Zaharia
bcce6e8d01
Various work to use the 2.9 interpreter
2011-05-31 17:31:51 -07:00
Matei Zaharia
4096c2287e
Various fixes
2011-05-29 18:46:01 -07:00
Ismael Juma
0c62ee4321
Depend on jetty-server in compile scope and upgrade to 7.4.2.
...
As Matei described: "We're using Jetty to run an HTTP server, not to embed Spark
in a webapp"
2011-05-29 20:12:50 +01:00
Ismael Juma
1d75c6060a
Update to Scala 2.9.0-1 and disable repl module for now.
...
The repl module requires more complex work.
2011-05-27 14:59:23 +01:00
Ismael Juma
e3b323321d
Use ManagedStyle.Maven.
2011-05-27 14:56:01 +01:00
Ismael Juma
3a6b0b8a57
Publish javadoc and sources.
2011-05-27 14:55:51 +01:00
Ismael Juma
3af6003c87
Update sbt to 0.7.7.
2011-05-27 11:22:59 +01:00
Ismael Juma
1396678baa
Move REPL classes to separate module.
2011-05-27 11:22:50 +01:00
Ismael Juma
3e8114ddbd
Change project.organization to org.spark-project to fit Maven convention.
2011-05-27 11:22:10 +01:00
Ismael Juma
7b7dfdb085
Set project.version to 0.3-SNAPSHOT.
2011-05-27 11:22:10 +01:00
Ismael Juma
ae1a1f91f1
Remove several dependencies from git and configure them as SBT managed dependencies.
...
Upgrade some of the dependencies while at it.
2011-05-27 11:22:01 +01:00
Ismael Juma
164ef4c751
Use explicit asInstanceOf instead of misleading unchecked pattern matching.
...
Also enable -unchecked warnings in SBT build file.
2011-05-27 07:57:10 +01:00
Ismael Juma
222729171e
Upgrade sbt-idea to 0.4.0.
2011-05-26 21:48:08 +01:00
Matei Zaharia
4b1f0f1ce4
Merge pull request #48 from ankurdave/bagel-new
...
Bagel: Large-scale graph processing on Spark
2011-05-12 21:34:38 -07:00
Matei Zaharia
7e20648914
Upgraded to SBT 0.7.5
2011-05-09 14:48:39 -07:00
Ankur Dave
1c8ca0ebe1
Add Bagel test suite
...
Note: This test suite currently fails for the same reason that the
Spark Core test suite fails: Spark currently seems to have a bug where
any test after the first one fails.
2011-05-03 15:40:31 -07:00
Ankur Dave
c0736f6f68
Add Bagel, an implementation of Pregel on Spark
2011-05-03 15:37:08 -07:00
Matei Zaharia
62f1c6f5a8
Remove build.properties from version control
2011-02-09 11:52:56 -08:00
Matei Zaharia
d3df963a13
Brought in some reorganization of build file from Hive branch
2011-02-08 21:27:36 -08:00
Matei Zaharia
50df43bf7b
Added SBT target for building a single JAR with Spark Core and its
...
dependencies
2011-02-02 19:08:14 -08:00
Matei Zaharia
ec28b607fd
Merge branch 'master' into sbt
...
Conflicts:
Makefile
core/src/main/java/spark/compress/lzf/LZF.java
core/src/main/java/spark/compress/lzf/LZFInputStream.java
core/src/main/java/spark/compress/lzf/LZFOutputStream.java
core/src/main/native/spark_compress_lzf_LZF.c
run
2011-02-02 00:25:54 -08:00
Matei Zaharia
7f74ee99f6
Added support for IntelliJ IDEA
2011-02-02 00:08:13 -08:00
Matei Zaharia
e5c4cd8a5e
Made examples and core subprojects
2011-02-01 15:11:08 -08:00
Matei Zaharia
dcfa2ce83b
Further improvements -- build native stuff in target directory and add a
...
test-report target for XML test reports
2010-11-14 00:46:19 -08:00
Matei Zaharia
e86b620f9e
Fixed some more stuff (Eclispe target and native build)
2010-11-13 22:46:00 -08:00
Matei Zaharia
89fcd96702
Initial work to get Spark compiling with SBT 0.7.5 RC0
2010-11-13 22:07:08 -08:00