ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
30957 commits 6 branches 181 tags 397 MiB
Commit graph

1000 commits

Author SHA1 Message Date
Raymond Liu aec96dd108 Change profile name new-yarn to hadoop2.2-yarn 2014-01-03 12:12:37 +08:00
Raymond Liu d1528c7c8c Fix pom for yarn code reorgnaize commit 2014-01-03 12:12:37 +08:00
liguoqiang b5d0b3b0f7 restore core/pom.xml file modification 2014-01-01 11:30:08 +08:00
Reynold Xin 8b8e70ebde Merge pull request #73 from falaki/ApproximateDistinctCount 
Approximate distinct count

Added countApproxDistinct() to RDD and countApproxDistinctByKey() to PairRDDFunctions to approximately count distinct number of elements and distinct number of values per key, respectively. Both functions use HyperLogLog from stream-lib for counting. Both functions take a parameter that controls the trade-off between accuracy and memory consumption. Also added Scala docs and test suites for both methods.
 2013-12-31 17:48:24 -08:00
Tathagata Das 97630849ff Added pom.xml for external projects and removed unnecessary dependencies and repositoris from other poms and sbt. 2013-12-31 00:28:57 -08:00
Hossein Falaki d50ccc5ca9 Using origin version 2013-12-30 15:08:34 -08:00
Binh Nguyen 040dd3ecd5 upgrade Netty from 4.0.0.Beta2 to 4.0.13.Final 2013-12-24 14:58:18 -08:00
Patrick Wendell c1c0f8099f Clean-up 2013-12-16 22:01:27 -08:00
Patrick Wendell c1fec89895 Cleanup 2013-12-16 21:56:21 -08:00
Patrick Wendell ceb013f8b9 Remove trailing slashes from repository specifications. 
The correct format is to not have a trailing slash.

For me this caused non-deterministic failures due to issues fetching
certain artifacts. The issue was that some of the maven caches would
fail to fetch the artifact (due to the way that the artifact
path was concatenated with the repository) and this short-circuited
the download process in a silent way. Here is what the log output
looked like:

    Downloading: http://repo.maven.apache.org/maven2/org/spark-project/akka/akka-remote_2.10/2.2.3-shaded-protobuf/akka-remote_2.10-2.2.3-shaded-protobuf.pom
    [WARNING] The POM for org.spark-project.akka:akka-remote_2.10🫙2.2.3-shaded-protobuf is missing, no dependency information available

This was pretty brutal to debug since there was no error message
anywhere and the path *looks* correct as reported by the Maven log.
 2013-12-16 21:53:51 -08:00
Patrick Wendell c6f95e603e Attempt with extra repositories 2013-12-16 21:53:51 -08:00
Mark Hamstra 09ed7ddfa0 Use scala.binary.version in POMs 2013-12-15 12:39:58 -08:00
Patrick Wendell 6e8a96c7e7 Fix maven build issues in 2.10 branch 2013-12-13 23:14:08 -08:00
Prashant Sharma 589b83a18f Disabled yarn 2.2 and added a message in the sbt build 2013-12-12 16:25:30 +05:30
Prashant Sharma 603af51bb5 Merge branch 'master' into akka-bug-fix 
Conflicts:
	core/pom.xml
	core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
	pom.xml
	project/SparkBuild.scala
	streaming/pom.xml
	yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
 2013-12-11 10:21:53 +05:30
Prashant Sharma 17db6a9041 Style fixes and addressed review comments at #221 2013-12-10 11:47:16 +05:30
Prashant Sharma 7ad6921ae0 Incorporated Patrick's feedback comment on #211 and made maven build/dep-resolution atleast a bit faster. 2013-12-07 12:45:57 +05:30
Raymond Liu 4738818dd6 Fix pom.xml for maven build 2013-12-03 16:36:05 +08:00
Prashant Sharma 44fd30d3fb Merge branch 'master' into scala-2.10-wip 
Conflicts:
	core/src/main/scala/org/apache/spark/rdd/RDD.scala
	project/SparkBuild.scala
 2013-11-25 18:10:54 +05:30
Reynold Xin 6bcac986b2 Merge branch 'master' of github.com:apache/incubator-spark 2013-11-25 15:47:47 +08:00
LiGuoqiang 989203604e Fix Maven build for metrics-graphite 2013-11-25 11:23:11 +08:00
Raymond Liu a60620b76a Merge branch 'master' into scala-2.10 2013-11-14 12:44:19 +08:00
Raymond Liu 0f2e3c6e31 Merge branch 'master' into scala-2.10 2013-11-13 16:55:11 +08:00
tgravescs a35472e1dd Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs. 2013-11-04 16:16:28 -06:00
Ankur Dave 5064f9b2d2 Merge remote-tracking branch 'spark-upstream/master' 
Conflicts:
	project/SparkBuild.scala
 2013-10-30 15:59:09 -07:00
Matei Zaharia dadfc63b03 Fix Maven build to use MQTT repository 2013-10-23 15:29:22 -07:00
Reynold Xin 4e44d65b5e Exclusion rules for Maven build files. 2013-10-19 12:35:55 -07:00
Hossein Falaki 13227aaa28 Added stream-lib dependency to Maven build 2013-10-18 14:10:24 -07:00
Joseph E. Gonzalez 1b22eef744 Merge branch 'master' of https://github.com/apache/incubator-spark into indexedrdd_graphx 2013-10-15 16:15:19 -07:00
Henry Saputra 3fed3e2283 Update pom.xml to use version 13 of the ASF parent pom and add mailingLists element. 2013-10-14 23:10:54 -07:00
Joseph E. Gonzalez ef7c369092 merged with upstream changes 2013-10-14 22:56:42 -07:00
Matei Zaharia c71499b779 Merge pull request #19 from aarondav/master-zk 
Standalone Scheduler fault tolerance using ZooKeeper

This patch implements full distributed fault tolerance for standalone scheduler Masters.
There is only one master Leader at a time, which is actively serving scheduling
requests. If this Leader crashes, another master will eventually be elected, reconstruct
the state from the first Master, and continue serving scheduling requests.

Leader election is performed using the ZooKeeper leader election pattern. We try to minimize
the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of
retries and session monitoring on top of the ZooKeeper client.

Master failover follows directly from the single-node Master recovery via the file
system (patch d5a96fe), save that the Master state is stored in ZooKeeper instead.

Configuration:
By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE).
By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url
to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled.
By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory
to an appropriate directory accessible by the Master, we will keep the behavior of from d5a96fe.

Additionally, places where a Master could be specificied by a spark:// url can now take
comma-delimited lists to specify backup masters. Note that this is only used for registration
of NEW Workers and application Clients. Once a Worker or Client has registered with the
Master Leader, it is "in the system" and will never need to register again.
 2013-10-10 17:16:42 -07:00
Prashant Sharma 26860639c5 Merge branch 'scala-2.10' of github.com:ScrapCodes/spark into scala-2.10 
Conflicts:
	core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
	project/SparkBuild.scala
 2013-10-10 09:42:23 +05:30
Prashant Sharma 7be75682b9 Merge branch 'master' into wip-merge-master 
Conflicts:
	bagel/pom.xml
	core/pom.xml
	core/src/test/scala/org/apache/spark/ui/UISuite.scala
	examples/pom.xml
	mllib/pom.xml
	pom.xml
	project/SparkBuild.scala
	repl/pom.xml
	streaming/pom.xml
	tools/pom.xml

In scala 2.10, a shorter representation is used for naming artifacts
 so changed to shorter scala version for artifacts and made it a property in pom.
 2013-10-08 11:29:40 +05:30
Patrick Wendell aa9fb84994 Merging build changes in from 0.8 2013-10-05 22:07:00 -07:00
Martin Weindel 9b0c9c893d scala 2.10 requires Java 1.6, 
using Scala 2.10.3,
resolved maven-scala-plugin warning
 2013-10-05 21:41:09 +02:00
Prashant Sharma 5829692885 Merge branch 'master' into scala-2.10 
Conflicts:
	core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala
	docs/_config.yml
	project/SparkBuild.scala
	repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
 2013-10-01 11:57:24 +05:30
Aaron Davidson f549ea33d3 Standalone Scheduler fault tolerance using ZooKeeper 
This patch implements full distributed fault tolerance for standalone scheduler Masters.
There is only one master Leader at a time, which is actively serving scheduling
requests. If this Leader crashes, another master will eventually be elected, reconstruct
the state from the first Master, and continue serving scheduling requests.

Leader election is performed using the ZooKeeper leader election pattern. We try to minimize
the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of
retries and session monitoring on top of the ZooKeeper client.

Master failover follows directly from the single-node Master recovery via the file
system (patch 194ba4b8), save that the Master state is stored in ZooKeeper instead.

Configuration:
By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE).
By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url
to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled.
By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory
to an appropriate directory accessible by the Master, we will keep the behavior of from 194ba4b8.

Additionally, places where a Master could be specificied by a spark:// url can now take
comma-delimited lists to specify backup masters. Note that this is only used for registration
of NEW Workers and application Clients. Once a Worker or Client has registered with the
Master Leader, it is "in the system" and will never need to register again.

Forthcoming:
Documentation, tests (! - only ad hoc testing has been performed so far)
I do not intend for this commit to be merged until tests are added, but this patch should
still be mostly reviewable until then.
 2013-09-26 15:04:23 -07:00
Reynold Xin 3f283278b0 Removed scala -optimize flag. 2013-09-26 13:58:10 -07:00
Prashant Sharma 604dc40996 Sync with master and some build fixes 2013-09-26 11:40:02 +05:30
Prashant Sharma 7ff4c2d399 fixed maven build for scala 2.10 2013-09-26 10:48:24 +05:30
Patrick Wendell 6079721fa1 Update build version in master 2013-09-24 11:41:51 -07:00
Joseph E. Gonzalez 8b59fb72c4 Merging latest changes from spark main branch 2013-09-17 20:56:12 -07:00
Patrick Wendell c856860c5b Bumping Mesos version to 0.13.0 2013-09-15 12:46:26 -07:00
Prashant Sharma a90e0eff59 version changed 2.9.3 -> 2.10 in shell script. 2013-09-15 12:47:20 +05:30
Prashant Sharma 383e151fd7 Merge branch 'master' of git://github.com/mesos/spark into scala-2.10 
Conflicts:
	core/src/main/scala/org/apache/spark/SparkContext.scala
	project/SparkBuild.scala
 2013-09-15 10:55:12 +05:30
Patrick Wendell e9eba8c3ce Use different Hadoop version for YARN artifacts. 
This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.
 2013-09-13 15:34:57 -07:00
Patrick Wendell 45ec1cc90e Add git scm url for publishing 2013-09-12 13:47:31 -07:00
Matei Zaharia f117dc6d0d Add explicit jets3t dependency, which is excluded in hadoop-client 2013-09-10 06:39:25 +00:00
Matei Zaharia 0456384939 Merge pull request #911 from pwendell/ganglia-sink 
Adding Manen dependency for Ganglia
 2013-09-09 09:57:54 -07:00
Patrick Wendell 528fdbae97 Adding Manen dependency 2013-09-09 09:32:18 -07:00
Jey Kottalam 70661246fd Fix YARN assembly generation under Maven 2013-09-06 11:31:16 -07:00
Prashant Sharma 4106ae9fbf Merged with master 2013-09-06 17:53:01 +05:30
Matei Zaharia 59218bdd49 Add Apache parent POM 2013-09-02 18:34:03 -07:00
Matei Zaharia 5701eb92c7 Fix some URLs 2013-09-01 14:13:16 -07:00
Matei Zaharia 46eecd110a Initial work to rename package to org.apache.spark 2013-09-01 14:13:13 -07:00
Matei Zaharia 666d93c294 Update Maven build to create assemblies expected by new scripts 
This includes the following changes:
- The "assembly" package now builds in Maven by default, and creates an
  assembly containing both hadoop-client and Spark, unlike the old
  BigTop distribution assembly that skipped hadoop-client
- There is now a bigtop-dist package to build the old BigTop assembly
- The repl-bin package is no longer built by default since the scripts
  don't reply on it; instead it can be enabled with -Prepl-bin
- Py4J is now included in the assembly/lib folder as a local Maven repo,
  so that the Maven package can link to it
- run-example now adds the original Spark classpath as well because the
  Maven examples assembly lists spark-core and such as provided
- The various Maven projects add a spark-yarn dependency correctly
 2013-08-29 21:19:06 -07:00
Matei Zaharia 8d81358a05 Provide more memory for tests 2013-08-29 21:19:06 -07:00
Reynold Xin 9db1e50344 Revert "Merge pull request #841 from rxin/json" 
This reverts commit 1fb1b09928, reversing
changes made to c69c48947d.
 2013-08-26 11:05:14 -07:00
Matei Zaharia ae8ba83ef2 Merge pull request #855 from jey/update-build-docs 
Update build docs
 2013-08-22 10:14:54 -07:00
Mark Hamstra ff6f1b0500 Synced sbt and maven builds 2013-08-21 13:50:24 -07:00
Jey Kottalam 31644a011c Use "hadoop.version" property when specifying Hadoop YARN version too 2013-08-21 13:24:28 -07:00
Reynold Xin af602ba9d3 Downgraded default build hadoop version to 1.0.4. 2013-08-21 11:38:24 -07:00
Matei Zaharia aa2b89d98d Merge remote-tracking branch 'jey/hadoop-agnostic' 
Conflicts:
	core/src/main/scala/spark/PairRDDFunctions.scala
 2013-08-20 10:14:15 -07:00
Jey Kottalam bdd861c6c3 Fix Maven build with Hadoop 0.23.9 2013-08-18 18:28:57 -07:00
Jey Kottalam ad580b94d5 Maven build now also works with YARN 2013-08-16 13:50:12 -07:00
Jey Kottalam 11b42a84db Maven build now works with CDH hadoop-2.0.0-mr1 2013-08-16 13:50:12 -07:00
Jey Kottalam 353fab2440 Initial changes to make Maven build agnostic of hadoop version 2013-08-16 13:50:12 -07:00
Reynold Xin c961c19b7b Use the JSON formatter from Scala library and removed dependency on lift-json. 
It made the JSON creation slightly more complicated, but reduces one external dependency. The scala library also properly escape "/" (which lift-json doesn't).
 2013-08-15 18:23:01 -07:00
Jey Kottalam a0f0848463 Update default version of Hadoop to 1.2.1 2013-08-15 16:50:37 -07:00
Matei Zaharia d9588183fa Update to Mesos 0.12.1 2013-08-13 18:51:35 -07:00
Patrick Wendell a0133bfbad Merge pull request #784 from jerryshao/dev-metrics-servlet 
Add MetricsServlet for Spark metrics system
 2013-08-13 09:28:18 -07:00
jerryshao 320e87e7ab Add MetricsServlet for Spark metrics system 2013-08-12 13:23:23 +08:00
Alexander Pivovarov ca28f2e639 Changed yarn.version to 2.0.5 in pom.xml 2013-08-10 22:50:04 -07:00
Matei Zaharia 5a4003c1ac Update to Chill 0.3.1 2013-08-08 13:30:27 -07:00
Joseph E. Gonzalez 499a0d8383 Merged graphx from @rxin into master 2013-08-06 12:28:29 -07:00
Matei Zaharia b2b86c2575 Merge pull request #753 from shivaram/glm-refactor 
Build changes for ML lib
 2013-07-31 15:51:39 -07:00
Reynold Xin 311aae76a2 Added Snappy dependency to Maven build files. 2013-07-30 17:25:42 -07:00
Shivaram Venkataraman 48851d4dd9 Add bagel, mllib to SBT assembly. 
Also add jblas dependency to mllib pom.xml
 2013-07-30 14:03:15 -07:00
Matei Zaharia 8eb8b52997 Fix Chill version in Maven 2013-07-25 08:58:02 -07:00
ryanlecompte 30a369a808 update pom.xml 2013-07-24 20:55:48 -07:00
Matei Zaharia c258718606 Fix Maven build errors after previous commits 2013-07-24 16:12:32 -07:00
Matei Zaharia 5584ebcbd3 Merge pull request #675 from c0s/assembly 
Building spark assembly for further consumption of the Spark project with a deployed cluster
 2013-07-24 11:46:46 -07:00
jerryshao a79f6077f0 Add Maven metrics library dependency and code changes 2013-07-24 14:57:47 +08:00
Josh Rosen c83680434b Add JavaAPICompletenessChecker. 
This is used to find methods in the Scala API that
need to be ported to the Java API.  To use it:

  ./run spark.tools.JavaAPICompletenessChecker
Conflicts:
	project/SparkBuild.scala
	run
	run2.cmd
 2013-07-22 16:11:49 -07:00
Konstantin Boudnik f4d514810e Building spark assembly for further consumption of the Spark project with a deployed cluster 2013-07-21 11:47:29 -07:00
Matei Zaharia af3c9d5042 Add Apache license headers and LICENSE and NOTICE files 2013-07-16 17:21:33 -07:00
Matei Zaharia 00a14deb6d Update to latest Scala Maven plugin and allow Zinc external compiler 2013-07-16 11:52:20 -07:00
Prashant Sharma e86d5dbaad Merge branch 'master' into master-merge 
Conflicts:
	README.md
	core/pom.xml
	core/src/main/scala/spark/deploy/JsonProtocol.scala
	core/src/main/scala/spark/deploy/LocalSparkCluster.scala
	core/src/main/scala/spark/deploy/master/Master.scala
	core/src/main/scala/spark/deploy/master/MasterWebUI.scala
	core/src/main/scala/spark/deploy/worker/Worker.scala
	core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
	core/src/main/scala/spark/storage/BlockManagerUI.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	pom.xml
	project/SparkBuild.scala
	streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
 2013-07-12 14:49:16 +05:30
Prashant Sharma 69ae7ea227 Removed some unnecessary code and fixed dependencies 2013-07-11 18:30:18 +05:30
Mark Hamstra 0b39d66f3f pom cleanup 2013-07-08 16:07:09 -07:00
Matei Zaharia fd6665122b Fix some other references to Cloudera Avro and updated Avro version 2013-07-06 16:45:15 -07:00
Matei Zaharia 22161887ee Merge pull request #676 from c0s/asf-avro 
Use standard ASF published avro module instead of a proprietory built one
 2013-07-06 16:18:15 -07:00
Matei Zaharia 1ffadb2d9e Merge remote-tracking branch 'pwendell/ui-updates' 
Conflicts:
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	pom.xml
 2013-07-06 15:51:41 -07:00
Konstantin Boudnik 7687ed5292 Use standard ASF published avro module instead of a proprietory built one 2013-07-04 13:48:33 -07:00
Prashant Sharma a5f1f6a907 Merge branch 'master' into master-merge 
Conflicts:
	core/pom.xml
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/RDDCheckpointData.scala
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/Utils.scala
	core/src/main/scala/spark/api/python/PythonRDD.scala
	core/src/main/scala/spark/deploy/client/Client.scala
	core/src/main/scala/spark/deploy/master/MasterWebUI.scala
	core/src/main/scala/spark/deploy/worker/Worker.scala
	core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/ZippedRDD.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/main/scala/spark/storage/BlockManagerMasterActor.scala
	core/src/main/scala/spark/storage/BlockManagerUI.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	core/src/test/scala/spark/SizeEstimatorSuite.scala
	pom.xml
	project/SparkBuild.scala
	repl/src/main/scala/spark/repl/SparkILoop.scala
	repl/src/test/scala/spark/repl/ReplSuite.scala
	streaming/src/main/scala/spark/streaming/StreamingContext.scala
	streaming/src/main/scala/spark/streaming/api/java/JavaStreamingContext.scala
	streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
	streaming/src/main/scala/spark/streaming/util/MasterFailureTest.scala
 2013-07-03 11:43:26 +05:30
Matei Zaharia 39ae073b5c Increase SLF4j version in Maven too 2013-06-30 17:11:14 -07:00
Matei Zaharia 5cfcd3c336 Remove Twitter4J specific repo since it's in Maven central 2013-06-29 15:37:27 -07:00
Reynold Xin 564d902d79 Merge branch 'master' of github.com:mesos/spark into graph 
Conflicts:
	run
	run2.cmd
 2013-06-29 15:30:21 -07:00
Patrick Wendell 17f145f3bc Updating Maven build 2013-06-22 10:31:36 -07:00
Matei Zaharia 7902baddc7 Update ASM to version 4.0 2013-06-19 13:34:30 +02:00
Prashant Sharma 6f28067f8d Fixed maven build without netty fix 2013-06-14 21:03:21 +05:30
Mark Hamstra 3f96c6f27b Fixed jvmArgs in maven build. 2013-06-12 17:24:22 -07:00
Reynold Xin 8abf4a8447 Merge branch 'master' of github.com:mesos/spark into graph 2013-06-10 12:28:43 -07:00
Konstantin Boudnik d1286231e0 Sometime Maven build runs out of PermGen space. 2013-06-03 15:55:44 -07:00
Reynold Xin b0403d3f2b Merge branch 'master' of github.com:mesos/spark into graph 
Conflicts:
	run
 2013-06-01 00:48:27 -07:00
Jey Kottalam e7982c798e Exclude old versions of Netty from Maven-based build 2013-05-18 21:24:58 -07:00
Reynold Xin 61cf176238 Added dependency on netty-all in Maven. 2013-05-16 14:31:26 -07:00
Reynold Xin 404f9ff617 Added derby dependency to Maven pom files for the JDBC Java test. 2013-05-14 23:28:34 -07:00
Jey Kottalam 4d8919d330 Update Maven build to Scala 2.9.3 2013-05-06 17:05:50 -07:00
Reynold Xin f54bc544c5 Merge branch 'master' of github.com:mesos/spark into graph 2013-05-02 17:25:09 -07:00
Prashant Sharma 4041a2689e Updated to latest stable scala 2.10.1 and akka 2.1.2 2013-05-01 11:35:35 +05:30
Mridul Muralidharan dd515ca3ee Attempt at fixing merge conflict 2013-04-24 09:24:17 +05:30
Mridul Muralidharan 5b85c715c8 Revert back to 2.0.2-alpha : 0.23.7 has protocol changes which break against cloudera 2013-04-24 02:57:51 +05:30
Mridul Muralidharan 8faf5c51c3 Patch from Thomas Graves to improve the YARN Client, and move to more production ready hadoop yarn branch 2013-04-24 02:31:57 +05:30
Prashant Sharma 185bb9525a Manually merged scala-2.10 and master 2013-04-22 14:14:03 +05:30
Mridul Muralidharan 46779b4745 Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet 2013-04-17 05:53:28 +05:30
Mridul Muralidharan 323ab8ff3b Scala does not prevent variable shadowing ! Sick error due to it ... 2013-04-16 17:05:10 +05:30
Mridul Muralidharan 6798a09df8 Add support for building against hadoop2-yarn : adding new maven profile for it 2013-04-07 17:47:38 +05:30
Jey Kottalam bc8ba222ff Bump development version to 0.8.0 2013-03-28 15:42:01 -07:00
Reynold Xin ba9d00c44a Merge branch 'master' into graph 
Conflicts:
	run2.cmd
 2013-03-18 18:30:14 +08:00
Mikhail Bautin 7fd2708eda Add a log4j compile dependency to fix build in IntelliJ 
Also rename parent project to spark-parent (otherwise it shows up as
"parent" in IntelliJ, which is very confusing).
 2013-03-15 11:41:51 -07:00
Matei Zaharia d4e29ea878 Update kryo-serializers version in pom.xml to match previous commit 2013-03-10 15:49:11 -07:00
Matei Zaharia db9b90fdbd Change version to 0.7.1-SNAPSHOT for development branch 2013-02-27 09:15:26 -08:00
Matei Zaharia 7e67c626ee Change version number to 0.7.0 2013-02-25 20:30:47 -08:00
Matei Zaharia 6494cab19d Update Hadoop dependency to 1.0.4 2013-02-25 15:38:21 -08:00
Tathagata Das 5ab37be983 Fixed class paths and dependencies based on Matei's comments. 2013-02-24 16:24:52 -08:00
Reynold Xin 19d3b059e3 Merge branch 'master' into graph 2013-02-19 12:44:05 -08:00
Reynold Xin 81c4d19c61 Maven and sbt build changes for SparkGraph. 2013-02-19 12:43:13 -08:00
Charles Reiss 6107957962 Merge remote-tracking branch 'base/master' into dag-sched-tests 
Conflicts:
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
 2013-02-02 00:33:30 -08:00
Mikhail Bautin fe3eceab57 Remove activation of profiles by default 
See the discussion at https://github.com/mesos/spark/pull/355 for why
default profile activation is a problem.
 2013-01-31 13:30:41 -08:00
Charles Reiss a34096a76d Add easymock to POMs 2013-01-29 10:04:33 -08:00
Mikhail Bautin 325297e5c3 Add an Avro dependency to REPL to make it compile with Hadoop 2 2013-01-22 18:11:51 -08:00
Matei Zaharia 6e3754bf47 Add Maven build file for streaming, and fix some issues in SBT file 
As part of this, changed our Scala 2.9.2 Kafka library to be available
as a local Maven repository, following the example in
(http://blog.dub.podval.org/2010/01/maven-in-project-repository.html)
 2013-01-20 19:22:24 -08:00
folone a5403acd4e Updated maven build for scala 2.10. 2013-01-20 14:42:16 +01:00
Mikhail Bautin 88d8f11365 Add missing dependency spray-json to Maven build 2013-01-13 00:46:25 -08:00
Shivaram Venkataraman bbc56d85ed Rename environment variable for hadoop profiles to hadoopVersion 2013-01-12 15:24:13 -08:00
Shivaram Venkataraman 9262522306 Activate hadoop2 profile in pom.xml with -Dhadoop=2 2013-01-10 22:07:34 -08:00
Shivaram Venkataraman f7adb382ac Activate hadoop1 if property hadoop is missing. hadoop2 can be activated now 
by using -Dhadoop -Phadoop2.
 2013-01-08 03:19:43 -08:00
Shivaram Venkataraman 4bbe07e5ec Activate hadoop1 profile by default for maven builds 2013-01-07 17:46:22 -08:00
Shivaram Venkataraman fb3d4d5e85 Make default hadoop version 1.0.3 in pom.xml 2013-01-07 16:46:06 -08:00
Reynold Xin a6bb41c6d3 Updated Kryo version for Maven pom file. 2012-12-21 16:25:50 -08:00
Thomas Dudziak 4af6cad37a Fixed repl maven build to produce artifacts with the appropriate hadoop classifier and extracted repl fat-jar and debian packaging into a separate project to make Maven happy 2012-12-18 12:08:19 -08:00
Matei Zaharia c10b229992 Merge pull request #319 from mbautin/cdh4.1.2 
Bump CDH version for the Hadoop 2 profile to 4.1.2
 2012-12-10 15:21:01 -08:00
Thomas Dudziak c1d15ae3d5 Shaded repl jar for hadoop1 profile needs to include hadoop classes 2012-12-10 15:06:28 -08:00
Mikhail Bautin 450659079a Bump CDH version for the Hadoop 2 profile to 4.1.2 2012-12-10 11:27:20 -08:00
Matei Zaharia ccff0a089a Use the same output directories that SBT had in subprojects 
This will make it easier to make the "run" script work with a Maven build
 2012-12-10 10:58:56 -08:00
Thomas Dudziak 3b643e86bc Updated versions in the pom.xml files to match current master 2012-11-27 17:50:42 -08:00
Thomas Dudziak 69297c64be Addressed code review comments 2012-11-27 15:45:16 -08:00
Thomas Dudziak 811a32257b Added maven and debian build files 2012-11-20 16:19:51 -08:00