Merge branch 'master' into Pool_UI
Conflicts: core/src/main/scala/spark/SparkContext.scala core/src/main/scala/spark/scheduler/DAGScheduler.scala core/src/main/scala/spark/scheduler/SparkListener.scala core/src/main/scala/spark/scheduler/cluster/ClusterTaskSetManager.scala core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala core/src/main/scala/spark/scheduler/local/LocalTaskSetManager.scala core/src/main/scala/spark/ui/jobs/IndexPage.scala core/src/main/scala/spark/ui/jobs/JobProgressUI.scala
This commit is contained in:
Andrew xia
commit
5670c96f29
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -38,3 +38,5 @@ dependency-reduced-pom.xml
|
||||||
.ensime_lucene
|
.ensime_lucene
|
||||||
checkpoint
|
checkpoint
|
||||||
derby.log
|
derby.log
|
||||||
|
dist/
|
||||||
|
spark-*-bin.tar.gz
|
||||||
|
|
|
||||||
225
LICENSE
225
LICENSE
|
|
@ -1,27 +1,202 @@
|
||||||
Copyright (c) 2010, Regents of the University of California.
|
|
||||||
All rights reserved.
|
|
||||||
|
|
||||||
Redistribution and use in source and binary forms, with or without
|
Apache License
|
||||||
modification, are permitted provided that the following conditions
|
Version 2.0, January 2004
|
||||||
are met:
|
http://www.apache.org/licenses/
|
||||||
* Redistributions of source code must retain the above copyright
|
|
||||||
notice, this list of conditions and the following disclaimer.
|
|
||||||
* Redistributions in binary form must reproduce the above copyright
|
|
||||||
notice, this list of conditions and the following disclaimer in the
|
|
||||||
documentation and/or other materials provided with the distribution.
|
|
||||||
* Neither the name of the University of California, Berkeley nor the
|
|
||||||
names of its contributors may be used to endorse or promote
|
|
||||||
products derived from this software without specific prior written
|
|
||||||
permission.
|
|
||||||
|
|
||||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
||||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
1. Definitions.
|
||||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
||||||
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
"License" shall mean the terms and conditions for use, reproduction,
|
||||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
|
and distribution as defined by Sections 1 through 9 of this document.
|
||||||
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
|
|
||||||
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
|
"Licensor" shall mean the copyright owner or entity authorized by
|
||||||
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
|
the copyright owner that is granting the License.
|
||||||
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
|
|
||||||
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
"Legal Entity" shall mean the union of the acting entity and all
|
||||||
|
other entities that control, are controlled by, or are under common
|
||||||
|
control with that entity. For the purposes of this definition,
|
||||||
|
"control" means (i) the power, direct or indirect, to cause the
|
||||||
|
direction or management of such entity, whether by contract or
|
||||||
|
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||||
|
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||||
|
|
||||||
|
"You" (or "Your") shall mean an individual or Legal Entity
|
||||||
|
exercising permissions granted by this License.
|
||||||
|
|
||||||
|
"Source" form shall mean the preferred form for making modifications,
|
||||||
|
including but not limited to software source code, documentation
|
||||||
|
source, and configuration files.
|
||||||
|
|
||||||
|
"Object" form shall mean any form resulting from mechanical
|
||||||
|
transformation or translation of a Source form, including but
|
||||||
|
not limited to compiled object code, generated documentation,
|
||||||
|
and conversions to other media types.
|
||||||
|
|
||||||
|
"Work" shall mean the work of authorship, whether in Source or
|
||||||
|
Object form, made available under the License, as indicated by a
|
||||||
|
copyright notice that is included in or attached to the work
|
||||||
|
(an example is provided in the Appendix below).
|
||||||
|
|
||||||
|
"Derivative Works" shall mean any work, whether in Source or Object
|
||||||
|
form, that is based on (or derived from) the Work and for which the
|
||||||
|
editorial revisions, annotations, elaborations, or other modifications
|
||||||
|
represent, as a whole, an original work of authorship. For the purposes
|
||||||
|
of this License, Derivative Works shall not include works that remain
|
||||||
|
separable from, or merely link (or bind by name) to the interfaces of,
|
||||||
|
the Work and Derivative Works thereof.
|
||||||
|
|
||||||
|
"Contribution" shall mean any work of authorship, including
|
||||||
|
the original version of the Work and any modifications or additions
|
||||||
|
to that Work or Derivative Works thereof, that is intentionally
|
||||||
|
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||||
|
or by an individual or Legal Entity authorized to submit on behalf of
|
||||||
|
the copyright owner. For the purposes of this definition, "submitted"
|
||||||
|
means any form of electronic, verbal, or written communication sent
|
||||||
|
to the Licensor or its representatives, including but not limited to
|
||||||
|
communication on electronic mailing lists, source code control systems,
|
||||||
|
and issue tracking systems that are managed by, or on behalf of, the
|
||||||
|
Licensor for the purpose of discussing and improving the Work, but
|
||||||
|
excluding communication that is conspicuously marked or otherwise
|
||||||
|
designated in writing by the copyright owner as "Not a Contribution."
|
||||||
|
|
||||||
|
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||||
|
on behalf of whom a Contribution has been received by Licensor and
|
||||||
|
subsequently incorporated within the Work.
|
||||||
|
|
||||||
|
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
copyright license to reproduce, prepare Derivative Works of,
|
||||||
|
publicly display, publicly perform, sublicense, and distribute the
|
||||||
|
Work and such Derivative Works in Source or Object form.
|
||||||
|
|
||||||
|
3. Grant of Patent License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
(except as stated in this section) patent license to make, have made,
|
||||||
|
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||||
|
where such license applies only to those patent claims licensable
|
||||||
|
by such Contributor that are necessarily infringed by their
|
||||||
|
Contribution(s) alone or by combination of their Contribution(s)
|
||||||
|
with the Work to which such Contribution(s) was submitted. If You
|
||||||
|
institute patent litigation against any entity (including a
|
||||||
|
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||||
|
or a Contribution incorporated within the Work constitutes direct
|
||||||
|
or contributory patent infringement, then any patent licenses
|
||||||
|
granted to You under this License for that Work shall terminate
|
||||||
|
as of the date such litigation is filed.
|
||||||
|
|
||||||
|
4. Redistribution. You may reproduce and distribute copies of the
|
||||||
|
Work or Derivative Works thereof in any medium, with or without
|
||||||
|
modifications, and in Source or Object form, provided that You
|
||||||
|
meet the following conditions:
|
||||||
|
|
||||||
|
(a) You must give any other recipients of the Work or
|
||||||
|
Derivative Works a copy of this License; and
|
||||||
|
|
||||||
|
(b) You must cause any modified files to carry prominent notices
|
||||||
|
stating that You changed the files; and
|
||||||
|
|
||||||
|
(c) You must retain, in the Source form of any Derivative Works
|
||||||
|
that You distribute, all copyright, patent, trademark, and
|
||||||
|
attribution notices from the Source form of the Work,
|
||||||
|
excluding those notices that do not pertain to any part of
|
||||||
|
the Derivative Works; and
|
||||||
|
|
||||||
|
(d) If the Work includes a "NOTICE" text file as part of its
|
||||||
|
distribution, then any Derivative Works that You distribute must
|
||||||
|
include a readable copy of the attribution notices contained
|
||||||
|
within such NOTICE file, excluding those notices that do not
|
||||||
|
pertain to any part of the Derivative Works, in at least one
|
||||||
|
of the following places: within a NOTICE text file distributed
|
||||||
|
as part of the Derivative Works; within the Source form or
|
||||||
|
documentation, if provided along with the Derivative Works; or,
|
||||||
|
within a display generated by the Derivative Works, if and
|
||||||
|
wherever such third-party notices normally appear. The contents
|
||||||
|
of the NOTICE file are for informational purposes only and
|
||||||
|
do not modify the License. You may add Your own attribution
|
||||||
|
notices within Derivative Works that You distribute, alongside
|
||||||
|
or as an addendum to the NOTICE text from the Work, provided
|
||||||
|
that such additional attribution notices cannot be construed
|
||||||
|
as modifying the License.
|
||||||
|
|
||||||
|
You may add Your own copyright statement to Your modifications and
|
||||||
|
may provide additional or different license terms and conditions
|
||||||
|
for use, reproduction, or distribution of Your modifications, or
|
||||||
|
for any such Derivative Works as a whole, provided Your use,
|
||||||
|
reproduction, and distribution of the Work otherwise complies with
|
||||||
|
the conditions stated in this License.
|
||||||
|
|
||||||
|
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||||
|
any Contribution intentionally submitted for inclusion in the Work
|
||||||
|
by You to the Licensor shall be under the terms and conditions of
|
||||||
|
this License, without any additional terms or conditions.
|
||||||
|
Notwithstanding the above, nothing herein shall supersede or modify
|
||||||
|
the terms of any separate license agreement you may have executed
|
||||||
|
with Licensor regarding such Contributions.
|
||||||
|
|
||||||
|
6. Trademarks. This License does not grant permission to use the trade
|
||||||
|
names, trademarks, service marks, or product names of the Licensor,
|
||||||
|
except as required for reasonable and customary use in describing the
|
||||||
|
origin of the Work and reproducing the content of the NOTICE file.
|
||||||
|
|
||||||
|
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||||
|
agreed to in writing, Licensor provides the Work (and each
|
||||||
|
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
implied, including, without limitation, any warranties or conditions
|
||||||
|
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||||
|
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||||
|
appropriateness of using or redistributing the Work and assume any
|
||||||
|
risks associated with Your exercise of permissions under this License.
|
||||||
|
|
||||||
|
8. Limitation of Liability. In no event and under no legal theory,
|
||||||
|
whether in tort (including negligence), contract, or otherwise,
|
||||||
|
unless required by applicable law (such as deliberate and grossly
|
||||||
|
negligent acts) or agreed to in writing, shall any Contributor be
|
||||||
|
liable to You for damages, including any direct, indirect, special,
|
||||||
|
incidental, or consequential damages of any character arising as a
|
||||||
|
result of this License or out of the use or inability to use the
|
||||||
|
Work (including but not limited to damages for loss of goodwill,
|
||||||
|
work stoppage, computer failure or malfunction, or any and all
|
||||||
|
other commercial damages or losses), even if such Contributor
|
||||||
|
has been advised of the possibility of such damages.
|
||||||
|
|
||||||
|
9. Accepting Warranty or Additional Liability. While redistributing
|
||||||
|
the Work or Derivative Works thereof, You may choose to offer,
|
||||||
|
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||||
|
or other liability obligations and/or rights consistent with this
|
||||||
|
License. However, in accepting such obligations, You may act only
|
||||||
|
on Your own behalf and on Your sole responsibility, not on behalf
|
||||||
|
of any other Contributor, and only if You agree to indemnify,
|
||||||
|
defend, and hold each Contributor harmless for any liability
|
||||||
|
incurred by, or claims asserted against, such Contributor by reason
|
||||||
|
of your accepting any such warranty or additional liability.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
APPENDIX: How to apply the Apache License to your work.
|
||||||
|
|
||||||
|
To apply the Apache License to your work, attach the following
|
||||||
|
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||||
|
replaced with your own identifying information. (Don't include
|
||||||
|
the brackets!) The text should be enclosed in the appropriate
|
||||||
|
comment syntax for the file format. We also recommend that a
|
||||||
|
file or class name and description of purpose be included on the
|
||||||
|
same "printed page" as the copyright notice for easier
|
||||||
|
identification within third-party archives.
|
||||||
|
|
||||||
|
Copyright [yyyy] [name of copyright owner]
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
|
|
||||||
5
NOTICE
Normal file
5
NOTICE
Normal file
|
|
@ -0,0 +1,5 @@
|
||||||
|
Apache Spark
|
||||||
|
Copyright 2013 The Apache Software Foundation.
|
||||||
|
|
||||||
|
This product includes software developed at
|
||||||
|
The Apache Software Foundation (http://www.apache.org/).
|
||||||
|
|
@ -12,7 +12,7 @@ This README file only contains basic setup instructions.
|
||||||
|
|
||||||
## Building
|
## Building
|
||||||
|
|
||||||
Spark requires Scala 2.9.2 (Scala 2.10 is not yet supported). The project is
|
Spark requires Scala 2.9.3 (Scala 2.10 is not yet supported). The project is
|
||||||
built using Simple Build Tool (SBT), which is packaged with it. To build
|
built using Simple Build Tool (SBT), which is packaged with it. To build
|
||||||
Spark and its example programs, run:
|
Spark and its example programs, run:
|
||||||
|
|
||||||
|
|
|
||||||
13
assembly/README
Normal file
13
assembly/README
Normal file
|
|
@ -0,0 +1,13 @@
|
||||||
|
This is an assembly module for Spark project.
|
||||||
|
|
||||||
|
It creates a single tar.gz file that includes all needed dependency of the project
|
||||||
|
except for org.apache.hadoop.* jars that are supposed to be available from the
|
||||||
|
deployed Hadoop cluster.
|
||||||
|
|
||||||
|
This module is off by default to avoid spending extra time on top of repl-bin
|
||||||
|
module. To activate it specify the profile in the command line
|
||||||
|
-Passembly
|
||||||
|
|
||||||
|
In case you want to avoid building time-expensive repl-bin module, that shaders
|
||||||
|
all the dependency into a big flat jar supplement maven command with
|
||||||
|
-DnoExpensive
|
||||||
104
assembly/pom.xml
Normal file
104
assembly/pom.xml
Normal file
|
|
@ -0,0 +1,104 @@
|
||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
<parent>
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-parent</artifactId>
|
||||||
|
<version>0.8.0-SNAPSHOT</version>
|
||||||
|
<relativePath>../pom.xml</relativePath>
|
||||||
|
</parent>
|
||||||
|
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-assembly</artifactId>
|
||||||
|
<name>Spark Project Assembly</name>
|
||||||
|
<url>http://spark-project.org/</url>
|
||||||
|
|
||||||
|
<build>
|
||||||
|
<plugins>
|
||||||
|
<plugin>
|
||||||
|
<groupId>org.apache.maven.plugins</groupId>
|
||||||
|
<artifactId>maven-assembly-plugin</artifactId>
|
||||||
|
<version>2.4</version>
|
||||||
|
<executions>
|
||||||
|
<execution>
|
||||||
|
<id>dist</id>
|
||||||
|
<phase>package</phase>
|
||||||
|
<goals>
|
||||||
|
<goal>single</goal>
|
||||||
|
</goals>
|
||||||
|
<configuration>
|
||||||
|
<descriptors>
|
||||||
|
<descriptor>src/main/assembly/assembly.xml</descriptor>
|
||||||
|
</descriptors>
|
||||||
|
</configuration>
|
||||||
|
</execution>
|
||||||
|
</executions>
|
||||||
|
</plugin>
|
||||||
|
</plugins>
|
||||||
|
</build>
|
||||||
|
|
||||||
|
<profiles>
|
||||||
|
<profile>
|
||||||
|
<id>hadoop1</id>
|
||||||
|
<properties>
|
||||||
|
<classifier.name>hadoop1</classifier.name>
|
||||||
|
</properties>
|
||||||
|
</profile>
|
||||||
|
<profile>
|
||||||
|
<id>hadoop2</id>
|
||||||
|
<properties>
|
||||||
|
<classifier.name>hadoop2</classifier.name>
|
||||||
|
</properties>
|
||||||
|
</profile>
|
||||||
|
<profile>
|
||||||
|
<id>hadoop2-yarn</id>
|
||||||
|
<properties>
|
||||||
|
<classifier.name>hadoop2-yarn</classifier.name>
|
||||||
|
</properties>
|
||||||
|
</profile>
|
||||||
|
</profiles>
|
||||||
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-core</artifactId>
|
||||||
|
<classifier>${classifier.name}</classifier>
|
||||||
|
<version>0.8.0-SNAPSHOT</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-bagel</artifactId>
|
||||||
|
<classifier>${classifier.name}</classifier>
|
||||||
|
<version>0.8.0-SNAPSHOT</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-examples</artifactId>
|
||||||
|
<classifier>${classifier.name}</classifier>
|
||||||
|
<version>0.8.0-SNAPSHOT</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-examples</artifactId>
|
||||||
|
<classifier>javadoc</classifier>
|
||||||
|
<version>0.8.0-SNAPSHOT</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-examples</artifactId>
|
||||||
|
<classifier>sources</classifier>
|
||||||
|
<version>0.8.0-SNAPSHOT</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-repl</artifactId>
|
||||||
|
<classifier>${classifier.name}</classifier>
|
||||||
|
<version>0.8.0-SNAPSHOT</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.spark-project</groupId>
|
||||||
|
<artifactId>spark-streaming</artifactId>
|
||||||
|
<classifier>${classifier.name}</classifier>
|
||||||
|
<version>0.8.0-SNAPSHOT</version>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
</project>
|
||||||
68
assembly/src/main/assembly/assembly.xml
Normal file
68
assembly/src/main/assembly/assembly.xml
Normal file
|
|
@ -0,0 +1,68 @@
|
||||||
|
<assembly>
|
||||||
|
<id>dist</id>
|
||||||
|
<formats>
|
||||||
|
<format>tar.gz</format>
|
||||||
|
<format>dir</format>
|
||||||
|
</formats>
|
||||||
|
<includeBaseDirectory>false</includeBaseDirectory>
|
||||||
|
|
||||||
|
<fileSets>
|
||||||
|
<fileSet>
|
||||||
|
<includes>
|
||||||
|
<include>README</include>
|
||||||
|
</includes>
|
||||||
|
</fileSet>
|
||||||
|
<fileSet>
|
||||||
|
<directory>
|
||||||
|
${project.parent.basedir}/core/src/main/resources/spark/ui/static/
|
||||||
|
</directory>
|
||||||
|
<outputDirectory>/ui-resources/spark/ui/static</outputDirectory>
|
||||||
|
<includes>
|
||||||
|
<include>**/*</include>
|
||||||
|
</includes>
|
||||||
|
</fileSet>
|
||||||
|
<fileSet>
|
||||||
|
<directory>
|
||||||
|
${project.parent.basedir}/bin/
|
||||||
|
</directory>
|
||||||
|
<outputDirectory>/bin</outputDirectory>
|
||||||
|
<includes>
|
||||||
|
<include>**/*</include>
|
||||||
|
</includes>
|
||||||
|
</fileSet>
|
||||||
|
<fileSet>
|
||||||
|
<directory>
|
||||||
|
${project.parent.basedir}
|
||||||
|
</directory>
|
||||||
|
<outputDirectory>/bin</outputDirectory>
|
||||||
|
<includes>
|
||||||
|
<include>run*</include>
|
||||||
|
<include>spark-shell*</include>
|
||||||
|
<include>spark-executor*</include>
|
||||||
|
</includes>
|
||||||
|
</fileSet>
|
||||||
|
</fileSets>
|
||||||
|
|
||||||
|
<dependencySets>
|
||||||
|
<dependencySet>
|
||||||
|
<includes>
|
||||||
|
<include>org.spark-project:*:jar</include>
|
||||||
|
</includes>
|
||||||
|
<excludes>
|
||||||
|
<exclude>org.spark-project:spark-dist:jar</exclude>
|
||||||
|
</excludes>
|
||||||
|
</dependencySet>
|
||||||
|
<dependencySet>
|
||||||
|
<outputDirectory>lib</outputDirectory>
|
||||||
|
<useTransitiveDependencies>true</useTransitiveDependencies>
|
||||||
|
<unpack>false</unpack>
|
||||||
|
<scope>runtime</scope>
|
||||||
|
<useProjectArtifact>false</useProjectArtifact>
|
||||||
|
<excludes>
|
||||||
|
<exclude>org.apache.hadoop:*:jar</exclude>
|
||||||
|
<exclude>org.spark-project:*:jar</exclude>
|
||||||
|
</excludes>
|
||||||
|
</dependencySet>
|
||||||
|
</dependencySets>
|
||||||
|
|
||||||
|
</assembly>
|
||||||
|
|
@ -1,4 +1,21 @@
|
||||||
<?xml version="1.0" encoding="UTF-8"?>
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!--
|
||||||
|
~ Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
~ contributor license agreements. See the NOTICE file distributed with
|
||||||
|
~ this work for additional information regarding copyright ownership.
|
||||||
|
~ The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
~ (the "License"); you may not use this file except in compliance with
|
||||||
|
~ the License. You may obtain a copy of the License at
|
||||||
|
~
|
||||||
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
~
|
||||||
|
~ Unless required by applicable law or agreed to in writing, software
|
||||||
|
~ distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
~ See the License for the specific language governing permissions and
|
||||||
|
~ limitations under the License.
|
||||||
|
-->
|
||||||
|
|
||||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
<modelVersion>4.0.0</modelVersion>
|
<modelVersion>4.0.0</modelVersion>
|
||||||
<parent>
|
<parent>
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.bagel
|
package spark.bagel
|
||||||
|
|
||||||
import spark._
|
import spark._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.bagel.examples
|
package spark.bagel.examples
|
||||||
|
|
||||||
import spark._
|
import spark._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.bagel.examples
|
package spark.bagel.examples
|
||||||
|
|
||||||
import spark._
|
import spark._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.bagel.examples
|
package spark.bagel.examples
|
||||||
|
|
||||||
import spark._
|
import spark._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Set everything to be logged to the file bagel/target/unit-tests.log
|
# Set everything to be logged to the file bagel/target/unit-tests.log
|
||||||
log4j.rootCategory=INFO, file
|
log4j.rootCategory=INFO, file
|
||||||
log4j.appender.file=org.apache.log4j.FileAppender
|
log4j.appender.file=org.apache.log4j.FileAppender
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.bagel
|
package spark.bagel
|
||||||
|
|
||||||
import org.scalatest.{FunSuite, Assertions, BeforeAndAfter}
|
import org.scalatest.{FunSuite, Assertions, BeforeAndAfter}
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,22 @@
|
||||||
@echo off
|
@echo off
|
||||||
|
|
||||||
|
rem
|
||||||
|
rem Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
rem contributor license agreements. See the NOTICE file distributed with
|
||||||
|
rem this work for additional information regarding copyright ownership.
|
||||||
|
rem The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
rem (the "License"); you may not use this file except in compliance with
|
||||||
|
rem the License. You may obtain a copy of the License at
|
||||||
|
rem
|
||||||
|
rem http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
rem
|
||||||
|
rem Unless required by applicable law or agreed to in writing, software
|
||||||
|
rem distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
rem See the License for the specific language governing permissions and
|
||||||
|
rem limitations under the License.
|
||||||
|
rem
|
||||||
|
|
||||||
rem This script computes Spark's classpath and prints it to stdout; it's used by both the "run"
|
rem This script computes Spark's classpath and prints it to stdout; it's used by both the "run"
|
||||||
rem script and the ExecutorRunner in standalone cluster mode.
|
rem script and the ExecutorRunner in standalone cluster mode.
|
||||||
|
|
||||||
|
|
@ -16,6 +33,7 @@ set REPL_DIR=%FWDIR%repl
|
||||||
set EXAMPLES_DIR=%FWDIR%examples
|
set EXAMPLES_DIR=%FWDIR%examples
|
||||||
set BAGEL_DIR=%FWDIR%bagel
|
set BAGEL_DIR=%FWDIR%bagel
|
||||||
set MLLIB_DIR=%FWDIR%mllib
|
set MLLIB_DIR=%FWDIR%mllib
|
||||||
|
set TOOLS_DIR=%FWDIR%tools
|
||||||
set STREAMING_DIR=%FWDIR%streaming
|
set STREAMING_DIR=%FWDIR%streaming
|
||||||
set PYSPARK_DIR=%FWDIR%python
|
set PYSPARK_DIR=%FWDIR%python
|
||||||
|
|
||||||
|
|
@ -31,6 +49,7 @@ set CLASSPATH=%CLASSPATH%;%FWDIR%repl\lib\*
|
||||||
set CLASSPATH=%CLASSPATH%;%FWDIR%python\lib\*
|
set CLASSPATH=%CLASSPATH%;%FWDIR%python\lib\*
|
||||||
set CLASSPATH=%CLASSPATH%;%BAGEL_DIR%\target\scala-%SCALA_VERSION%\classes
|
set CLASSPATH=%CLASSPATH%;%BAGEL_DIR%\target\scala-%SCALA_VERSION%\classes
|
||||||
set CLASSPATH=%CLASSPATH%;%MLLIB_DIR%\target\scala-%SCALA_VERSION%\classes
|
set CLASSPATH=%CLASSPATH%;%MLLIB_DIR%\target\scala-%SCALA_VERSION%\classes
|
||||||
|
set CLASSPATH=%CLASSPATH%;%TOOLS_DIR%\target\scala-%SCALA_VERSION%\classes
|
||||||
|
|
||||||
rem Add hadoop conf dir - else FileSystem.*, etc fail
|
rem Add hadoop conf dir - else FileSystem.*, etc fail
|
||||||
rem Note, this assumes that there is either a HADOOP_CONF_DIR or YARN_CONF_DIR which hosts
|
rem Note, this assumes that there is either a HADOOP_CONF_DIR or YARN_CONF_DIR which hosts
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,21 @@
|
||||||
#!/bin/bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# This script computes Spark's classpath and prints it to stdout; it's used by both the "run"
|
# This script computes Spark's classpath and prints it to stdout; it's used by both the "run"
|
||||||
# script and the ExecutorRunner in standalone cluster mode.
|
# script and the ExecutorRunner in standalone cluster mode.
|
||||||
|
|
@ -19,51 +36,67 @@ REPL_BIN_DIR="$FWDIR/repl-bin"
|
||||||
EXAMPLES_DIR="$FWDIR/examples"
|
EXAMPLES_DIR="$FWDIR/examples"
|
||||||
BAGEL_DIR="$FWDIR/bagel"
|
BAGEL_DIR="$FWDIR/bagel"
|
||||||
MLLIB_DIR="$FWDIR/mllib"
|
MLLIB_DIR="$FWDIR/mllib"
|
||||||
|
TOOLS_DIR="$FWDIR/tools"
|
||||||
STREAMING_DIR="$FWDIR/streaming"
|
STREAMING_DIR="$FWDIR/streaming"
|
||||||
PYSPARK_DIR="$FWDIR/python"
|
PYSPARK_DIR="$FWDIR/python"
|
||||||
|
|
||||||
# Build up classpath
|
# Build up classpath
|
||||||
CLASSPATH="$SPARK_CLASSPATH"
|
CLASSPATH="$SPARK_CLASSPATH"
|
||||||
CLASSPATH="$CLASSPATH:$FWDIR/conf"
|
|
||||||
CLASSPATH="$CLASSPATH:$CORE_DIR/target/scala-$SCALA_VERSION/classes"
|
function dev_classpath {
|
||||||
if [ -n "$SPARK_TESTING" ] ; then
|
CLASSPATH="$CLASSPATH:$FWDIR/conf"
|
||||||
|
CLASSPATH="$CLASSPATH:$CORE_DIR/target/scala-$SCALA_VERSION/classes"
|
||||||
|
if [ -n "$SPARK_TESTING" ] ; then
|
||||||
CLASSPATH="$CLASSPATH:$CORE_DIR/target/scala-$SCALA_VERSION/test-classes"
|
CLASSPATH="$CLASSPATH:$CORE_DIR/target/scala-$SCALA_VERSION/test-classes"
|
||||||
CLASSPATH="$CLASSPATH:$STREAMING_DIR/target/scala-$SCALA_VERSION/test-classes"
|
CLASSPATH="$CLASSPATH:$STREAMING_DIR/target/scala-$SCALA_VERSION/test-classes"
|
||||||
fi
|
fi
|
||||||
CLASSPATH="$CLASSPATH:$CORE_DIR/src/main/resources"
|
CLASSPATH="$CLASSPATH:$CORE_DIR/src/main/resources"
|
||||||
CLASSPATH="$CLASSPATH:$REPL_DIR/target/scala-$SCALA_VERSION/classes"
|
CLASSPATH="$CLASSPATH:$REPL_DIR/target/scala-$SCALA_VERSION/classes"
|
||||||
CLASSPATH="$CLASSPATH:$EXAMPLES_DIR/target/scala-$SCALA_VERSION/classes"
|
CLASSPATH="$CLASSPATH:$EXAMPLES_DIR/target/scala-$SCALA_VERSION/classes"
|
||||||
CLASSPATH="$CLASSPATH:$STREAMING_DIR/target/scala-$SCALA_VERSION/classes"
|
CLASSPATH="$CLASSPATH:$STREAMING_DIR/target/scala-$SCALA_VERSION/classes"
|
||||||
CLASSPATH="$CLASSPATH:$STREAMING_DIR/lib/org/apache/kafka/kafka/0.7.2-spark/*" # <-- our in-project Kafka Jar
|
CLASSPATH="$CLASSPATH:$STREAMING_DIR/lib/org/apache/kafka/kafka/0.7.2-spark/*" # <-- our in-project Kafka Jar
|
||||||
if [ -e "$FWDIR/lib_managed" ]; then
|
if [ -e "$FWDIR/lib_managed" ]; then
|
||||||
CLASSPATH="$CLASSPATH:$FWDIR/lib_managed/jars/*"
|
CLASSPATH="$CLASSPATH:$FWDIR/lib_managed/jars/*"
|
||||||
CLASSPATH="$CLASSPATH:$FWDIR/lib_managed/bundles/*"
|
CLASSPATH="$CLASSPATH:$FWDIR/lib_managed/bundles/*"
|
||||||
fi
|
fi
|
||||||
CLASSPATH="$CLASSPATH:$REPL_DIR/lib/*"
|
CLASSPATH="$CLASSPATH:$REPL_DIR/lib/*"
|
||||||
# Add the shaded JAR for Maven builds
|
# Add the shaded JAR for Maven builds
|
||||||
if [ -e $REPL_BIN_DIR/target ]; then
|
if [ -e $REPL_BIN_DIR/target ]; then
|
||||||
for jar in `find "$REPL_BIN_DIR/target" -name 'spark-repl-*-shaded-hadoop*.jar'`; do
|
for jar in `find "$REPL_BIN_DIR/target" -name 'spark-repl-*-shaded-hadoop*.jar'`; do
|
||||||
CLASSPATH="$CLASSPATH:$jar"
|
CLASSPATH="$CLASSPATH:$jar"
|
||||||
done
|
done
|
||||||
# The shaded JAR doesn't contain examples, so include those separately
|
# The shaded JAR doesn't contain examples, so include those separately
|
||||||
EXAMPLES_JAR=`ls "$EXAMPLES_DIR/target/spark-examples"*[0-9T].jar`
|
EXAMPLES_JAR=`ls "$EXAMPLES_DIR/target/spark-examples"*[0-9T].jar`
|
||||||
CLASSPATH+=":$EXAMPLES_JAR"
|
CLASSPATH+=":$EXAMPLES_JAR"
|
||||||
fi
|
fi
|
||||||
CLASSPATH="$CLASSPATH:$BAGEL_DIR/target/scala-$SCALA_VERSION/classes"
|
CLASSPATH="$CLASSPATH:$BAGEL_DIR/target/scala-$SCALA_VERSION/classes"
|
||||||
CLASSPATH="$CLASSPATH:$MLLIB_DIR/target/scala-$SCALA_VERSION/classes"
|
CLASSPATH="$CLASSPATH:$MLLIB_DIR/target/scala-$SCALA_VERSION/classes"
|
||||||
for jar in `find $PYSPARK_DIR/lib -name '*jar'`; do
|
CLASSPATH="$CLASSPATH:$TOOLS_DIR/target/scala-$SCALA_VERSION/classes"
|
||||||
|
for jar in `find $PYSPARK_DIR/lib -name '*jar'`; do
|
||||||
CLASSPATH="$CLASSPATH:$jar"
|
CLASSPATH="$CLASSPATH:$jar"
|
||||||
done
|
done
|
||||||
|
|
||||||
# Figure out the JAR file that our examples were packaged into. This includes a bit of a hack
|
# Add Scala standard library
|
||||||
# to avoid the -sources and -doc packages that are built by publish-local.
|
if [ -z "$SCALA_LIBRARY_PATH" ]; then
|
||||||
if [ -e "$EXAMPLES_DIR/target/scala-$SCALA_VERSION/spark-examples"*[0-9T].jar ]; then
|
if [ -z "$SCALA_HOME" ]; then
|
||||||
# Use the JAR from the SBT build
|
echo "SCALA_HOME is not set" >&2
|
||||||
export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR/target/scala-$SCALA_VERSION/spark-examples"*[0-9T].jar`
|
exit 1
|
||||||
fi
|
fi
|
||||||
if [ -e "$EXAMPLES_DIR/target/spark-examples"*[0-9T].jar ]; then
|
SCALA_LIBRARY_PATH="$SCALA_HOME/lib"
|
||||||
# Use the JAR from the Maven build
|
fi
|
||||||
export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR/target/spark-examples"*[0-9T].jar`
|
CLASSPATH="$CLASSPATH:$SCALA_LIBRARY_PATH/scala-library.jar"
|
||||||
|
CLASSPATH="$CLASSPATH:$SCALA_LIBRARY_PATH/scala-compiler.jar"
|
||||||
|
CLASSPATH="$CLASSPATH:$SCALA_LIBRARY_PATH/jline.jar"
|
||||||
|
}
|
||||||
|
|
||||||
|
function release_classpath {
|
||||||
|
CLASSPATH="$CLASSPATH:$FWDIR/jars/*"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ -f "$FWDIR/RELEASE" ]; then
|
||||||
|
release_classpath
|
||||||
|
else
|
||||||
|
dev_classpath
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Add hadoop conf dir - else FileSystem.*, etc fail !
|
# Add hadoop conf dir - else FileSystem.*, etc fail !
|
||||||
|
|
@ -76,16 +109,4 @@ if [ "x" != "x$YARN_CONF_DIR" ]; then
|
||||||
CLASSPATH="$CLASSPATH:$YARN_CONF_DIR"
|
CLASSPATH="$CLASSPATH:$YARN_CONF_DIR"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Add Scala standard library
|
|
||||||
if [ -z "$SCALA_LIBRARY_PATH" ]; then
|
|
||||||
if [ -z "$SCALA_HOME" ]; then
|
|
||||||
echo "SCALA_HOME is not set" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
SCALA_LIBRARY_PATH="$SCALA_HOME/lib"
|
|
||||||
fi
|
|
||||||
CLASSPATH="$CLASSPATH:$SCALA_LIBRARY_PATH/scala-library.jar"
|
|
||||||
CLASSPATH="$CLASSPATH:$SCALA_LIBRARY_PATH/scala-compiler.jar"
|
|
||||||
CLASSPATH="$CLASSPATH:$SCALA_LIBRARY_PATH/jline.jar"
|
|
||||||
|
|
||||||
echo "$CLASSPATH"
|
echo "$CLASSPATH"
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,5 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
# This Spark deploy script is a modified version of the Apache Hadoop deploy
|
|
||||||
# script, available under the Apache 2 license:
|
|
||||||
#
|
#
|
||||||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
# contributor license agreements. See the NOTICE file distributed with
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
|
@ -17,6 +15,7 @@
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Run a shell command on all slave hosts.
|
# Run a shell command on all slave hosts.
|
||||||
#
|
#
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# included in all the spark scripts with source command
|
# included in all the spark scripts with source command
|
||||||
# should not be executable directly
|
# should not be executable directly
|
||||||
# also should not be passed any arguments, since we need original $*
|
# also should not be passed any arguments, since we need original $*
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,5 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
# This Spark deploy script is a modified version of the Apache Hadoop deploy
|
|
||||||
# script, available under the Apache 2 license:
|
|
||||||
#
|
#
|
||||||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
# contributor license agreements. See the NOTICE file distributed with
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
|
@ -17,6 +15,7 @@
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Runs a Spark command as a daemon.
|
# Runs a Spark command as a daemon.
|
||||||
#
|
#
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,22 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Run a Spark command on all slave hosts.
|
# Run a Spark command on all slave hosts.
|
||||||
|
|
||||||
usage="Usage: spark-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] command instance-number args..."
|
usage="Usage: spark-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] command instance-number args..."
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,22 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Start all spark daemons.
|
# Start all spark daemons.
|
||||||
# Starts the master on this node.
|
# Starts the master on this node.
|
||||||
# Starts a worker on each node specified in conf/slaves
|
# Starts a worker on each node specified in conf/slaves
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,22 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Starts the master on the machine this script is executed on.
|
# Starts the master on the machine this script is executed on.
|
||||||
|
|
||||||
bin=`dirname "$0"`
|
bin=`dirname "$0"`
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,25 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
|
# Usage: start-slave.sh <worker#> <master-spark-URL>
|
||||||
|
# where <master-spark-URL> is like "spark://localhost:7077"
|
||||||
|
|
||||||
bin=`dirname "$0"`
|
bin=`dirname "$0"`
|
||||||
bin=`cd "$bin"; pwd`
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,22 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
bin=`dirname "$0"`
|
bin=`dirname "$0"`
|
||||||
bin=`cd "$bin"; pwd`
|
bin=`cd "$bin"; pwd`
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,22 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Start all spark daemons.
|
# Start all spark daemons.
|
||||||
# Run this on the master nde
|
# Run this on the master nde
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,22 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Starts the master on the machine this script is executed on.
|
# Starts the master on the machine this script is executed on.
|
||||||
|
|
||||||
bin=`dirname "$0"`
|
bin=`dirname "$0"`
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,22 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
#
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
# contributor license agreements. See the NOTICE file distributed with
|
||||||
|
# this work for additional information regarding copyright ownership.
|
||||||
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
# (the "License"); you may not use this file except in compliance with
|
||||||
|
# the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
#
|
||||||
|
|
||||||
# Starts the master on the machine this script is executed on.
|
# Starts the master on the machine this script is executed on.
|
||||||
|
|
||||||
bin=`dirname "$0"`
|
bin=`dirname "$0"`
|
||||||
|
|
|
||||||
87
conf/metrics.properties.template
Normal file
87
conf/metrics.properties.template
Normal file
|
|
@ -0,0 +1,87 @@
|
||||||
|
# syntax: [instance].[sink|source].[name].[options]
|
||||||
|
|
||||||
|
# "instance" specify "who" (the role) use metrics system. In spark there are
|
||||||
|
# several roles like master, worker, executor, driver, these roles will
|
||||||
|
# create metrics system for monitoring. So instance represents these roles.
|
||||||
|
# Currently in Spark, several instances have already implemented: master,
|
||||||
|
# worker, executor, driver.
|
||||||
|
#
|
||||||
|
# [instance] field can be "master", "worker", "executor", "driver", which means
|
||||||
|
# only the specified instance has this property.
|
||||||
|
# a wild card "*" can be used to represent instance name, which means all the
|
||||||
|
# instances will have this property.
|
||||||
|
#
|
||||||
|
# "source" specify "where" (source) to collect metrics data. In metrics system,
|
||||||
|
# there exists two kinds of source:
|
||||||
|
# 1. Spark internal source, like MasterSource, WorkerSource, etc, which will
|
||||||
|
# collect Spark component's internal state, these sources are related to
|
||||||
|
# instance and will be added after specific metrics system is created.
|
||||||
|
# 2. Common source, like JvmSource, which will collect low level state, is
|
||||||
|
# configured by configuration and loaded through reflection.
|
||||||
|
#
|
||||||
|
# "sink" specify "where" (destination) to output metrics data to. Several sinks
|
||||||
|
# can be coexisted and flush metrics to all these sinks.
|
||||||
|
#
|
||||||
|
# [sink|source] field specify this property is source related or sink, this
|
||||||
|
# field can only be source or sink.
|
||||||
|
#
|
||||||
|
# [name] field specify the name of source or sink, this is custom defined.
|
||||||
|
#
|
||||||
|
# [options] field is the specific property of this source or sink, this source
|
||||||
|
# or sink is responsible for parsing this property.
|
||||||
|
#
|
||||||
|
# Notes:
|
||||||
|
# 1. Sinks should be added through configuration, like console sink, class
|
||||||
|
# full name should be specified by class property.
|
||||||
|
# 2. Some sinks can specify polling period, like console sink, which is 10 seconds,
|
||||||
|
# it should be attention minimal polling period is 1 seconds, any period
|
||||||
|
# below than 1s is illegal.
|
||||||
|
# 3. Wild card property can be overlapped by specific instance property, for
|
||||||
|
# example, *.sink.console.period can be overlapped by master.sink.console.period.
|
||||||
|
# 4. A metrics specific configuration
|
||||||
|
# "spark.metrics.conf=${SPARK_HOME}/conf/metrics.properties" should be
|
||||||
|
# added to Java property using -Dspark.metrics.conf=xxx if you want to
|
||||||
|
# customize metrics system, or you can put it in ${SPARK_HOME}/conf,
|
||||||
|
# metrics system will search and load it automatically.
|
||||||
|
|
||||||
|
# Enable JmxSink for all instances by class name
|
||||||
|
#*.sink.jmx.class=spark.metrics.sink.JmxSink
|
||||||
|
|
||||||
|
# Enable ConsoleSink for all instances by class name
|
||||||
|
#*.sink.console.class=spark.metrics.sink.ConsoleSink
|
||||||
|
|
||||||
|
# Polling period for ConsoleSink
|
||||||
|
#*.sink.console.period=10
|
||||||
|
|
||||||
|
#*.sink.console.unit=seconds
|
||||||
|
|
||||||
|
# Master instance overlap polling period
|
||||||
|
#master.sink.console.period=15
|
||||||
|
|
||||||
|
#master.sink.console.unit=seconds
|
||||||
|
|
||||||
|
# Enable CsvSink for all instances
|
||||||
|
#*.sink.csv.class=spark.metrics.sink.CsvSink
|
||||||
|
|
||||||
|
# Polling period for CsvSink
|
||||||
|
#*.sink.csv.period=1
|
||||||
|
|
||||||
|
#*.sink.csv.unit=minutes
|
||||||
|
|
||||||
|
# Polling directory for CsvSink
|
||||||
|
#*.sink.csv.directory=/tmp/
|
||||||
|
|
||||||
|
# Worker instance overlap polling period
|
||||||
|
#worker.sink.csv.period=10
|
||||||
|
|
||||||
|
#worker.sink.csv.unit=minutes
|
||||||
|
|
||||||
|
# Enable jvm source for instance master, worker, driver and executor
|
||||||
|
#master.source.jvm.class=spark.metrics.source.JvmSource
|
||||||
|
|
||||||
|
#worker.source.jvm.class=spark.metrics.source.JvmSource
|
||||||
|
|
||||||
|
#driver.source.jvm.class=spark.metrics.source.JvmSource
|
||||||
|
|
||||||
|
#executor.source.jvm.class=spark.metrics.source.JvmSource
|
||||||
|
|
||||||
43
core/pom.xml
43
core/pom.xml
|
|
@ -1,4 +1,21 @@
|
||||||
<?xml version="1.0" encoding="UTF-8"?>
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!--
|
||||||
|
~ Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
~ contributor license agreements. See the NOTICE file distributed with
|
||||||
|
~ this work for additional information regarding copyright ownership.
|
||||||
|
~ The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
~ (the "License"); you may not use this file except in compliance with
|
||||||
|
~ the License. You may obtain a copy of the License at
|
||||||
|
~
|
||||||
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
~
|
||||||
|
~ Unless required by applicable law or agreed to in writing, software
|
||||||
|
~ distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
~ See the License for the specific language governing permissions and
|
||||||
|
~ limitations under the License.
|
||||||
|
-->
|
||||||
|
|
||||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
<modelVersion>4.0.0</modelVersion>
|
<modelVersion>4.0.0</modelVersion>
|
||||||
<parent>
|
<parent>
|
||||||
|
|
@ -40,8 +57,14 @@
|
||||||
<artifactId>protobuf-java</artifactId>
|
<artifactId>protobuf-java</artifactId>
|
||||||
</dependency>
|
</dependency>
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>de.javakaffee</groupId>
|
<groupId>com.twitter</groupId>
|
||||||
<artifactId>kryo-serializers</artifactId>
|
<artifactId>chill_2.9.3</artifactId>
|
||||||
|
<version>0.3.0</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.twitter</groupId>
|
||||||
|
<artifactId>chill-java</artifactId>
|
||||||
|
<version>0.3.0</version>
|
||||||
</dependency>
|
</dependency>
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>com.typesafe.akka</groupId>
|
<groupId>com.typesafe.akka</groupId>
|
||||||
|
|
@ -55,6 +78,14 @@
|
||||||
<groupId>com.typesafe.akka</groupId>
|
<groupId>com.typesafe.akka</groupId>
|
||||||
<artifactId>akka-slf4j</artifactId>
|
<artifactId>akka-slf4j</artifactId>
|
||||||
</dependency>
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.scala-lang</groupId>
|
||||||
|
<artifactId>scalap</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.scala-lang</groupId>
|
||||||
|
<artifactId>scala-library</artifactId>
|
||||||
|
</dependency>
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>net.liftweb</groupId>
|
<groupId>net.liftweb</groupId>
|
||||||
<artifactId>lift-json_2.9.2</artifactId>
|
<artifactId>lift-json_2.9.2</artifactId>
|
||||||
|
|
@ -83,6 +114,14 @@
|
||||||
<groupId>log4j</groupId>
|
<groupId>log4j</groupId>
|
||||||
<artifactId>log4j</artifactId>
|
<artifactId>log4j</artifactId>
|
||||||
</dependency>
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.codahale.metrics</groupId>
|
||||||
|
<artifactId>metrics-core</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.codahale.metrics</groupId>
|
||||||
|
<artifactId>metrics-jvm</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.apache.derby</groupId>
|
<groupId>org.apache.derby</groupId>
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package org.apache.hadoop.mapred
|
package org.apache.hadoop.mapred
|
||||||
|
|
||||||
trait HadoopMapRedUtil {
|
trait HadoopMapRedUtil {
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package org.apache.hadoop.mapreduce
|
package org.apache.hadoop.mapreduce
|
||||||
|
|
||||||
import org.apache.hadoop.conf.Configuration
|
import org.apache.hadoop.conf.Configuration
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy
|
package spark.deploy
|
||||||
import org.apache.hadoop.conf.Configuration
|
import org.apache.hadoop.conf.Configuration
|
||||||
import org.apache.hadoop.mapred.JobConf
|
import org.apache.hadoop.mapred.JobConf
|
||||||
|
|
@ -24,4 +41,7 @@ object SparkHadoopUtil {
|
||||||
|
|
||||||
// add any user credentials to the job conf which are necessary for running on a secure Hadoop cluster
|
// add any user credentials to the job conf which are necessary for running on a secure Hadoop cluster
|
||||||
def addCredentials(conf: JobConf) {}
|
def addCredentials(conf: JobConf) {}
|
||||||
|
|
||||||
|
def isYarnMode(): Boolean = { false }
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,21 @@
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package org.apache.hadoop.mapred
|
package org.apache.hadoop.mapred
|
||||||
|
|
||||||
import org.apache.hadoop.mapreduce.TaskType
|
import org.apache.hadoop.mapreduce.TaskType
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package org.apache.hadoop.mapreduce
|
package org.apache.hadoop.mapreduce
|
||||||
|
|
||||||
import org.apache.hadoop.conf.Configuration
|
import org.apache.hadoop.conf.Configuration
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy
|
package spark.deploy
|
||||||
|
|
||||||
import collection.mutable.HashMap
|
import collection.mutable.HashMap
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy.yarn
|
package spark.deploy.yarn
|
||||||
|
|
||||||
import java.net.Socket
|
import java.net.Socket
|
||||||
|
|
@ -27,6 +44,7 @@ class ApplicationMaster(args: ApplicationMasterArguments, conf: Configuration) e
|
||||||
private val yarnConf: YarnConfiguration = new YarnConfiguration(conf)
|
private val yarnConf: YarnConfiguration = new YarnConfiguration(conf)
|
||||||
|
|
||||||
private var yarnAllocator: YarnAllocationHandler = null
|
private var yarnAllocator: YarnAllocationHandler = null
|
||||||
|
private var isFinished:Boolean = false
|
||||||
|
|
||||||
def run() {
|
def run() {
|
||||||
|
|
||||||
|
|
@ -69,9 +87,6 @@ class ApplicationMaster(args: ApplicationMasterArguments, conf: Configuration) e
|
||||||
// Wait for the user class to Finish
|
// Wait for the user class to Finish
|
||||||
userThread.join()
|
userThread.join()
|
||||||
|
|
||||||
// Finish the ApplicationMaster
|
|
||||||
finishApplicationMaster()
|
|
||||||
// TODO: Exit based on success/failure
|
|
||||||
System.exit(0)
|
System.exit(0)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -131,10 +146,23 @@ class ApplicationMaster(args: ApplicationMasterArguments, conf: Configuration) e
|
||||||
.getMethod("main", classOf[Array[String]])
|
.getMethod("main", classOf[Array[String]])
|
||||||
val t = new Thread {
|
val t = new Thread {
|
||||||
override def run() {
|
override def run() {
|
||||||
|
var successed = false
|
||||||
|
try {
|
||||||
// Copy
|
// Copy
|
||||||
var mainArgs: Array[String] = new Array[String](args.userArgs.size())
|
var mainArgs: Array[String] = new Array[String](args.userArgs.size())
|
||||||
args.userArgs.copyToArray(mainArgs, 0, args.userArgs.size())
|
args.userArgs.copyToArray(mainArgs, 0, args.userArgs.size())
|
||||||
mainMethod.invoke(null, mainArgs)
|
mainMethod.invoke(null, mainArgs)
|
||||||
|
// some job script has "System.exit(0)" at the end, for example SparkPi, SparkLR
|
||||||
|
// userThread will stop here unless it has uncaught exception thrown out
|
||||||
|
// It need shutdown hook to set SUCCEEDED
|
||||||
|
successed = true
|
||||||
|
} finally {
|
||||||
|
if (successed) {
|
||||||
|
ApplicationMaster.this.finishApplicationMaster(FinalApplicationStatus.SUCCEEDED)
|
||||||
|
} else {
|
||||||
|
ApplicationMaster.this.finishApplicationMaster(FinalApplicationStatus.FAILED)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
t.start()
|
t.start()
|
||||||
|
|
@ -179,7 +207,7 @@ class ApplicationMaster(args: ApplicationMasterArguments, conf: Configuration) e
|
||||||
logInfo("All workers have launched.")
|
logInfo("All workers have launched.")
|
||||||
|
|
||||||
// Launch a progress reporter thread, else app will get killed after expiration (def: 10mins) timeout
|
// Launch a progress reporter thread, else app will get killed after expiration (def: 10mins) timeout
|
||||||
if (userThread.isAlive){
|
if (userThread.isAlive) {
|
||||||
// ensure that progress is sent before YarnConfiguration.RM_AM_EXPIRY_INTERVAL_MS elapse.
|
// ensure that progress is sent before YarnConfiguration.RM_AM_EXPIRY_INTERVAL_MS elapse.
|
||||||
|
|
||||||
val timeoutInterval = yarnConf.getInt(YarnConfiguration.RM_AM_EXPIRY_INTERVAL_MS, 120000)
|
val timeoutInterval = yarnConf.getInt(YarnConfiguration.RM_AM_EXPIRY_INTERVAL_MS, 120000)
|
||||||
|
|
@ -197,7 +225,7 @@ class ApplicationMaster(args: ApplicationMasterArguments, conf: Configuration) e
|
||||||
|
|
||||||
val t = new Thread {
|
val t = new Thread {
|
||||||
override def run() {
|
override def run() {
|
||||||
while (userThread.isAlive){
|
while (userThread.isAlive) {
|
||||||
val missingWorkerCount = args.numWorkers - yarnAllocator.getNumWorkersRunning
|
val missingWorkerCount = args.numWorkers - yarnAllocator.getNumWorkersRunning
|
||||||
if (missingWorkerCount > 0) {
|
if (missingWorkerCount > 0) {
|
||||||
logInfo("Allocating " + missingWorkerCount + " containers to make up for (potentially ?) lost containers")
|
logInfo("Allocating " + missingWorkerCount + " containers to make up for (potentially ?) lost containers")
|
||||||
|
|
@ -236,13 +264,22 @@ class ApplicationMaster(args: ApplicationMasterArguments, conf: Configuration) e
|
||||||
}
|
}
|
||||||
*/
|
*/
|
||||||
|
|
||||||
def finishApplicationMaster() {
|
def finishApplicationMaster(status: FinalApplicationStatus) {
|
||||||
|
|
||||||
|
synchronized {
|
||||||
|
if (isFinished) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
isFinished = true
|
||||||
|
}
|
||||||
|
|
||||||
|
logInfo("finishApplicationMaster with " + status)
|
||||||
val finishReq = Records.newRecord(classOf[FinishApplicationMasterRequest])
|
val finishReq = Records.newRecord(classOf[FinishApplicationMasterRequest])
|
||||||
.asInstanceOf[FinishApplicationMasterRequest]
|
.asInstanceOf[FinishApplicationMasterRequest]
|
||||||
finishReq.setAppAttemptId(appAttemptId)
|
finishReq.setAppAttemptId(appAttemptId)
|
||||||
// TODO: Check if the application has failed or succeeded
|
finishReq.setFinishApplicationStatus(status)
|
||||||
finishReq.setFinishApplicationStatus(FinalApplicationStatus.SUCCEEDED)
|
|
||||||
resourceManager.finishApplicationMaster(finishReq)
|
resourceManager.finishApplicationMaster(finishReq)
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
@ -256,7 +293,7 @@ object ApplicationMaster {
|
||||||
private val ALLOCATOR_LOOP_WAIT_COUNT = 30
|
private val ALLOCATOR_LOOP_WAIT_COUNT = 30
|
||||||
def incrementAllocatorLoop(by: Int) {
|
def incrementAllocatorLoop(by: Int) {
|
||||||
val count = yarnAllocatorLoop.getAndAdd(by)
|
val count = yarnAllocatorLoop.getAndAdd(by)
|
||||||
if (count >= ALLOCATOR_LOOP_WAIT_COUNT){
|
if (count >= ALLOCATOR_LOOP_WAIT_COUNT) {
|
||||||
yarnAllocatorLoop.synchronized {
|
yarnAllocatorLoop.synchronized {
|
||||||
// to wake threads off wait ...
|
// to wake threads off wait ...
|
||||||
yarnAllocatorLoop.notifyAll()
|
yarnAllocatorLoop.notifyAll()
|
||||||
|
|
@ -291,14 +328,16 @@ object ApplicationMaster {
|
||||||
logInfo("Invoking sc stop from shutdown hook")
|
logInfo("Invoking sc stop from shutdown hook")
|
||||||
sc.stop()
|
sc.stop()
|
||||||
// best case ...
|
// best case ...
|
||||||
for (master <- applicationMasters) master.finishApplicationMaster
|
for (master <- applicationMasters) {
|
||||||
|
master.finishApplicationMaster(FinalApplicationStatus.SUCCEEDED)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
} )
|
} )
|
||||||
}
|
}
|
||||||
|
|
||||||
// Wait for initialization to complete and atleast 'some' nodes can get allocated
|
// Wait for initialization to complete and atleast 'some' nodes can get allocated
|
||||||
yarnAllocatorLoop.synchronized {
|
yarnAllocatorLoop.synchronized {
|
||||||
while (yarnAllocatorLoop.get() <= ALLOCATOR_LOOP_WAIT_COUNT){
|
while (yarnAllocatorLoop.get() <= ALLOCATOR_LOOP_WAIT_COUNT) {
|
||||||
yarnAllocatorLoop.wait(1000L)
|
yarnAllocatorLoop.wait(1000L)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy.yarn
|
package spark.deploy.yarn
|
||||||
|
|
||||||
import spark.util.IntParam
|
import spark.util.IntParam
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy.yarn
|
package spark.deploy.yarn
|
||||||
|
|
||||||
import java.net.{InetSocketAddress, URI}
|
import java.net.{InetSocketAddress, URI}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy.yarn
|
package spark.deploy.yarn
|
||||||
|
|
||||||
import spark.util.MemoryParam
|
import spark.util.MemoryParam
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy.yarn
|
package spark.deploy.yarn
|
||||||
|
|
||||||
import java.net.URI
|
import java.net.URI
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy.yarn
|
package spark.deploy.yarn
|
||||||
|
|
||||||
import spark.{Logging, Utils}
|
import spark.{Logging, Utils}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.scheduler.cluster
|
package spark.scheduler.cluster
|
||||||
|
|
||||||
import spark._
|
import spark._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package org.apache.hadoop.mapred
|
package org.apache.hadoop.mapred
|
||||||
|
|
||||||
trait HadoopMapRedUtil {
|
trait HadoopMapRedUtil {
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package org.apache.hadoop.mapreduce
|
package org.apache.hadoop.mapreduce
|
||||||
|
|
||||||
import org.apache.hadoop.conf.Configuration
|
import org.apache.hadoop.conf.Configuration
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.deploy
|
package spark.deploy
|
||||||
import org.apache.hadoop.conf.Configuration
|
import org.apache.hadoop.conf.Configuration
|
||||||
import org.apache.hadoop.mapred.JobConf
|
import org.apache.hadoop.mapred.JobConf
|
||||||
|
|
@ -24,4 +41,7 @@ object SparkHadoopUtil {
|
||||||
|
|
||||||
// add any user credentials to the job conf which are necessary for running on a secure Hadoop cluster
|
// add any user credentials to the job conf which are necessary for running on a secure Hadoop cluster
|
||||||
def addCredentials(conf: JobConf) {}
|
def addCredentials(conf: JobConf) {}
|
||||||
|
|
||||||
|
def isYarnMode(): Boolean = { false }
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.network.netty;
|
package spark.network.netty;
|
||||||
|
|
||||||
import io.netty.bootstrap.Bootstrap;
|
import io.netty.bootstrap.Bootstrap;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.network.netty;
|
package spark.network.netty;
|
||||||
|
|
||||||
import io.netty.buffer.BufType;
|
import io.netty.buffer.BufType;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.network.netty;
|
package spark.network.netty;
|
||||||
|
|
||||||
import io.netty.buffer.ByteBuf;
|
import io.netty.buffer.ByteBuf;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.network.netty;
|
package spark.network.netty;
|
||||||
|
|
||||||
import java.net.InetSocketAddress;
|
import java.net.InetSocketAddress;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.network.netty;
|
package spark.network.netty;
|
||||||
|
|
||||||
import io.netty.channel.ChannelInitializer;
|
import io.netty.channel.ChannelInitializer;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.network.netty;
|
package spark.network.netty;
|
||||||
|
|
||||||
import java.io.File;
|
import java.io.File;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.network.netty;
|
package spark.network.netty;
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io._
|
import java.io._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.util.{HashMap => JHashMap}
|
import java.util.{HashMap => JHashMap}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import scala.collection.mutable.ArrayBuffer
|
import scala.collection.mutable.ArrayBuffer
|
||||||
|
|
|
||||||
|
|
@ -1,63 +0,0 @@
|
||||||
package spark
|
|
||||||
|
|
||||||
import java.util.concurrent.atomic.AtomicInteger
|
|
||||||
|
|
||||||
private[spark] sealed trait CachePutResponse
|
|
||||||
private[spark] case class CachePutSuccess(size: Long) extends CachePutResponse
|
|
||||||
private[spark] case class CachePutFailure() extends CachePutResponse
|
|
||||||
|
|
||||||
/**
|
|
||||||
* An interface for caches in Spark, to allow for multiple implementations. Caches are used to store
|
|
||||||
* both partitions of cached RDDs and broadcast variables on Spark executors. Caches are also aware
|
|
||||||
* of which entries are part of the same dataset (for example, partitions in the same RDD). The key
|
|
||||||
* for each value in a cache is a (datasetID, partition) pair.
|
|
||||||
*
|
|
||||||
* A single Cache instance gets created on each machine and is shared by all caches (i.e. both the
|
|
||||||
* RDD split cache and the broadcast variable cache), to enable global replacement policies.
|
|
||||||
* However, because these several independent modules all perform caching, it is important to give
|
|
||||||
* them separate key namespaces, so that an RDD and a broadcast variable (for example) do not use
|
|
||||||
* the same key. For this purpose, Cache has the notion of KeySpaces. Each client module must first
|
|
||||||
* ask for a KeySpace, and then call get() and put() on that space using its own keys.
|
|
||||||
*
|
|
||||||
* This abstract class handles the creation of key spaces, so that subclasses need only deal with
|
|
||||||
* keys that are unique across modules.
|
|
||||||
*/
|
|
||||||
private[spark] abstract class Cache {
|
|
||||||
private val nextKeySpaceId = new AtomicInteger(0)
|
|
||||||
private def newKeySpaceId() = nextKeySpaceId.getAndIncrement()
|
|
||||||
|
|
||||||
def newKeySpace() = new KeySpace(this, newKeySpaceId())
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get the value for a given (datasetId, partition), or null if it is not
|
|
||||||
* found.
|
|
||||||
*/
|
|
||||||
def get(datasetId: Any, partition: Int): Any
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Attempt to put a value in the cache; returns CachePutFailure if this was
|
|
||||||
* not successful (e.g. because the cache replacement policy forbids it), and
|
|
||||||
* CachePutSuccess if successful. If size estimation is available, the cache
|
|
||||||
* implementation should set the size field in CachePutSuccess.
|
|
||||||
*/
|
|
||||||
def put(datasetId: Any, partition: Int, value: Any): CachePutResponse
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Report the capacity of the cache partition. By default this just reports
|
|
||||||
* zero. Specific implementations can choose to provide the capacity number.
|
|
||||||
*/
|
|
||||||
def getCapacity: Long = 0L
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* A key namespace in a Cache.
|
|
||||||
*/
|
|
||||||
private[spark] class KeySpace(cache: Cache, val keySpaceId: Int) {
|
|
||||||
def get(datasetId: Any, partition: Int): Any =
|
|
||||||
cache.get((keySpaceId, datasetId), partition)
|
|
||||||
|
|
||||||
def put(datasetId: Any, partition: Int, value: Any): CachePutResponse =
|
|
||||||
cache.put((keySpaceId, datasetId), partition, value)
|
|
||||||
|
|
||||||
def getCapacity: Long = cache.getCapacity
|
|
||||||
}
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import scala.collection.mutable.{ArrayBuffer, HashSet}
|
import scala.collection.mutable.{ArrayBuffer, HashSet}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.lang.reflect.Field
|
import java.lang.reflect.Field
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import spark.partial.BoundedDouble
|
import spark.partial.BoundedDouble
|
||||||
|
|
@ -37,7 +54,13 @@ class DoubleRDDFunctions(self: RDD[Double]) extends Logging with Serializable {
|
||||||
* Compute the sample standard deviation of this RDD's elements (which corrects for bias in
|
* Compute the sample standard deviation of this RDD's elements (which corrects for bias in
|
||||||
* estimating the standard deviation by dividing by N-1 instead of N).
|
* estimating the standard deviation by dividing by N-1 instead of N).
|
||||||
*/
|
*/
|
||||||
def sampleStdev(): Double = stats().stdev
|
def sampleStdev(): Double = stats().sampleStdev
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Compute the sample variance of this RDD's elements (which corrects for bias in
|
||||||
|
* estimating the variance by dividing by N-1 instead of N).
|
||||||
|
*/
|
||||||
|
def sampleVariance(): Double = stats().sampleVariance
|
||||||
|
|
||||||
/** (Experimental) Approximate operation to return the mean within a timeout. */
|
/** (Experimental) Approximate operation to return the mean within a timeout. */
|
||||||
def meanApprox(timeout: Long, confidence: Double = 0.95): PartialResult[BoundedDouble] = {
|
def meanApprox(timeout: Long, confidence: Double = 0.95): PartialResult[BoundedDouble] = {
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import spark.storage.BlockManagerId
|
import spark.storage.BlockManagerId
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package org.apache.hadoop.mapred
|
package org.apache.hadoop.mapred
|
||||||
|
|
||||||
import org.apache.hadoop.fs.FileSystem
|
import org.apache.hadoop.fs.FileSystem
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io.{File}
|
import java.io.{File}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io.File
|
import java.io.File
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io._
|
import java.io._
|
||||||
|
|
|
||||||
|
|
@ -1,25 +1,34 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io._
|
import java.io._
|
||||||
import java.nio.ByteBuffer
|
import java.nio.ByteBuffer
|
||||||
import java.nio.channels.Channels
|
import com.esotericsoftware.kryo.{Kryo, KryoException}
|
||||||
|
|
||||||
import scala.collection.immutable
|
|
||||||
import scala.collection.mutable
|
|
||||||
|
|
||||||
import com.esotericsoftware.kryo._
|
|
||||||
import com.esotericsoftware.kryo.{Serializer => KSerializer}
|
|
||||||
import com.esotericsoftware.kryo.io.{Input => KryoInput, Output => KryoOutput}
|
import com.esotericsoftware.kryo.io.{Input => KryoInput, Output => KryoOutput}
|
||||||
import com.esotericsoftware.kryo.serializers.{JavaSerializer => KryoJavaSerializer}
|
import com.esotericsoftware.kryo.serializers.{JavaSerializer => KryoJavaSerializer}
|
||||||
import de.javakaffee.kryoserializers.KryoReflectionFactorySupport
|
import com.twitter.chill.ScalaKryoInstantiator
|
||||||
|
|
||||||
import serializer.{SerializerInstance, DeserializationStream, SerializationStream}
|
import serializer.{SerializerInstance, DeserializationStream, SerializationStream}
|
||||||
import spark.broadcast._
|
import spark.broadcast._
|
||||||
import spark.storage._
|
import spark.storage._
|
||||||
|
|
||||||
private[spark]
|
private[spark]
|
||||||
class KryoSerializationStream(kryo: Kryo, outStream: OutputStream) extends SerializationStream {
|
class KryoSerializationStream(kryo: Kryo, outStream: OutputStream) extends SerializationStream {
|
||||||
|
|
||||||
val output = new KryoOutput(outStream)
|
val output = new KryoOutput(outStream)
|
||||||
|
|
||||||
def writeObject[T](t: T): SerializationStream = {
|
def writeObject[T](t: T): SerializationStream = {
|
||||||
|
|
@ -33,7 +42,6 @@ class KryoSerializationStream(kryo: Kryo, outStream: OutputStream) extends Seria
|
||||||
|
|
||||||
private[spark]
|
private[spark]
|
||||||
class KryoDeserializationStream(kryo: Kryo, inStream: InputStream) extends DeserializationStream {
|
class KryoDeserializationStream(kryo: Kryo, inStream: InputStream) extends DeserializationStream {
|
||||||
|
|
||||||
val input = new KryoInput(inStream)
|
val input = new KryoInput(inStream)
|
||||||
|
|
||||||
def readObject[T](): T = {
|
def readObject[T](): T = {
|
||||||
|
|
@ -41,7 +49,7 @@ class KryoDeserializationStream(kryo: Kryo, inStream: InputStream) extends Deser
|
||||||
kryo.readClassAndObject(input).asInstanceOf[T]
|
kryo.readClassAndObject(input).asInstanceOf[T]
|
||||||
} catch {
|
} catch {
|
||||||
// DeserializationStream uses the EOF exception to indicate stopping condition.
|
// DeserializationStream uses the EOF exception to indicate stopping condition.
|
||||||
case e: com.esotericsoftware.kryo.KryoException => throw new java.io.EOFException
|
case _: KryoException => throw new EOFException
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -52,10 +60,9 @@ class KryoDeserializationStream(kryo: Kryo, inStream: InputStream) extends Deser
|
||||||
}
|
}
|
||||||
|
|
||||||
private[spark] class KryoSerializerInstance(ks: KryoSerializer) extends SerializerInstance {
|
private[spark] class KryoSerializerInstance(ks: KryoSerializer) extends SerializerInstance {
|
||||||
|
val kryo = ks.newKryo()
|
||||||
val kryo = ks.kryo.get()
|
val output = ks.newKryoOutput()
|
||||||
val output = ks.output.get()
|
val input = ks.newKryoInput()
|
||||||
val input = ks.input.get()
|
|
||||||
|
|
||||||
def serialize[T](t: T): ByteBuffer = {
|
def serialize[T](t: T): ByteBuffer = {
|
||||||
output.clear()
|
output.clear()
|
||||||
|
|
@ -91,130 +98,59 @@ private[spark] class KryoSerializerInstance(ks: KryoSerializer) extends Serializ
|
||||||
* serialization.
|
* serialization.
|
||||||
*/
|
*/
|
||||||
trait KryoRegistrator {
|
trait KryoRegistrator {
|
||||||
def registerClasses(kryo: Kryo): Unit
|
def registerClasses(kryo: Kryo)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A Spark serializer that uses the [[http://code.google.com/p/kryo/wiki/V1Documentation Kryo 1.x library]].
|
* A Spark serializer that uses the [[http://code.google.com/p/kryo/wiki/V1Documentation Kryo 1.x library]].
|
||||||
*/
|
*/
|
||||||
class KryoSerializer extends spark.serializer.Serializer with Logging {
|
class KryoSerializer extends spark.serializer.Serializer with Logging {
|
||||||
|
private val bufferSize = System.getProperty("spark.kryoserializer.buffer.mb", "2").toInt * 1024 * 1024
|
||||||
|
|
||||||
val bufferSize = System.getProperty("spark.kryoserializer.buffer.mb", "2").toInt * 1024 * 1024
|
def newKryoOutput() = new KryoOutput(bufferSize)
|
||||||
|
|
||||||
val kryo = new ThreadLocal[Kryo] {
|
def newKryoInput() = new KryoInput(bufferSize)
|
||||||
override def initialValue = createKryo()
|
|
||||||
}
|
|
||||||
|
|
||||||
val output = new ThreadLocal[KryoOutput] {
|
def newKryo(): Kryo = {
|
||||||
override def initialValue = new KryoOutput(bufferSize)
|
val instantiator = new ScalaKryoInstantiator
|
||||||
}
|
val kryo = instantiator.newKryo()
|
||||||
|
val classLoader = Thread.currentThread.getContextClassLoader
|
||||||
val input = new ThreadLocal[KryoInput] {
|
|
||||||
override def initialValue = new KryoInput(bufferSize)
|
|
||||||
}
|
|
||||||
|
|
||||||
def createKryo(): Kryo = {
|
|
||||||
val kryo = new KryoReflectionFactorySupport()
|
|
||||||
|
|
||||||
// Register some commonly used classes
|
// Register some commonly used classes
|
||||||
val toRegister: Seq[AnyRef] = Seq(
|
val toRegister: Seq[AnyRef] = Seq(
|
||||||
// Arrays
|
|
||||||
Array(1), Array(1.0), Array(1.0f), Array(1L), Array(""), Array(("", "")),
|
|
||||||
Array(new java.lang.Object), Array(1.toByte), Array(true), Array('c'),
|
|
||||||
// Specialized Tuple2s
|
|
||||||
("", ""), ("", 1), (1, 1), (1.0, 1.0), (1L, 1L),
|
|
||||||
(1, 1.0), (1.0, 1), (1L, 1.0), (1.0, 1L), (1, 1L), (1L, 1),
|
|
||||||
// Scala collections
|
|
||||||
List(1), mutable.ArrayBuffer(1),
|
|
||||||
// Options and Either
|
|
||||||
Some(1), Left(1), Right(1),
|
|
||||||
// Higher-dimensional tuples
|
|
||||||
(1, 1, 1), (1, 1, 1, 1), (1, 1, 1, 1, 1),
|
|
||||||
None,
|
|
||||||
ByteBuffer.allocate(1),
|
ByteBuffer.allocate(1),
|
||||||
StorageLevel.MEMORY_ONLY,
|
StorageLevel.MEMORY_ONLY,
|
||||||
PutBlock("1", ByteBuffer.allocate(1), StorageLevel.MEMORY_ONLY),
|
PutBlock("1", ByteBuffer.allocate(1), StorageLevel.MEMORY_ONLY),
|
||||||
GotBlock("1", ByteBuffer.allocate(1)),
|
GotBlock("1", ByteBuffer.allocate(1)),
|
||||||
GetBlock("1")
|
GetBlock("1")
|
||||||
)
|
)
|
||||||
for (obj <- toRegister) {
|
|
||||||
kryo.register(obj.getClass)
|
for (obj <- toRegister) kryo.register(obj.getClass)
|
||||||
}
|
|
||||||
|
|
||||||
// Allow sending SerializableWritable
|
// Allow sending SerializableWritable
|
||||||
kryo.register(classOf[SerializableWritable[_]], new KryoJavaSerializer())
|
kryo.register(classOf[SerializableWritable[_]], new KryoJavaSerializer())
|
||||||
kryo.register(classOf[HttpBroadcast[_]], new KryoJavaSerializer())
|
kryo.register(classOf[HttpBroadcast[_]], new KryoJavaSerializer())
|
||||||
|
|
||||||
// Register some commonly used Scala singleton objects. Because these
|
|
||||||
// are singletons, we must return the exact same local object when we
|
|
||||||
// deserialize rather than returning a clone as FieldSerializer would.
|
|
||||||
class SingletonSerializer[T](obj: T) extends KSerializer[T] {
|
|
||||||
override def write(kryo: Kryo, output: KryoOutput, obj: T) {}
|
|
||||||
override def read(kryo: Kryo, input: KryoInput, cls: java.lang.Class[T]): T = obj
|
|
||||||
}
|
|
||||||
kryo.register(None.getClass, new SingletonSerializer[AnyRef](None))
|
|
||||||
kryo.register(Nil.getClass, new SingletonSerializer[AnyRef](Nil))
|
|
||||||
|
|
||||||
// Register maps with a special serializer since they have complex internal structure
|
|
||||||
class ScalaMapSerializer(buildMap: Array[(Any, Any)] => scala.collection.Map[Any, Any])
|
|
||||||
extends KSerializer[Array[(Any, Any)] => scala.collection.Map[Any, Any]] {
|
|
||||||
|
|
||||||
//hack, look at https://groups.google.com/forum/#!msg/kryo-users/Eu5V4bxCfws/k-8UQ22y59AJ
|
|
||||||
private final val FAKE_REFERENCE = new Object()
|
|
||||||
override def write(
|
|
||||||
kryo: Kryo,
|
|
||||||
output: KryoOutput,
|
|
||||||
obj: Array[(Any, Any)] => scala.collection.Map[Any, Any]) {
|
|
||||||
val map = obj.asInstanceOf[scala.collection.Map[Any, Any]]
|
|
||||||
output.writeInt(map.size)
|
|
||||||
for ((k, v) <- map) {
|
|
||||||
kryo.writeClassAndObject(output, k)
|
|
||||||
kryo.writeClassAndObject(output, v)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
override def read (
|
|
||||||
kryo: Kryo,
|
|
||||||
input: KryoInput,
|
|
||||||
cls: Class[Array[(Any, Any)] => scala.collection.Map[Any, Any]])
|
|
||||||
: Array[(Any, Any)] => scala.collection.Map[Any, Any] = {
|
|
||||||
kryo.reference(FAKE_REFERENCE)
|
|
||||||
val size = input.readInt()
|
|
||||||
val elems = new Array[(Any, Any)](size)
|
|
||||||
for (i <- 0 until size) {
|
|
||||||
val k = kryo.readClassAndObject(input)
|
|
||||||
val v = kryo.readClassAndObject(input)
|
|
||||||
elems(i)=(k,v)
|
|
||||||
}
|
|
||||||
buildMap(elems).asInstanceOf[Array[(Any, Any)] => scala.collection.Map[Any, Any]]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
kryo.register(mutable.HashMap().getClass, new ScalaMapSerializer(mutable.HashMap() ++ _))
|
|
||||||
// TODO: add support for immutable maps too; this is more annoying because there are many
|
|
||||||
// subclasses of immutable.Map for small maps (with <= 4 entries)
|
|
||||||
val map1 = Map[Any, Any](1 -> 1)
|
|
||||||
val map2 = Map[Any, Any](1 -> 1, 2 -> 2)
|
|
||||||
val map3 = Map[Any, Any](1 -> 1, 2 -> 2, 3 -> 3)
|
|
||||||
val map4 = Map[Any, Any](1 -> 1, 2 -> 2, 3 -> 3, 4 -> 4)
|
|
||||||
val map5 = Map[Any, Any](1 -> 1, 2 -> 2, 3 -> 3, 4 -> 4, 5 -> 5)
|
|
||||||
kryo.register(map1.getClass, new ScalaMapSerializer(mutable.HashMap() ++ _ toMap))
|
|
||||||
kryo.register(map2.getClass, new ScalaMapSerializer(mutable.HashMap() ++ _ toMap))
|
|
||||||
kryo.register(map3.getClass, new ScalaMapSerializer(mutable.HashMap() ++ _ toMap))
|
|
||||||
kryo.register(map4.getClass, new ScalaMapSerializer(mutable.HashMap() ++ _ toMap))
|
|
||||||
kryo.register(map5.getClass, new ScalaMapSerializer(mutable.HashMap() ++ _ toMap))
|
|
||||||
|
|
||||||
// Allow the user to register their own classes by setting spark.kryo.registrator
|
// Allow the user to register their own classes by setting spark.kryo.registrator
|
||||||
val regCls = System.getProperty("spark.kryo.registrator")
|
try {
|
||||||
if (regCls != null) {
|
Option(System.getProperty("spark.kryo.registrator")).foreach { regCls =>
|
||||||
logInfo("Running user registrator: " + regCls)
|
logDebug("Running user registrator: " + regCls)
|
||||||
val classLoader = Thread.currentThread.getContextClassLoader
|
|
||||||
val reg = Class.forName(regCls, true, classLoader).newInstance().asInstanceOf[KryoRegistrator]
|
val reg = Class.forName(regCls, true, classLoader).newInstance().asInstanceOf[KryoRegistrator]
|
||||||
reg.registerClasses(kryo)
|
reg.registerClasses(kryo)
|
||||||
}
|
}
|
||||||
|
} catch {
|
||||||
|
case _: Exception => println("Failed to register spark.kryo.registrator")
|
||||||
|
}
|
||||||
|
|
||||||
|
kryo.setClassLoader(classLoader)
|
||||||
|
|
||||||
|
// Allow disabling Kryo reference tracking if user knows their object graphs don't have loops
|
||||||
|
kryo.setReferences(System.getProperty("spark.kryo.referenceTracking", "true").toBoolean)
|
||||||
|
|
||||||
kryo
|
kryo
|
||||||
}
|
}
|
||||||
|
|
||||||
def newInstance(): SerializerInstance = {
|
def newInstance(): SerializerInstance = {
|
||||||
this.kryo.get().setClassLoader(Thread.currentThread().getContextClassLoader)
|
|
||||||
new KryoSerializerInstance(this)
|
new KryoSerializerInstance(this)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import org.slf4j.Logger
|
import org.slf4j.Logger
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io._
|
import java.io._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.nio.ByteBuffer
|
import java.nio.ByteBuffer
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.util.Random
|
import java.util.Random
|
||||||
|
|
@ -778,9 +795,19 @@ abstract class RDD[T: ClassManifest](
|
||||||
}.reduce { (queue1, queue2) =>
|
}.reduce { (queue1, queue2) =>
|
||||||
queue1 ++= queue2
|
queue1 ++= queue2
|
||||||
queue1
|
queue1
|
||||||
}.toArray
|
}.toArray.sorted(ord.reverse)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Returns the first K elements from this RDD as defined by
|
||||||
|
* the specified implicit Ordering[T] and maintains the
|
||||||
|
* ordering.
|
||||||
|
* @param num the number of top elements to return
|
||||||
|
* @param ord the implicit ordering for T
|
||||||
|
* @return an array of top elements
|
||||||
|
*/
|
||||||
|
def takeOrdered(num: Int)(implicit ord: Ordering[T]): Array[T] = top(num)(ord.reverse)
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Save this RDD as a text file, using string representations of elements.
|
* Save this RDD as a text file, using string representations of elements.
|
||||||
*/
|
*/
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import org.apache.hadoop.fs.Path
|
import org.apache.hadoop.fs.Path
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io.EOFException
|
import java.io.EOFException
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io._
|
import java.io._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import spark.executor.TaskMetrics
|
import spark.executor.TaskMetrics
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.lang.reflect.Field
|
import java.lang.reflect.Field
|
||||||
|
|
|
||||||
|
|
@ -1,18 +0,0 @@
|
||||||
package spark
|
|
||||||
|
|
||||||
import com.google.common.collect.MapMaker
|
|
||||||
|
|
||||||
/**
|
|
||||||
* An implementation of Cache that uses soft references.
|
|
||||||
*/
|
|
||||||
private[spark] class SoftReferenceCache extends Cache {
|
|
||||||
val map = new MapMaker().softValues().makeMap[Any, Any]()
|
|
||||||
|
|
||||||
override def get(datasetId: Any, partition: Int): Any =
|
|
||||||
map.get((datasetId, partition))
|
|
||||||
|
|
||||||
override def put(datasetId: Any, partition: Int, value: Any): CachePutResponse = {
|
|
||||||
map.put((datasetId, partition), value)
|
|
||||||
return CachePutSuccess(0)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io._
|
import java.io._
|
||||||
|
|
@ -44,15 +61,16 @@ import org.apache.mesos.MesosNativeLibrary
|
||||||
import spark.deploy.{LocalSparkCluster, SparkHadoopUtil}
|
import spark.deploy.{LocalSparkCluster, SparkHadoopUtil}
|
||||||
import spark.partial.{ApproximateEvaluator, PartialResult}
|
import spark.partial.{ApproximateEvaluator, PartialResult}
|
||||||
import spark.rdd.{CheckpointRDD, HadoopRDD, NewHadoopRDD, UnionRDD, ParallelCollectionRDD}
|
import spark.rdd.{CheckpointRDD, HadoopRDD, NewHadoopRDD, UnionRDD, ParallelCollectionRDD}
|
||||||
import spark.scheduler.{DAGScheduler, ResultTask, ShuffleMapTask, SparkListener,
|
import spark.scheduler.{DAGScheduler, DAGSchedulerSource, ResultTask, ShuffleMapTask, SparkListener,
|
||||||
SplitInfo, Stage, StageInfo, TaskScheduler, ActiveJob}
|
SplitInfo, Stage, StageInfo, TaskScheduler, ActiveJob}
|
||||||
import spark.scheduler.cluster.{StandaloneSchedulerBackend, SparkDeploySchedulerBackend,
|
import spark.scheduler.cluster.{StandaloneSchedulerBackend, SparkDeploySchedulerBackend,
|
||||||
ClusterScheduler, Schedulable, SchedulingMode}
|
ClusterScheduler, Schedulable, SchedulingMode}
|
||||||
import spark.scheduler.local.LocalScheduler
|
import spark.scheduler.local.LocalScheduler
|
||||||
import spark.scheduler.mesos.{CoarseMesosSchedulerBackend, MesosSchedulerBackend}
|
import spark.scheduler.mesos.{CoarseMesosSchedulerBackend, MesosSchedulerBackend}
|
||||||
import spark.storage.{StorageStatus, StorageUtils, RDDInfo}
|
import spark.storage.{StorageStatus, StorageUtils, RDDInfo, BlockManagerSource}
|
||||||
import spark.util.{MetadataCleaner, TimeStampedHashMap}
|
import spark.util.{MetadataCleaner, TimeStampedHashMap}
|
||||||
import ui.{SparkUI}
|
import ui.{SparkUI}
|
||||||
|
import spark.metrics._
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Main entry point for Spark functionality. A SparkContext represents the connection to a Spark
|
* Main entry point for Spark functionality. A SparkContext represents the connection to a Spark
|
||||||
|
|
@ -256,6 +274,16 @@ class SparkContext(
|
||||||
// Post init
|
// Post init
|
||||||
taskScheduler.postStartHook()
|
taskScheduler.postStartHook()
|
||||||
|
|
||||||
|
val dagSchedulerSource = new DAGSchedulerSource(this.dagScheduler)
|
||||||
|
val blockManagerSource = new BlockManagerSource(SparkEnv.get.blockManager)
|
||||||
|
|
||||||
|
def initDriverMetrics() {
|
||||||
|
SparkEnv.get.metricsSystem.registerSource(dagSchedulerSource)
|
||||||
|
SparkEnv.get.metricsSystem.registerSource(blockManagerSource)
|
||||||
|
}
|
||||||
|
|
||||||
|
initDriverMetrics()
|
||||||
|
|
||||||
// Methods for creating RDDs
|
// Methods for creating RDDs
|
||||||
|
|
||||||
/** Distribute a local Scala collection to form an RDD. */
|
/** Distribute a local Scala collection to form an RDD. */
|
||||||
|
|
@ -532,6 +560,12 @@ class SparkContext(
|
||||||
StorageUtils.rddInfoFromStorageStatus(getExecutorStorageStatus, this)
|
StorageUtils.rddInfoFromStorageStatus(getExecutorStorageStatus, this)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.
|
||||||
|
* Note that this does not necessarily mean the caching or computation was successful.
|
||||||
|
*/
|
||||||
|
def getPersistentRDDs: Map[Int, RDD[_]] = persistentRdds.toMap
|
||||||
|
|
||||||
def getStageInfo: Map[Stage,StageInfo] = {
|
def getStageInfo: Map[Stage,StageInfo] = {
|
||||||
dagScheduler.stageToInfos
|
dagScheduler.stageToInfos
|
||||||
}
|
}
|
||||||
|
|
@ -545,7 +579,7 @@ class SparkContext(
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Return pools for fair scheduler
|
* Return pools for fair scheduler
|
||||||
* TODO:now, we have not taken nested pools into account
|
* TODO(xiajunluan):now, we have not taken nested pools into account
|
||||||
*/
|
*/
|
||||||
def getPools: ArrayBuffer[Schedulable] = {
|
def getPools: ArrayBuffer[Schedulable] = {
|
||||||
taskScheduler.rootPool.schedulableQueue
|
taskScheduler.rootPool.schedulableQueue
|
||||||
|
|
@ -582,7 +616,12 @@ class SparkContext(
|
||||||
} else {
|
} else {
|
||||||
val uri = new URI(path)
|
val uri = new URI(path)
|
||||||
val key = uri.getScheme match {
|
val key = uri.getScheme match {
|
||||||
case null | "file" => env.httpFileServer.addJar(new File(uri.getPath))
|
case null | "file" =>
|
||||||
|
if (SparkHadoopUtil.isYarnMode()) {
|
||||||
|
logWarning("local jar specified as parameter to addJar under Yarn mode")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
env.httpFileServer.addJar(new File(uri.getPath))
|
||||||
case _ => path
|
case _ => path
|
||||||
}
|
}
|
||||||
addedJars(key) = System.currentTimeMillis
|
addedJars(key) = System.currentTimeMillis
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import collection.mutable
|
import collection.mutable
|
||||||
|
|
@ -7,6 +24,7 @@ import akka.actor.{Actor, ActorRef, Props, ActorSystemImpl, ActorSystem}
|
||||||
import akka.remote.RemoteActorRefProvider
|
import akka.remote.RemoteActorRefProvider
|
||||||
|
|
||||||
import spark.broadcast.BroadcastManager
|
import spark.broadcast.BroadcastManager
|
||||||
|
import spark.metrics.MetricsSystem
|
||||||
import spark.storage.BlockManager
|
import spark.storage.BlockManager
|
||||||
import spark.storage.BlockManagerMaster
|
import spark.storage.BlockManagerMaster
|
||||||
import spark.network.ConnectionManager
|
import spark.network.ConnectionManager
|
||||||
|
|
@ -36,6 +54,7 @@ class SparkEnv (
|
||||||
val connectionManager: ConnectionManager,
|
val connectionManager: ConnectionManager,
|
||||||
val httpFileServer: HttpFileServer,
|
val httpFileServer: HttpFileServer,
|
||||||
val sparkFilesDir: String,
|
val sparkFilesDir: String,
|
||||||
|
val metricsSystem: MetricsSystem,
|
||||||
// To be set only as part of initialization of SparkContext.
|
// To be set only as part of initialization of SparkContext.
|
||||||
// (executorId, defaultHostPort) => executorHostPort
|
// (executorId, defaultHostPort) => executorHostPort
|
||||||
// If executorId is NOT found, return defaultHostPort
|
// If executorId is NOT found, return defaultHostPort
|
||||||
|
|
@ -51,6 +70,7 @@ class SparkEnv (
|
||||||
broadcastManager.stop()
|
broadcastManager.stop()
|
||||||
blockManager.stop()
|
blockManager.stop()
|
||||||
blockManager.master.stop()
|
blockManager.master.stop()
|
||||||
|
metricsSystem.stop()
|
||||||
actorSystem.shutdown()
|
actorSystem.shutdown()
|
||||||
// Unfortunately Akka's awaitTermination doesn't actually wait for the Netty server to shut
|
// Unfortunately Akka's awaitTermination doesn't actually wait for the Netty server to shut
|
||||||
// down, but let's call it anyway in case it gets fixed in a later release
|
// down, but let's call it anyway in case it gets fixed in a later release
|
||||||
|
|
@ -167,6 +187,13 @@ object SparkEnv extends Logging {
|
||||||
httpFileServer.initialize()
|
httpFileServer.initialize()
|
||||||
System.setProperty("spark.fileserver.uri", httpFileServer.serverUri)
|
System.setProperty("spark.fileserver.uri", httpFileServer.serverUri)
|
||||||
|
|
||||||
|
val metricsSystem = if (isDriver) {
|
||||||
|
MetricsSystem.createMetricsSystem("driver")
|
||||||
|
} else {
|
||||||
|
MetricsSystem.createMetricsSystem("executor")
|
||||||
|
}
|
||||||
|
metricsSystem.start()
|
||||||
|
|
||||||
// Set the sparkFiles directory, used when downloading dependencies. In local mode,
|
// Set the sparkFiles directory, used when downloading dependencies. In local mode,
|
||||||
// this is a temporary directory; in distributed mode, this is the executor's current working
|
// this is a temporary directory; in distributed mode, this is the executor's current working
|
||||||
// directory.
|
// directory.
|
||||||
|
|
@ -196,6 +223,7 @@ object SparkEnv extends Logging {
|
||||||
connectionManager,
|
connectionManager,
|
||||||
httpFileServer,
|
httpFileServer,
|
||||||
sparkFilesDir,
|
sparkFilesDir,
|
||||||
|
metricsSystem,
|
||||||
None)
|
None)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
class SparkException(message: String, cause: Throwable)
|
class SparkException(message: String, cause: Throwable)
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark;
|
package spark;
|
||||||
|
|
||||||
import java.io.File;
|
import java.io.File;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import executor.TaskMetrics
|
import executor.TaskMetrics
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import spark.executor.TaskMetrics
|
import spark.executor.TaskMetrics
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import org.apache.mesos.Protos.{TaskState => MesosTaskState}
|
import org.apache.mesos.Protos.{TaskState => MesosTaskState}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark
|
package spark
|
||||||
|
|
||||||
import java.io._
|
import java.io._
|
||||||
|
|
@ -6,6 +23,7 @@ import java.util.{Locale, Random, UUID}
|
||||||
import java.util.concurrent.{ConcurrentHashMap, Executors, ThreadFactory, ThreadPoolExecutor}
|
import java.util.concurrent.{ConcurrentHashMap, Executors, ThreadFactory, ThreadPoolExecutor}
|
||||||
import java.util.regex.Pattern
|
import java.util.regex.Pattern
|
||||||
|
|
||||||
|
import scala.collection.Map
|
||||||
import scala.collection.mutable.{ArrayBuffer, HashMap}
|
import scala.collection.mutable.{ArrayBuffer, HashMap}
|
||||||
import scala.collection.JavaConversions._
|
import scala.collection.JavaConversions._
|
||||||
import scala.io.Source
|
import scala.io.Source
|
||||||
|
|
@ -545,10 +563,15 @@ private object Utils extends Logging {
|
||||||
/**
|
/**
|
||||||
* Execute a command and get its output, throwing an exception if it yields a code other than 0.
|
* Execute a command and get its output, throwing an exception if it yields a code other than 0.
|
||||||
*/
|
*/
|
||||||
def executeAndGetOutput(command: Seq[String], workingDir: File = new File(".")): String = {
|
def executeAndGetOutput(command: Seq[String], workingDir: File = new File("."),
|
||||||
val process = new ProcessBuilder(command: _*)
|
extraEnvironment: Map[String, String] = Map.empty): String = {
|
||||||
|
val builder = new ProcessBuilder(command: _*)
|
||||||
.directory(workingDir)
|
.directory(workingDir)
|
||||||
.start()
|
val environment = builder.environment()
|
||||||
|
for ((key, value) <- extraEnvironment) {
|
||||||
|
environment.put(key, value)
|
||||||
|
}
|
||||||
|
val process = builder.start()
|
||||||
new Thread("read stderr for " + command(0)) {
|
new Thread("read stderr for " + command(0)) {
|
||||||
override def run() {
|
override def run() {
|
||||||
for (line <- Source.fromInputStream(process.getErrorStream).getLines) {
|
for (line <- Source.fromInputStream(process.getErrorStream).getLines) {
|
||||||
|
|
@ -573,8 +596,15 @@ private object Utils extends Logging {
|
||||||
output.toString
|
output.toString
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A regular expression to match classes of the "core" Spark API that we want to skip when
|
||||||
|
* finding the call site of a method.
|
||||||
|
*/
|
||||||
|
private val SPARK_CLASS_REGEX = """^spark(\.api\.java)?(\.rdd)?\.[A-Z]""".r
|
||||||
|
|
||||||
private[spark] class CallSiteInfo(val lastSparkMethod: String, val firstUserFile: String,
|
private[spark] class CallSiteInfo(val lastSparkMethod: String, val firstUserFile: String,
|
||||||
val firstUserLine: Int, val firstUserClass: String)
|
val firstUserLine: Int, val firstUserClass: String)
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* When called inside a class in the spark package, returns the name of the user code class
|
* When called inside a class in the spark package, returns the name of the user code class
|
||||||
* (outside the spark package) that called into Spark, as well as which Spark method they called.
|
* (outside the spark package) that called into Spark, as well as which Spark method they called.
|
||||||
|
|
@ -596,7 +626,7 @@ private object Utils extends Logging {
|
||||||
|
|
||||||
for (el <- trace) {
|
for (el <- trace) {
|
||||||
if (!finished) {
|
if (!finished) {
|
||||||
if (el.getClassName.startsWith("spark.") && !el.getClassName.startsWith("spark.examples.")) {
|
if (SPARK_CLASS_REGEX.findFirstIn(el.getClassName) != None) {
|
||||||
lastSparkMethod = if (el.getMethodName == "<init>") {
|
lastSparkMethod = if (el.getMethodName == "<init>") {
|
||||||
// Spark method is a constructor; get its class name
|
// Spark method is a constructor; get its class name
|
||||||
el.getClassName.substring(el.getClassName.lastIndexOf('.') + 1)
|
el.getClassName.substring(el.getClassName.lastIndexOf('.') + 1)
|
||||||
|
|
@ -621,15 +651,16 @@ private object Utils extends Logging {
|
||||||
callSiteInfo.firstUserLine)
|
callSiteInfo.firstUserLine)
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Return a string containing the last `n` bytes of a file. */
|
/** Return a string containing part of a file from byte 'start' to 'end'. */
|
||||||
def lastNBytes(path: String, n: Int): String = {
|
def offsetBytes(path: String, start: Long, end: Long): String = {
|
||||||
val file = new File(path)
|
val file = new File(path)
|
||||||
val length = file.length()
|
val length = file.length()
|
||||||
val buff = new Array[Byte](math.min(n, length.toInt))
|
val effectiveEnd = math.min(length, end)
|
||||||
val skip = math.max(0, length - n)
|
val effectiveStart = math.max(0, start)
|
||||||
|
val buff = new Array[Byte]((effectiveEnd-effectiveStart).toInt)
|
||||||
val stream = new FileInputStream(file)
|
val stream = new FileInputStream(file)
|
||||||
|
|
||||||
stream.skip(skip)
|
stream.skip(effectiveStart)
|
||||||
stream.read(buff)
|
stream.read(buff)
|
||||||
stream.close()
|
stream.close()
|
||||||
Source.fromBytes(buff).mkString
|
Source.fromBytes(buff).mkString
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java
|
package spark.api.java
|
||||||
|
|
||||||
import spark.RDD
|
import spark.RDD
|
||||||
|
|
@ -98,33 +115,48 @@ class JavaDoubleRDD(val srdd: RDD[scala.Double]) extends JavaRDDLike[Double, Jav
|
||||||
|
|
||||||
// Double RDD functions
|
// Double RDD functions
|
||||||
|
|
||||||
/** Return the sum of the elements in this RDD. */
|
/** Add up the elements in this RDD. */
|
||||||
def sum(): Double = srdd.sum()
|
def sum(): Double = srdd.sum()
|
||||||
|
|
||||||
/** Return a [[spark.StatCounter]] describing the elements in this RDD. */
|
/**
|
||||||
|
* Return a [[spark.util.StatCounter]] object that captures the mean, variance and count
|
||||||
|
* of the RDD's elements in one operation.
|
||||||
|
*/
|
||||||
def stats(): StatCounter = srdd.stats()
|
def stats(): StatCounter = srdd.stats()
|
||||||
|
|
||||||
/** Return the mean of the elements in this RDD. */
|
/** Compute the mean of this RDD's elements. */
|
||||||
def mean(): Double = srdd.mean()
|
def mean(): Double = srdd.mean()
|
||||||
|
|
||||||
/** Return the variance of the elements in this RDD. */
|
/** Compute the variance of this RDD's elements. */
|
||||||
def variance(): Double = srdd.variance()
|
def variance(): Double = srdd.variance()
|
||||||
|
|
||||||
/** Return the standard deviation of the elements in this RDD. */
|
/** Compute the standard deviation of this RDD's elements. */
|
||||||
def stdev(): Double = srdd.stdev()
|
def stdev(): Double = srdd.stdev()
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Compute the sample standard deviation of this RDD's elements (which corrects for bias in
|
||||||
|
* estimating the standard deviation by dividing by N-1 instead of N).
|
||||||
|
*/
|
||||||
|
def sampleStdev(): Double = srdd.sampleStdev()
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Compute the sample variance of this RDD's elements (which corrects for bias in
|
||||||
|
* estimating the standard variance by dividing by N-1 instead of N).
|
||||||
|
*/
|
||||||
|
def sampleVariance(): Double = srdd.sampleVariance()
|
||||||
|
|
||||||
/** Return the approximate mean of the elements in this RDD. */
|
/** Return the approximate mean of the elements in this RDD. */
|
||||||
def meanApprox(timeout: Long, confidence: Double): PartialResult[BoundedDouble] =
|
def meanApprox(timeout: Long, confidence: Double): PartialResult[BoundedDouble] =
|
||||||
srdd.meanApprox(timeout, confidence)
|
srdd.meanApprox(timeout, confidence)
|
||||||
|
|
||||||
/** Return the approximate mean of the elements in this RDD. */
|
/** (Experimental) Approximate operation to return the mean within a timeout. */
|
||||||
def meanApprox(timeout: Long): PartialResult[BoundedDouble] = srdd.meanApprox(timeout)
|
def meanApprox(timeout: Long): PartialResult[BoundedDouble] = srdd.meanApprox(timeout)
|
||||||
|
|
||||||
/** Return the approximate sum of the elements in this RDD. */
|
/** (Experimental) Approximate operation to return the sum within a timeout. */
|
||||||
def sumApprox(timeout: Long, confidence: Double): PartialResult[BoundedDouble] =
|
def sumApprox(timeout: Long, confidence: Double): PartialResult[BoundedDouble] =
|
||||||
srdd.sumApprox(timeout, confidence)
|
srdd.sumApprox(timeout, confidence)
|
||||||
|
|
||||||
/** Return the approximate sum of the elements in this RDD. */
|
/** (Experimental) Approximate operation to return the sum within a timeout. */
|
||||||
def sumApprox(timeout: Long): PartialResult[BoundedDouble] = srdd.sumApprox(timeout)
|
def sumApprox(timeout: Long): PartialResult[BoundedDouble] = srdd.sumApprox(timeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java
|
package spark.api.java
|
||||||
|
|
||||||
import java.util.{List => JList}
|
import java.util.{List => JList}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java
|
package spark.api.java
|
||||||
|
|
||||||
import spark._
|
import spark._
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java
|
package spark.api.java
|
||||||
|
|
||||||
import java.util.{List => JList, Comparator}
|
import java.util.{List => JList, Comparator}
|
||||||
|
|
@ -384,4 +401,29 @@ trait JavaRDDLike[T, This <: JavaRDDLike[T, This]] extends Serializable {
|
||||||
val comp = com.google.common.collect.Ordering.natural().asInstanceOf[Comparator[T]]
|
val comp = com.google.common.collect.Ordering.natural().asInstanceOf[Comparator[T]]
|
||||||
top(num, comp)
|
top(num, comp)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Returns the first K elements from this RDD as defined by
|
||||||
|
* the specified Comparator[T] and maintains the order.
|
||||||
|
* @param num the number of top elements to return
|
||||||
|
* @param comp the comparator that defines the order
|
||||||
|
* @return an array of top elements
|
||||||
|
*/
|
||||||
|
def takeOrdered(num: Int, comp: Comparator[T]): JList[T] = {
|
||||||
|
import scala.collection.JavaConversions._
|
||||||
|
val topElems = rdd.takeOrdered(num)(Ordering.comparatorToOrdering(comp))
|
||||||
|
val arr: java.util.Collection[T] = topElems.toSeq
|
||||||
|
new java.util.ArrayList(arr)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Returns the first K elements from this RDD using the
|
||||||
|
* natural ordering for T while maintain the order.
|
||||||
|
* @param num the number of top elements to return
|
||||||
|
* @return an array of top elements
|
||||||
|
*/
|
||||||
|
def takeOrdered(num: Int): JList[T] = {
|
||||||
|
val comp = com.google.common.collect.Ordering.natural().asInstanceOf[Comparator[T]]
|
||||||
|
takeOrdered(num, comp)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java
|
package spark.api.java
|
||||||
|
|
||||||
import java.util.{Map => JMap}
|
import java.util.{Map => JMap}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java;
|
package spark.api.java;
|
||||||
|
|
||||||
import java.util.Arrays;
|
import java.util.Arrays;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java;
|
package spark.api.java;
|
||||||
|
|
||||||
import spark.storage.StorageLevel;
|
import spark.storage.StorageLevel;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java.function;
|
package spark.api.java.function;
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java.function;
|
package spark.api.java.function;
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java.function
|
package spark.api.java.function
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java.function
|
package spark.api.java.function
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java.function;
|
package spark.api.java.function;
|
||||||
|
|
||||||
import scala.reflect.ClassManifest;
|
import scala.reflect.ClassManifest;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java.function;
|
package spark.api.java.function;
|
||||||
|
|
||||||
import scala.reflect.ClassManifest;
|
import scala.reflect.ClassManifest;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,20 @@
|
||||||
|
/*
|
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
* contributor license agreements. See the NOTICE file distributed with
|
||||||
|
* this work for additional information regarding copyright ownership.
|
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
* (the "License"); you may not use this file except in compliance with
|
||||||
|
* the License. You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
package spark.api.java.function;
|
package spark.api.java.function;
|
||||||
|
|
||||||
import scala.Tuple2;
|
import scala.Tuple2;
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue