[SPARK-27130][BUILD] Automatically select profile when executing sbt-checkstyle
## What changes were proposed in this pull request? This PR makes it automatically select profile when executing `sbt-checkstyle`. The reason for this is that `hadoop-2.7` and `hadoop-3.1` may have different `hive-thriftserver` module in the future. ## How was this patch tested? manual tests: ``` Update AbstractService.java file. export HADOOP_PROFILE=hadoop2.7 ./dev/run-tests ``` The result: ![image](https://user-images.githubusercontent.com/5399861/54197992-5337e780-4500-11e9-930c-722982cdcd45.png) Closes #24065 from wangyum/SPARK-27130. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
parent
f57af2286f
commit
dccf6615c3
|
@ -180,9 +180,13 @@ def run_scala_style_checks():
|
|||
run_cmd([os.path.join(SPARK_HOME, "dev", "lint-scala")])
|
||||
|
||||
|
||||
def run_java_style_checks():
|
||||
def run_java_style_checks(build_profiles):
|
||||
set_title_and_block("Running Java style checks", "BLOCK_JAVA_STYLE")
|
||||
run_cmd([os.path.join(SPARK_HOME, "dev", "sbt-checkstyle")])
|
||||
# The same profiles used for building are used to run Checkstyle by SBT as well because
|
||||
# the previous build looks reused for Checkstyle and affecting Checkstyle. See SPARK-27130.
|
||||
profiles = " ".join(build_profiles)
|
||||
print("[info] Checking Java style using SBT with these profiles: ", profiles)
|
||||
run_cmd([os.path.join(SPARK_HOME, "dev", "sbt-checkstyle"), profiles])
|
||||
|
||||
|
||||
def run_python_style_checks():
|
||||
|
@ -333,7 +337,7 @@ def build_spark_assembly_sbt(hadoop_version, checkstyle=False):
|
|||
exec_sbt(profiles_and_goals)
|
||||
|
||||
if checkstyle:
|
||||
run_java_style_checks()
|
||||
run_java_style_checks(build_profiles)
|
||||
|
||||
build_spark_unidoc_sbt(hadoop_version)
|
||||
|
||||
|
|
|
@ -17,17 +17,12 @@
|
|||
# limitations under the License.
|
||||
#
|
||||
|
||||
profiles=${1:-"-Pkinesis-asl -Pmesos -Pkubernetes -Pyarn -Phive -Phive-thriftserver"}
|
||||
|
||||
# NOTE: echo "q" is needed because SBT prompts the user for input on encountering a build file
|
||||
# with failure (either resolution or compilation); the "q" makes SBT quit.
|
||||
ERRORS=$(echo -e "q\n" \
|
||||
| build/sbt \
|
||||
-Pkinesis-asl \
|
||||
-Pmesos \
|
||||
-Pkubernetes \
|
||||
-Pyarn \
|
||||
-Phive \
|
||||
-Phive-thriftserver \
|
||||
checkstyle test:checkstyle \
|
||||
| build/sbt ${profiles} checkstyle test:checkstyle \
|
||||
| awk '{if($1~/error/)print}' \
|
||||
)
|
||||
|
||||
|
|
Loading…
Reference in a new issue