[SPARK-27130][BUILD] Automatically select profile when executing sbt-checkstyle

## What changes were proposed in this pull request?

This PR makes it automatically select profile when executing `sbt-checkstyle`. The reason for this is that `hadoop-2.7` and `hadoop-3.1` may have different `hive-thriftserver` module in the future.

## How was this patch tested?

manual tests:
```
Update AbstractService.java file.
export HADOOP_PROFILE=hadoop2.7
./dev/run-tests
```
The result:
![image](https://user-images.githubusercontent.com/5399861/54197992-5337e780-4500-11e9-930c-722982cdcd45.png)

Closes #24065 from wangyum/SPARK-27130.

Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
Yuming Wang 2019-03-13 08:03:46 +09:00 committed by Hyukjin Kwon
parent f57af2286f
commit dccf6615c3
2 changed files with 10 additions and 11 deletions

View file

@ -180,9 +180,13 @@ def run_scala_style_checks():
run_cmd([os.path.join(SPARK_HOME, "dev", "lint-scala")])
def run_java_style_checks():
def run_java_style_checks(build_profiles):
set_title_and_block("Running Java style checks", "BLOCK_JAVA_STYLE")
run_cmd([os.path.join(SPARK_HOME, "dev", "sbt-checkstyle")])
# The same profiles used for building are used to run Checkstyle by SBT as well because
# the previous build looks reused for Checkstyle and affecting Checkstyle. See SPARK-27130.
profiles = " ".join(build_profiles)
print("[info] Checking Java style using SBT with these profiles: ", profiles)
run_cmd([os.path.join(SPARK_HOME, "dev", "sbt-checkstyle"), profiles])
def run_python_style_checks():
@ -333,7 +337,7 @@ def build_spark_assembly_sbt(hadoop_version, checkstyle=False):
exec_sbt(profiles_and_goals)
if checkstyle:
run_java_style_checks()
run_java_style_checks(build_profiles)
build_spark_unidoc_sbt(hadoop_version)

View file

@ -17,17 +17,12 @@
# limitations under the License.
#
profiles=${1:-"-Pkinesis-asl -Pmesos -Pkubernetes -Pyarn -Phive -Phive-thriftserver"}
# NOTE: echo "q" is needed because SBT prompts the user for input on encountering a build file
# with failure (either resolution or compilation); the "q" makes SBT quit.
ERRORS=$(echo -e "q\n" \
| build/sbt \
-Pkinesis-asl \
-Pmesos \
-Pkubernetes \
-Pyarn \
-Phive \
-Phive-thriftserver \
checkstyle test:checkstyle \
| build/sbt ${profiles} checkstyle test:checkstyle \
| awk '{if($1~/error/)print}' \
)