e6dd237463
In the default Spark distribution, there are currently two separate log4j config files, with different default values for the root logger, so that when running the shell you have a different default log level. This makes the shell more usable, since the logs don't overwhelm the output. But if you install a custom log4j.properties, you lose that, because then it's going to be used no matter whether you're running a regular app or the shell. With this change, the overriding of the log level is done differently; the log level repl's main class (org.apache.spark.repl.Main) is used to define the root logger's level when running the shell, defaulting to WARN if it's not set explicitly. On a somewhat related change, the shell output about the "sc" variable was changed a bit to contain a little more useful information about the application, since when the root logger's log level is WARN, that information is never shown to the user. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #9816 from vanzin/shell-logging.
41 lines
2 KiB
Plaintext
41 lines
2 KiB
Plaintext
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
# Set everything to be logged to the console
|
|
log4j.rootCategory=INFO, console
|
|
log4j.appender.console=org.apache.log4j.ConsoleAppender
|
|
log4j.appender.console.target=System.err
|
|
log4j.appender.console.layout=org.apache.log4j.PatternLayout
|
|
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
|
|
|
|
# Set the default spark-shell log level to WARN. When running the spark-shell, the
|
|
# log level for this class is used to overwrite the root logger's log level, so that
|
|
# the user can have different defaults for the shell and regular Spark apps.
|
|
log4j.logger.org.apache.spark.repl.Main=WARN
|
|
|
|
# Settings to quiet third party logs that are too verbose
|
|
log4j.logger.org.spark-project.jetty=WARN
|
|
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
|
|
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
|
|
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
|
|
log4j.logger.org.apache.parquet=ERROR
|
|
log4j.logger.parquet=ERROR
|
|
|
|
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
|
|
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
|
|
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
|