10/29/2019

Change log level at runtime for logback

For Ops issues cares a lot about logging to store in elasticsearch, which is impacted to the performance. In past We tried to reduce logging as default and change to debug level, but sometimes devs also need to go to troubleshooting incidents in production. Here is change log level way which doesn't need to rebuild project based on logback jmx feature.

Enable JMX in logback configuration

jmxConfigurator need to be added in logback.

  <jmxConfigurator />
  
  <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
    <layout class="ch.qos.logback.classic.PatternLayout">
      <Pattern>%date [%thread] %-5level %logger{25} - %msg%n</Pattern>
    </layout>
  </appender>

  <root level="debug">
    <appender-ref ref="console" />
  </root>  
</configuration>

The logback with jmx can be verified by jconsole.

Use jmxterm command line based interactive to access

Welcome to JMX terminal. Type "help" for available commands.
$>open 2767  <--process id
#Connection to 2767 is opened
$>domains
#following domains are available
JMImplementation
ch.qos.logback.classic
com.sun.management
java.lang
java.nio
java.util.logging
kafka
kafka.consumer
kafka.producer
$>domain ch.qos.logback.classic
#domain is set to ch.qos.logback.classic
$>bean ch.qos.logback.classic:Name=default,Type=ch.qos.logback.classic.jmx.JMXConfigurator
#bean is set to ch.qos.logback.classic:Name=default,Type=ch.qos.logback.classic.jmx.JMXConfigurator
$>run setLoggerLevel com.abc.cde.consumer.MessageRequestConsumer DEBUG  
#calling operation setLoggerLevel of mbean ch.qos.logback.classic:Name=default,Type=ch.qos.logback.classic.jmx.JMXConfigurator with params [com.abc.cde.consumer.MessageRequestConsumer, DEBUG]
#operation returns:
null
$>close
#disconnected

jmxterm has been supported silent mode, so we can easily implement a shell script for some special handling as below example.

#!/bin/bash
file=jmxterm-1.0.0-uber.jar
if [ ! -f "$file" ]; then
    wget https://github.com/jiaqi/jmxterm/releases/download/v1.0.0/jmxterm-1.0.0-uber.jar
fi
processId=`jps -lvm | grep Bootstrap | awk '{print $1}'`
echo "Java App processId; $processId ; fm logging level to ;$1"
cat <<EOF > jmxcommands
open $processId
bean ch.qos.logback.classic:Name=default,Type=ch.qos.logback.classic.jmx.JMXConfigurator
run setLoggerLevel com.abc.cdef.Parser $1
close
EOF
java -jar "$file" -n < jmxcommands

References