Page tree
Skip to end of metadata
Go to start of metadata

Problems: Currently if we have to change CDAP program's log-level, we have to change the log level in logback.xml in (src/main/resources) of the app and redeploy the app.If the app's program is already running, it has to be restarted to get the logback changes in the app.

Reason: Currently when we start a program, we will look for “logback.xml” in the program classLoader. If that is not provided, we will use the default file “logback-container.xml”. Therefore, if we want to change the log level now, we will have to make changes to the “logback.xml” and have to redeploy the app to reflect the changes.

Requirement/Goals: Our goal is to make log level change dynamically at runtime if there is a request.

Use cases: 

Users can change log level of any applications or programs dynamically through the API. They should be able to see the log level change at runtime without redeploying the applications or restarting the programs. They should also be able to change log level at class level to specify which level they want to see for each class.

Proposal:

 We will have two parts to fulfill this goal. Part 1: we will enable log level change per program run without redeploying the app. With this, we should be able to see log level change after restarting the program. Part 2: we will enable the ability to update the aggregate log level for a particular runnable in a Twill application.

For Part 1: Enable log level change per program run without redeploying the app

Step 1: Use the preferences to pass log level information

Pass the log level information as preferences and change the log level correspondingly once the program starts. With the property of the preferences, the log level can be set through namespace, application and program level. The key of the preferences should be “system.log.level”, and the value will be one of the valid log levels. (ALL, OFF, TRACE, DEBUG, INFO, WARN, ERROR) If the value is invalid log levels, this log level change will be ignored and the default log level "INFO" will be used.

Screen Shot 2016-09-07 at 4.46.02 PM.png

In AbstractProgramTwillRunnable, DistributedMapReduceTaskContextProvider, and SparkRuntimeContextProvider, we get the log level information by the key “system.log.level” from ProgramOptions. Make LogAppenderInitializer’s initialize method accept the log level and change the root logger’s log level.

After step 1, the log level should be correctly changed for all classes in programs except for classes with same namespace prefixes in logback-container.xml. Changing the root logger is not enough since the loggers for these classes will get overridden and therefore their log level will be INFO.

Step 2: Enable the log level for class by passing the class name with prefix “system.log.level”.

 Screen Shot 2016-09-07 at 4.59.52 PM.png

Extract the prefix “system.log.level”, get all the classes and levels. Then pass them to LogAppenderInitializer to change the log level for their loggers. For a flow, preferences with key format "flowlet.[flowletname].system.log.level.[class]" can be passed to change the log level of a specific flowlet. The idea is like how scoping works for system.resources.memory, we extract the scope of flowlet.name, and set the log level in FlowTwillRunnable.

Step 3: Remove some settings in logback-container.xml

Screen Shot 2016-09-07 at 5.33.36 PM.png

For these loggers in logback-container.xml, it is unclear to the user when they want to change the log levels related to these loggers. So it is better to remove them from the xml and add them to system preferences.

For Part 2: Enable log level change dynamically

REST API changes: The changes will be made in PreferencesHttpHandler and happen in application or program level:

GET /v3/namespaces/<namespace-id>/apps/<app-id>/loglevel

GET /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/loglevel

Get the current log level of the application or the program. The response should be a JSON string map of the log level: {"loglevel":"value1"}

PUT  /v3/namespaces/<namespace-id>/apps/<app-id>/loglevel {request}

PUT /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/loglevel {request}

Set preferences for the application or programs, the request body should be a json related to the log level, one out of TRACE, DEBUG, INFO, WARN, ERROR, FATAL

DELETE /v3/namespaces/<namespace-id>/apps/<app-id>/loglevel

DELETE /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/loglevel

 Delete the preferences of the application or programs, the program should use the configuration in “logback.xml” as the log level.


Rest: To be decided...


  • No labels

11 Comments

  1. If the system.log.level is missing or invalid, we should just ignore it, instead of defaulting to DEBUG.

    1. Updated the doc, if the log level is invalid we ignore it and just use the default log level in logback.xml

  2. The log level through runtime argument should be able to set the scope to a particular flowlet as well. See how scoping applies on system.resources.memory for a flowlet.

  3. What about CDAP system service containers? are they in scope?  They also use logback-container.xml and so the proposed changes to that file will cause our own logs to be chatty.

    1. +1. We need to think of system service containers too

  4. Also need to figure out the implementation for standalone

    1. To enable it for standalone, the changes needed will be very complicated and I am not sure if it worths the effort. In order to do that, we need to make slf4j and logback classes loaded through a different classloader for different program run. However, the problem with that approach is, the LoggerEvent generated, will not be implementing the same ILoggerEvent interface, since that interface will be loaded by different classloader as well. The implication of that is, for the local log collection, it cannot operate on the ILoggerEvent interface at all, meaning non of the log appender configured at the system would work.

  5. Does it works in standalone cdap? I wanted to check this before implemented in distributed cdap

    1. No. This design is only for distributed cdap. This page is a bit out dated. You can find out the way to dynamically change log levels here: http://docs.cask.co/cdap/4.1.1/en/reference-manual/http-restful-api/logging.html#changing-program-log-levels. And to change the log levels before program starts without modifying logback.xml and redeploying app: http://docs.cask.co/cdap/4.1.1/en/admin-manual/operations/logging.html#changing-program-log-levels. In 4.2.0, we will support change cdap system service log levels dynamiclly.

      1. Hi Feng,

        Thanks for reply.  But i am not able to find the exact command to set the log level of a program before we run program 

        PUT /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/runs/<run-id>/loglevels
         In above syntax how do we find <run-id> before running program? our requirement is to set log level for particular program/workflow

        before running it.

         

        1. Hi Naresh,

          The above REST endpoints is for you to change log levels in realtime programs dynamically when they are running. To change the log level before the program starts, you can use preferences. You can add the logger name as the key and log level as the value in the preferences through the UI. The logger name should be prefixed with system.log.level. To find more details on this, pls check http://docs.cask.co/cdap/4.1.1/en/admin-manual/operations/logging.html#changing-program-log-levels. Please let me know if you have any other questions.