Page tree
Skip to end of metadata
Go to start of metadata

Checklist

  • User Stories Documented
  • User Stories Reviewed
  • Design Reviewed
  • APIs reviewed
  • Release priorities assigned
  • Test cases reviewed
  • Blog post

Introduction

For each RESTful http request, we currently only log the URL of the request. We need to log the request body or response body if useful information (dataset properties, program runtime arguments, etc.) is included. 

Goals

Improve the audit logging in the router.

User Stories 

User would like to know what more for some kind of HTTP request. For example, when user updates the dataset property, in the audit log, the request body which contains the newly updated dataset property will be very helpful. 

Design 

All RESTful endpoints that needs to be logged with more information are in this table, click to expand. This table has all POST, DELETE, PUT endpoints other than GET. All GET methods do not have a body or header params so the URL is enough for audit logging.

 Click here to expand...
MetricsHandler 
[main] - [POST] /v3/metrics//queryQuery param: metric, tag, groupby(optional), time range
[main] - [POST] /v3/metrics//searchQuery param: target, tag.
  
DatasetTypeHandler 
[main] - [DELETE] /v3/namespaces/{namespace-id}//data/modules/{name}URL is enough
[main] - [PUT] /v3/namespaces/{namespace-id}//data/modules/{name}Header param: class name of the module.
[main] - [DELETE] /v3/namespaces/{namespace-id}//data/modulesURL is enough
  
DatasetInstanceHandler 
[main] - [PUT] /v3/namespaces/{namespace-id}//data/datasets/{name}/propertiesrequest body: updated properties
[main] - [PUT] /v3/namespaces/{namespace-id}//data/datasets/{name}request body: name of the dataset type, properties, and description
[main] - [DELETE] /v3/namespaces/{namespace-id}//data/datasets/{name}URL is enough
[main] - [DELETE] /v3/namespaces/{namespace-id}//data/datasetsURL is enough
[main] - [POST] /v3/namespaces/{namespace-id}//data/datasets/{name}/admin/{method}URL is enough
[main] - [POST] /v3/namespaces/{namespace-id}//data/datasets/{name}/data/{method}Not implemented
  
NamespacedQueryExecutorHttpHandler 
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/queriesrequest body: query string
  
QueryExecutorHttpHandler 
[main] - [DELETE] /v3/data/explore/queries/{id}URL is enough
[main] - [POST] /v3/data/explore/queries/{id}/nextURL is enough
[main] - [POST] /v3/data/explore/queries/{id}/previewURL is enough
[main] - [POST] /v3/data/explore/queries/{id}/downloadURL is enough
  
NamespacedExploreMetadataHttpHandlerThese four endpoints have an issue: cdap-7625
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/jdbc/tablesrequest body: TableArgs
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/jdbc/columnsrequest body: ColumnArgs
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/jdbc/schemasrequest body: SchemaArgs
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/jdbc/functionsrequest body: FunctionArgs
  
ExploreMetadataHttpHandler 
[main] - [DELETE] /v3/data/explore/namespaces/{namespace-id}Internal use only, URL is enough
[main] - [PUT] /v3/data/explore/namespaces/{namespace-id}Internal use only, URL is enough
[main] - [POST] /v3/data/explore/jdbc/catalogsURL is enough
[main] - [POST] /v3/data/explore/jdbc/tableTypesURL is enough
[main] - [POST] /v3/data/explore/jdbc/typesURL is enough
  
ExploreExecutorHttpHandler 
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/streams/{stream}/tables/{table}/enablerequest body: FormatSpecification
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/streams/{stream}/tables/{table}/disableURL is enough
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/datasets/{dataset}/enable-internalrequest body: enabled DatasetSpecification
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/datasets/{dataset}/enableURL is enough
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/datasets/{dataset}/updaterequest body: old and new DatasetSpecification
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/datasets/{dataset}/disableURL is enough
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/datasets/{dataset}/partitionsrequest body: properties of partition
[main] - [POST] /v3/namespaces/{namespace-id}/data/explore/datasets/{dataset}/deletePartitionrequest body: properties of partition
  
MonitorHandler 
[StandaloneAppFabricServer] - [PUT] /v3//system/services/{service-name}/instancesrequest body: instance number
[StandaloneAppFabricServer] - [POST] /v3//system/services/{service-name}/restartURL is enough
[StandaloneAppFabricServer] - [POST] /v3//system/services/{service-name}/instances/{instance-id}/restartURL is enough
  
NamespaceHttpHandler 
[StandaloneAppFabricServer] - [DELETE] /v3//unrecoverable/namespaces/{namespace-id}URL is enough
[StandaloneAppFabricServer] - [PUT] /v3//namespaces/{namespace-id}request body: namespace properties
[StandaloneAppFabricServer] - [DELETE] /v3//unrecoverable/namespaces/{namespace-id}/datasetsURL is enough
[StandaloneAppFabricServer] - [PUT] /v3//namespaces/{namespace-id}/propertiesrequest body: updated namespace properties
  
NotificationFeedHttpHandler 
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//feeds/categories/{feed-category}/names/{feed-name}request body: description
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}//feeds/categories/{feed-category}/names/{feed-name}URL is enough
  
AppLifecycleHttpHandlerREST endpoints inconsistency?
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-id}request body: JSON object specifiy the artifact and optional config
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//appsHeader param: archiveName, configString, response body: appId
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/versions/{version-id}/createrequest body: JSON object specifiy the artifact and optional config
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}//apps/{app-id}URL is enough
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}//apps/{app-id}/versions/{version-id}URL is enough
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}//appsURL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/updaterequest body: JSON object specifiy the updated artifact and optional config
  
DashboardHttpHandler 
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}/configuration/dashboards//{dashboard-id}URL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}/configuration/dashboards//request body: dashboard config, response body: dashboard id
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}/configuration/dashboards//{dashboard-id}request body: dashboard config
  
ProgramLifecycleHttpHandler 
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-id}/flows/{flow-id}/flowlets/{flowlet-id}/instancesrequest body: flowlet instance number
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-id}/services/{service-id}/instancesrequest body: service instance number
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-id}/workers/{worker-id}/instancesrequest body: worker instance number
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-name}/{program-type}/{program-name}/runs/{run-id}/loglevelsrequest body: log levels
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-name}/versions/{app-version}/{program-type}/{program-name}/runs/{run-id}/loglevelsrequest body: log levels
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-name}/versions/{app-version}/{program-type}/{program-name}/runs/{run-id}/resetloglevelsrequest body: reset logger names
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-name}/{program-type}/{program-name}/runs/{run-id}/resetloglevelsrequest body: reset logger names
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/stopURL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/{program-type}/{program-id}/{action}request body: runtime arguments for action start/debug
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/versions/{app-version}/{program-type}/{program-id}/{action}request body: runtime arguments for action start/debug
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-name}/{program-type}/{program-name}/runtimeargsrequest body: runtime arguments
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-name}/versions/{app-version}/{program-type}/{program-name}/runtimeargsrequest body: runtime arguments
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//statusrequest body: program info
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//stoprequest body: program info, response body: stop status for each program
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//startrequest body: program info, response body: start status for each program
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-id}/flows/{flow-id}/flowlets/{flowlet-id}/runs/{run-id}/loglevelsrequest body: log levels
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}//apps/{app-id}/versions/{app-version}/flows/{flow-id}/flowlets/{flowlet-id}/runs/{run-id}/loglevelsrequest body: log levels
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/flows/{flow-id}/flowlets/{flowlet-id}/runs/{run-id}/resetloglevelsrequest body: reset logger names
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/versions/{app-version}/flows/{flow-id}/flowlets/{flowlet-id}/runs/{run-id}/resetloglevelsrequest body: reset logger names
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}//apps/{app-id}/flows/{flow-id}/queuesURL is enough
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}//queuesURL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//instancesrequest body: program info
  
PreferencesHttpHandler 
[StandaloneAppFabricServer] - [DELETE] /v3//preferencesURL is enough
[StandaloneAppFabricServer] - [PUT] /v3//preferencesrequest body: preferences
[StandaloneAppFabricServer] - [PUT] /v3//namespaces/{namespace-id}/preferencesrequest body: preferences
[StandaloneAppFabricServer] - [DELETE] /v3//namespaces/{namespace-id}/preferencesURL is enough
[StandaloneAppFabricServer] - [PUT] /v3//namespaces/{namespace-id}/apps/{application-id}/preferencesrequest body: preferences
[StandaloneAppFabricServer] - [DELETE] /v3//namespaces/{namespace-id}/apps/{application-id}/preferencesURL is enough
[StandaloneAppFabricServer] - [PUT] /v3//namespaces/{namespace-id}/apps/{application-id}/{program-type}/{program-id}/preferencesrequest body: preferences
[StandaloneAppFabricServer] - [DELETE] /v3//namespaces/{namespace-id}/apps/{application-id}/{program-type}/{program-id}/preferencesURL is enough
  
ConsoleSettingsHttpHandler 
[StandaloneAppFabricServer] - [DELETE] /v3/configuration/user//URL is enough
[StandaloneAppFabricServer] - [PUT] /v3/configuration/user//request body: JSON string of properties
  
TransactionHttpHandler 
[StandaloneAppFabricServer] - [POST] /v3//transactions/{tx-id}/invalidateURL is enough
[StandaloneAppFabricServer] - [POST] /v3//transactions/invalid/remove/untilrequest body: timestamp
[StandaloneAppFabricServer] - [POST] /v3//transactions/invalid/remove/idsrequest body: transaction ids
[StandaloneAppFabricServer] - [POST] /v3//transactions/stateURL is enough
  
WorkflowHttpHandler 
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/workflows/{workflow-name}/runs/{run-id}/suspendURL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//apps/{app-id}/workflows/{workflow-name}/runs/{run-id}/resumeURL is enough
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}//apps/{app-id}/workflows/{workflow-id}/runs/{run-id}/localdatasetsURl is enough
  
ArtifactHttpHandler 
[StandaloneAppFabricServer] - [POST] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}header-param: Artifact-Versions, Artifact-Extends, Artifact-Plugins
[StandaloneAppFabricServer] - [DELETE] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}URL is enough
[StandaloneAppFabricServer] - [DELETE] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/propertiesURL is enough
[StandaloneAppFabricServer] - [POST] /v3//namespaces/system/artifactsURL is enough
[StandaloneAppFabricServer] - [PUT] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/propertiesrequest body: artifact properties
[StandaloneAppFabricServer] - [PUT] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/properties/{property}request body: property value
[StandaloneAppFabricServer] - [DELETE] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/properties/{property}URL is enough
[StandaloneAppFabricServer] - [POST] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/plugintypes/{plugin-type}/plugins/{plugin-name}/methods/{plugin-method}request body: method parameter type, response body: result from invoking the method
  
AuthorizationHandler 
[StandaloneAppFabricServer] - [POST] /v3/security/authorization//privileges/revokerequest body: principle, entity, action
[StandaloneAppFabricServer] - [PUT] /v3/security/authorization//roles/{role-name}URL is enough
[StandaloneAppFabricServer] - [DELETE] /v3/security/authorization//roles/{role-name}URL is enough
[StandaloneAppFabricServer] - [PUT] /v3/security/authorization//{principal-type}/{principal-name}/roles/{role-name}URL is enough
[StandaloneAppFabricServer] - [DELETE] /v3/security/authorization//{principal-type}/{principal-name}/roles/{role-name}URL is enough
[StandaloneAppFabricServer] - [POST] /v3/security/authorization//privileges/grantrequest body: principle, entity, action
  
SecureStoreHandler 
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}/securekeys//{key-name}URL is enough
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}/securekeys//{key-name}request body: key information
  
RouteConfigHttpHandler 
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}/apps/{app-id}/services/{service-id}//routeconfigrequest body: a map of version strings
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}/apps/{app-id}/services/{service-id}//routeconfigURL is enough
  
StreamHandler 
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace-id}/streams//{stream}URL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}/streams//{stream}URL is enough
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}/streams//{stream}request body: stream properties
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}/streams//{stream}/truncateURL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}/streams//{stream}/asyncURL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}/streams//{stream}/batchURL is enough
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace-id}/streams//{stream}/propertiesrequest body: stream properties
  
StreamViewHttpHandler 
[StandaloneAppFabricServer] - [DELETE] /v3/namespaces/{namespace}//streams/{stream}/views/{view}URL is enough
[StandaloneAppFabricServer] - [PUT] /v3/namespaces/{namespace}//streams/{stream}/views/{view}request body: the property of the view
  
PreviewHttpHandlerThe preview run will exist in preview space and will not be available in some minutes, so I think URL is enough for all rest calls
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//previewsURL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//previews/{preview-id}/stopURL is enough
[StandaloneAppFabricServer] - [POST] /v3/namespaces/{namespace-id}//previews/{preview-id}/tracersURL is enough
  
MetadataHttpHandler 
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/apps/{app-id}/metadata/propertiesrequest body: metadata properties
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/metadata/propertiesrequest body: metadata properties
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/propertiesrequest body: metadata properties
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/datasets/{dataset-id}/metadata/propertiesrequest body: metadata properties
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/streams/{stream-id}/metadata/propertiesrequest body: metadata properties
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/propertiesrequest body: metadata properties
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/metadataURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/metadataURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadataURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/datasets/{dataset-id}/metadataURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/metadataURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadataURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/metadata/propertiesURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/metadata/properties/{property}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/metadata/propertiesURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/metadata/properties/{property}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/propertiesURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties/{property}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/datasets/{dataset-id}/metadata/propertiesURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties/{property}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/metadata/propertiesURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/propertiesURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/metadata/properties/{property}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties/{property}URL is enough
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/apps/{app-id}/metadata/tagsrequest body: metadata tags
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/metadata/tagsrequest body: metadata tags
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tagsrequest body: metadata tags
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tagsrequest body: metadata tags
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/streams/{stream-id}/metadata/tagsrequest body: metadata tags
[MetadataService] - [POST] /v3//namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tagsrequest body: metadata tags
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/metadata/tagsURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/metadata/tags/{tag}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/metadata/tagsURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/metadata/tags/{tag}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tagsURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags/{tag}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tagsURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags/{tag}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/metadata/tagsURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tagsURL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/metadata/tags/{tag}URL is enough
[MetadataService] - [DELETE] /v3//namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags/{tag}URL is enough

Approach

Currently, in SecurityAuthenticationHttpHandler, we already have each request audit logged. For each incoming request, we log the clientIP, the URL from HTTPRequest and the responseCode from HTTPResponse. We will still keep this as the default behavior for audit logging.

To log the request body, header name or response body, we annotate the RESTful endpoint methods with a new custom annotation AuditPolicy such as: 

/**
 * Annotates a REST endpoint method to indicate which content needs to be audit logged.
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface AuditPolicy {

  AuditDetail[] value();
}

And in the AuditDetail:

/**
 * Declares which content needs to be audit logged.
 */
public enum AuditDetail {
  HEADERS,
  REQUEST_BODY,
  RESPONSE_BODY
}


For example, for the deploy app REST call, we want to log several header names and the response body, so we annotate the method as:

/**
 * Annotate the method with the @AuditPolicy, with response body set to true, and set header names to true.
 */
@POST
@Path("/apps")
@AuditPolicy({AuditDetail.HEADERS, AuditDetail.RESPONSE_BODY})
public BodyConsumer deploy(HttpRequest request, HttpResponder responder, ...) throws Exception {
 ...
}


When the router starts up, we will walk through all methods in each handler class in the class path to find if the method has a @AuditPolicy annotation. If the method does not have this annotation, the default behavior is used(log the URL, response body, etc). If it is present, then we will log the additional information provided by this annotation.

How to walk through all handler classes

We will use the class path to get all the handler classes and filter out based on the handler class.

How to audit log

Currently we are doing audit log in SecurityAuthenticationHttpHandler, we will refactor the handler to separate the audit logging since they are unrelated features.

When we receive a http request(in MessageReceived() method), we will create a AuditLogEntry and have the username, URL, request body and headers if needed to log, etc, stored in the object, and attach it to the ChannelHandlerContext. 

When a http response comes(in WriteRequested() method), we get the AuditLogEntry in the context and have the response body, if necessary, stored in the object.

In WriteComplete() method, we log the AuditLogEntry:

Audit log entry

For each http REST call, we currently are using the AuditLogEntry to log the request. We will add the header, request body, response body fields to it. Here is how it looks:

/**
 * Annotate the method with the @AuditPolicy, with response body set to true, and set header names that need to be logged as an array.
 */
public final class AuditLogEntry {
  ...
  private InetAddress clientIP;
  private String userName;
  private Date date;
  private String requestLine;
  private String requestBody;
  private Integer responseCode;
  private Long responseContentLength;
  private String userIdentity;
  private Map<String, String> headers;
  private String responseBody;
  ...
  
  // log the string
  public String toString() {
  return String.format("%s %s %s [%s] \"%s\" %s %s %s %s %s",
                       clientIP != null ? clientIP.getHostAddress() : DEFAULT_VALUE,
                       fieldOrDefault(userIdentity),
                       fieldOrDefault(userName),
                       DEFAULT_DATE_FORMAT.format(date),
                       fieldOrDefault(requestLine),
                       fieldOrDefault(headers),
                       fieldOrDefault(requestBody),
                       fieldOrDefault(responseCode),
                       fieldOrDefault(responseContentLength),
                       fieldOrDefault(responseBody));
  }
}

Security Impact 

The audit logging only happens when security is enabled.

Impact on Infrastructure Outages 

Test Scenarios

Test ID
Test Description
Expected Results
   
   
   
  

 

Releases

release 4.1.0

Related Work

 

Future work

Refactor audit logging from SecurityAuthenticationHttpHandler

ITN for audit logging

  • No labels

26 Comments

  1. What will happen if the method is not annotated? Will the default behavior be to log the URL? 

    1. Yes, if it is not annotated, then we just log the URL by default, and this is already implemented in SecurityAuthenticationHttpHandler

      1. Let's state it in the design about what will happen for the default behavior

  2. How do we log the header params?

    1. Updated the doc, we will have header params that need to be logged in the AuditPolicy annotation

  3. How do we walk through all methods in all handler classes?

    1. I started with hard-coding the handler class in the list. Tried out the ClassPath lib but did not quite get how that can be used. Will explore further today.

  4. a new custom annotation AuditPolicy

    what is an example for usage of this annotation? Should these be multiple booleans, or should we have an enum?

    1. Updated the doc with an example.

  5. What's the format/encoding when logging body/headers? The body maybe binary and a given header name may have multiple values. Can you document the audit log file format here as well?

    1. Updated the doc. For the header, based on our REST endpoints, there are only three endpoints which need to be logged the header, and they cannot have multiple values. For the body, the body we want to log are basically something related to properties. Maybe we can do a check to not log large bodies?

      1. But the format should be generic enough to cater for such scenarios.

  6. String [] headerParams() default {};

    Can this be a boolean requestHeaders? 

    The headers are already declared as @HeaderParam in the handler method, it seems listing them here again is duplication.

    1. Yes, updated the doc.

  7. Introduction sounds more like Goal. 

  8. The design does not have all the sections from the template. If there are sections which are not required then keep them blank or say no changes. 

  9. Please add user stories and scenarios

  10. @AuditPolicy(responseBody = true, headerNames = true)

    Should this be headerNames or headerParams?

    1. oh, this is a typo, the method should be called headerNames

      1. So, we are only logging names and not values?

        1. No we are logging the values, the header names are specified with @HeaderParam in the method parameters. This boolean is just telling us whether we need to look up the header name in the parameters.

  11. In design it will be beneficial to add more details as how annotation are processed and the map is created. Currently, there is very minimal info of what is being done. 

    1. Will add user stories, test scenarios and other details in short

  12. Do we need to use boolean params here? Would just specifying what needs to be logged work, like an enum? I don't see 'param' = no being used.

  13. Are there any issues in using one annotation for each ? Say:

    LogRequest(), LogResponse, LogHeaders ?