Page tree
Skip to end of metadata
Go to start of metadata

Checklist

  • User Stories Documented
  • User Stories Reviewed
  • Design Reviewed
  • APIs reviewed
  • Release priorities assigned
  • Test cases reviewed
  • Blog post

Goals

  • Make CDAP authorization policy consistent across all entity types.
  • Allow admins to set granular privileges on entities. 
  • Ranger integration for CDAP authorization
  • Improve Sentry data model to fix existing issues seen on customer environments
  • Allow admins to use existing role/groups for authorization

User Stories 

Scenario 1

Overview

  • Privileges are managed at the entity level
  • App level impersonation
  • Dataset is owned by the application owner
  • Cross namespace dataset access allowed (question)

Details

  • admin1 creates a CDAP namespace etl with principal etl-owner
  • admin2 deploys an app feed1 with principal feed1-owner in namespace etl
  • During app feed1 configure, dataset gold is created with owner principal feed1-owner
  • ops1 starts workflow in app feed1, that runs as principal feed1-owner
  • During the workflow run, principal feed1-owner reads/writes to dataset gold
  • ops1 can list logs and metrics for workflow in app feed1
  • ops2 can list all apps/programs in namespace etl and view all their logs and metrics
  • ops2 can list all the datasets in namespace etl and view its properties (question)
  • ops2 cannot read any datasets in namespace etl

Scenario 2

Overview

  • Privileges are managed at the namespace level
  • Namespace level impersonation
  • Dataset is owned by the namespace owner
  • Cross namespace dataset access allowed

Details

  • admin1 creates a group etl-group in LDAP
  • admin1 creates namespaces in HDFS, HBase and Hive called etl
  • admin1 grants all privileges on the above namespaces to group etl-group
  • admin1 creates a CDAP namespace etl with principal etl-owner (question) using the namespaces from HDFS, HBase and Hive. Does etl-owner belong to etl-group(question)
  • admin1 grants all privileges on the CDAP namespace etl, and all entities under it to group etl-group
  • etl-user1 belonging to group etl-group deploys app feed1 in namespace etl
  • During app feed1 configure, dataset gold is created with owner principal etl-owner
  • etl-user2 belonging to group elt-group, starts workflow in app feed1, that runs as principal etl-owner
  • During the workflow run, principal feed1-owner reads/writes to dataset gold
  • etl-user3 belonging to group elt-group can list logs and metrics for workflow in app feed1
  • analyst1 belonging to group analyst-group is given privilege read on namespace etl and all entities under it, using which analyst1 can read dataset gold

Scenario 3

Overview

  • Privileges are managed at the namespace level
  • No impersonation
  • All data is owned by CDAP
  • All programs run as CDAP
  • Cross namespace dataset access is allowed

Details

  • admin1 creates a group etl-group in LDAP
  • admin1 creates namespaces in HDFS, HBase and Hive called etl
  • admin1 grants all privileges to the above namespaces to principal cdap
  • admin1 creates a CDAP namespace etl using the namespaces from HDFS, HBase and Hive.
  • admin1 grants all privileges on the CDAP namespace etl, and all entities under it to group etl-group
  • etl-user1 belonging to group etl-group deploys app feed1 in namespace etl
  • During app feed1 configure, dataset gold is created with owner principal cdap
  • etl-user2 belonging to group elt-group, starts workflow in app feed1, that runs as principal cdap
  • During the workflow run, principal cdap reads/writes to dataset gold
  • etl-user3 belonging to group elt-group can list logs and metrics for workflow in app feed1
  • etl-user3 belonging to group elt-group can also read from dataset gold
  • analyst1 belonging to group analyst-group is given privilege to read from dataset gold

Design

CDAP Authorization Policy

Existing CDAP Authorization Policy

The existing CDAP Authorization policy has the following limitations:

  • Granular privileges

    • Cannot grant a privilege to a user to read only one dataset or one stream in a namespace.
    • Cannot grant a privilege to a user to deploy/create an application/artifact/dataset/stream without granting WRITE on the namespace.
    • Cannot grant a privilege to a user to start/stop a program without granting READ on the namespace.
  • Visibility
    • User who has a privilege on a program cannot see the program in the UI or CLI if the user does not have any privilege on the namespace. 
  • Inconsistency
    • To write to a dataset user needs to have WRITE privilege on the dataset, but to write to a stream user needs to have both WRITE on the the stream and READ on the namespace.
    • ADMIN on an entity allows the user to delete the entity, whereas ADMIN on an entity does not allow user to create it.
    • Dataset read needs namespace READ, but dataset write does not need namespace WRITE.
  • Redundancy
    • Dataset READ and stream READ are redundant because they need namespace READ permission to be useful, and once a user has namespace READ the user can read all datasets and streams in the namespace.
    • List and View operations are equivalent but are listed separately in documentation.

Overview of the Proposed Authorization Policy

The proposed CDAP Authorization policy can be defined by the following principles:

  1. Access

    • Access defines who can perform an action (READ, WRITE, EXECUTE, ADMIN) on an entity. 

    • Access is not enforced in a hierarchical manner in CDAP.

    • Privileges in the authorization provider can be set up in a hierarchical manner (for instance by using wildcard privileges - how will this work in Sentry(question)).

  2. Visibility

    • Visibility defines whether an entity is visible to a user or not.
    • If a user has any privilege on an entity, it is visible to the user.
    • Visibility is hierarchical and flows bottom-up i.e. if a user has any privilege on a program then the user will be able to see the application that contains the program and namespace that contains the application.
  3. Grant

    • Grant is defined as action of giving a privilege on an entity to a user.
    • None of READ, WRITE, EXECUTE, ADMIN defined in CDAP will allow granting of privileges.
    • Only the administrator of the authorization provider can grant privileges to any entity. CDAP will not auto-grant privileges to creators.
  4. Impersonation

    • Impersonation is defined as the ability to -
      • deploy applications whose programs will execute as another user.
      • create a namespace/dataset/stream with a owner principal
      • run explore query in an impersonated namespace
    • alice needs ADMIN privilege on principal bob to deploy an application that can impersonate bob.
      • All operation that happens on the application/program entities are authorized using principal alice
      • All operations done by the running program/query are authorized as principal bob
        • This includes running the configure method and creating datasets from the application.

Decouple entity existence from privilege

In addition, CDAP will now support creating privileges for entities that are yet to be created. This will allow admins to grant fine grained privileges on entities. For example, an admin can grant a user ADMIN on an application before the application is deployed. This will allow the user to deploy only this specific application without having any other access to the namespace.

Changes to the authorization matrix

Instance

ADMIN on an Instance allows user to create Namespaces in the instance. No other operations are defined as of now. Also Instance is not a part of privilege hierarchy.

Note: The privilege marked in bold are the new one which will be added in 4.3

Namespaces

OperationPrivileges Required (Existing)Privileges Required (Proposed)
CreateADMIN (on the CDAP instance)

ADMIN

UpdateADMIN (on the namespace) 
DeleteADMIN (on the namespace)ADMIN on the namespace, and all entities in the namespace
View/ListAny of READ, WRITE, EXECUTE, or ADMINAny privilege on the namespace or any of its descendants.
Get Namespace Meta Any privilege on the namespace or any of its descendants.


Artifacts

OperationPrivileges Required (Existing)Privileges Required (Proposed)
AddWRITE (on the namespace)ADMIN
Add a propertyADMIN (on namespace) | ADMIN (on artifact)ADMIN
Remove a propertyADMIN (on namespace) | ADMIN (on artifact)ADMIN
Use to deploy an app ADMIN | READ | WRITE | EXECUTE
DeleteADMIN (on namespace) | ADMIN (on artifact)ADMIN
View/ListAny of READ, WRITE, EXECUTE, or ADMIN (on namespace) | Any of READ, WRITE, EXECUTE, or ADMIN (on artifact)Any privilege on the artifact
Get artifact info/summary/detail ADMIN | READ | WRITE | EXECUTE

 

Applications

OperationPrivileges Required (Existing)Privileges Required (Proposed)
AddWRITE (on the namespace) and READ (on the artifact if deployed from an artifact)

ADMIN

*Also see artifact privileges and principal privileges

DeleteADMIN (on the application) | ADMIN (on the namespace) ADMIN
View/ListAny of READ, WRITE, EXECUTE, or ADMIN (on namespace) | Any of READ, WRITE, EXECUTE, or ADMIN (on application)Any privilege on the application or any of its descendants.
Get application detail Any privilege on the application or any of its descendants.

 

Programs

OperationPrivileges Required (Existing)Privileges Required (Proposed)
Start, Stop, or Debug(EXECUTE (on the program) | EXECUTE (on the application) | EXECUTE (on the namespace)) & READ (on the namespace)EXECUTE
Set instancesADMIN (on the namespace) | ADMIN (on the application) | ADMIN (on the program)ADMIN
Set runtime argumentsADMIN (on the namespace) | ADMIN (on the application) | ADMIN (on the program)
ADMIN
Retrieve runtime argumentsREAD (on the namespace) | READ (on the application) | READ (on the program)
READ | EXECUTE | ADMIN
Retrieve statusAny of READ, WRITE, EXECUTE, or ADMIN 
View/ListAny of READ, WRITE, EXECUTE, or ADMIN 
Get program specification READ | WRITE | EXECUTE | ADMIN
Resume/Suspend schedule EXECUTE

 

Datasets

OperationPrivileges Required (Existing)Privileges Required (Proposed)
CreateWRITE (on the namespace)ADMIN
Read(READ (on the dataset) and READ (namespace)) | READ (on the namespace)READ
Retrieving propertiesNot DocumentedAny of READWRITEADMIN, or EXECUTE
WriteWRITE (on the dataset) | WRITE (on the namespace)WRITE
Update(ADMIN (on the dataset) and READ (on the namespace)) | (ADMIN (on the namespace) and READ (on the namespace))ADMIN
UpgradeADMIN (on the dataset) | ADMIN (on the namespace)ADMIN
TruncateADMIN (on the dataset) | ADMIN (on the namespace)ADMIN
DropADMIN (on the dataset) | ADMIN (on the namespace)ADMIN
View/ListAny of READ, WRITE, EXECUTE, or ADMIN 
Get dataset meta READ | WRITE | EXECUTE | ADMIN

 

Dataset Modules

OperationPrivileges Required (Existing)Privileges Required (Proposed)
DeployWRITE (on the namespace)ADMIN
DeleteADMIN (on the dataset module) | ADMIN (on the namespace)ADMIN
Delete-all in the namespaceADMIN (on the namespace)ADMIN on all dataset modules in the namespace
View/ListAny of READ, WRITE, EXECUTE, or ADMIN 
Get module meta READ | WRITE | EXECUTE | ADMIN

 

Dataset Types

OperationPrivileges Required (Existing)Privileges Required (Proposed)
View/ListAny of READ, WRITE, EXECUTE, or ADMIN 
Get dataset type meta READ | WRITE | EXECUTE | ADMIN

 

Secure Keys

OperationPrivileges Required (Existing)Privileges Required (Proposed)
CreateWRITE (on the namespace)ADMIN
DeleteADMIN (on the key) | ADMIN (on the namespace)ADMIN
View/ListAny of READ, WRITE, EXECUTE, or ADMIN 
ReadNot DocumentedREAD (on the key)

 

Streams

OperationPrivileges Required (Existing)Privileges Required (Proposed)
CreateWRITE (on the namespace)ADMIN
Retrieving eventsREAD (on the stream) & READ (on the namespace)READ
Retrieving propertiesAny of READWRITEADMIN, or EXECUTE 
Sending events to a stream (sync, async, or batch)(WRITE (on the stream) and READ (on the namespace)) | WRITE (on namespace & READ (on the namespace))WRITE
DropADMIN (on stream) | ADMIN (on namespace)ADMIN
Drop-all in the namespaceADMIN (on the namespace) | ADMIN (on the stream)ADMIN on all the streams in the namespace
UpdateADMIN (on the namespace) | ADMIN (on the stream)ADMIN
TruncateADMIN (on the namespace) | ADMIN (on the stream)ADMIN
View/ListAny of READ, WRITE, EXECUTE, or ADMIN 
Get stream property READ | WRITE | EXECUTE | ADMIN

 

Principal

OperationPrivileges Required (Existing)Privileges Required (Proposed)
Deploy an app to impersonate a principal ADMIN
Create a namespace with owner prinicpal ADMIN
Create a dataset with owner prinicpal ADMIN
Create a stream with owner prinicpal ADMIN

Open Questions

  1. How does authorization on CDAP system actions (like increasing instances of metrics processor, etc) happen?

CDAP Sentry Extension Improvements

Existing Model

CDAP allows privileges to be defined using entities and users. Sentry only allows privileges to be defined using roles and groups. CDAP is not aware of roles and groups, hence every grant made on entity and user has to be translated into a grant on roles and group.

For this translation, CDAP does the following

  1. Creates a proxy role per user and entity. This can lead to e x u roles being created, where e is the number of entities and u is number of users.
  2. Expects every user (say alice) to belong to a unique group in Hadoop User/Group mapping (group alice). Today this group name is expected to be the same as the username, and privileges for a user will be granted to that group. However, a user belonging to a group named same as the username is not true in all environments. This makes the privileges granted to the user ineffective during enforcement, and the user will not be able to access entities using these privileges.

In addition, revoking all privileges on an entity is expensive since it involves listing of all privileges for all users. This is because Sentry does not have an API to list all privileges for an entity. 

Proposed Model

Allow admins to use existing roles and groups in Sentry for authorization in CDAP. This means CDAP will not grant/revoke any privileges for entities. (note: this is a stretch goal for 4.3)

However in cases where an admin wants CDAP to grant privileges we propose the following model:

  1. Create a proxy role per user to which CDAP will grant privileges for all entities associated with the user. This limits the  number of roles created by CDAP in Sentry to u, where u is the number of users.
  2. Create a proxy group per user to which CDAP will grant the privileges. This removes the restriction of expecting a group with the same name as the username to be present, and will work in all environments. The proxy group so created will not be added to Hadoop User/Group mapping, and will only be part of Sentry privileges.

Investigate the new Sentry API (listPrivilegsbyAuthorizable) to list all privileges for a given entity so that we can avoid listing all privileges for all users during an entity deletion.

Backwards Compatibility

The above changes will be backward compatible with existing privileges.

  • Grant: All new grants will happen in the new format. 
  • Revoke: Revoke will happen in both old and new format.
  • Enforce: Enforce will work with both old and new privileges.
  • List: List will list both old and new privileges.

Reduce CDAP Startup Time Due to Authorization

Problem

We have observed that as the number of entities in CDAP grow, CDAP startup time increases due to authorization (more than 20 mins in some cases). During CDAP startup, CDAP revokes and grant privileges on all system entities. Revoking all privileges on an entity is expensive since it requires listing all privileges for all users.

Proposed Solution

  • cdap's access to system entities will bypass authorization (https://issues.cask.co/browse/CDAP-11659)
  • AuthorizationEnforcer will always return true if requesting user is cdap and namespace is system.
  • Authorizer grant/revoke will be no-op for the above case

Note: The underlying systems are still required to have appropriate permissions for cdap.

Use Existing Roles/Groups for Authorization

Currently, CDAP always grants/revokes privileges on an entity creation/deletion. Although this is a convenient feature, it does not work well in enterprise environments. Many enterprises prefer to manage privileges in a centralized authorization provider (like Sentry or Ranger). This will allow them to use existing role/groups to manage the privileges across all systems.

  • To support this we will introduce a property in cdap-site.xml which will specify whether CDAP should automatically grant privileges on entity creation. By default CDAP will continue granting privileges on entity creation to maintain backwards compatibility.
  • If an admin disables this feature, CDAP will not grant/revoke privileges on an entity automatically. In this case the admin is responsible for creating the appropriate privileges.
  • Not all authorization providers (like Sentry) have tools to manage privileges. CDAP will have to provide tools for admins to manage privileges using Sentry (stretch goal for 4.3, in 4.3 cdap-cli will be modified to allow creating privileges for non-existing entities - as what user will cdap-cli grant these privileges?).

CDAP Ranger Integration

Please see Ranger Integration Design Document

CLI Impact or Changes

  • CLI will be modified to not check for entity existence while granting privileges.

Test Scenarios

Test IDTest DescriptionExpected Results
   

Future work

  • No labels

13 Comments

  1. The doc said "ADMIN on an Instance allows user to create Namespaces in the instance", but then in the namespaces matrix, to create namespace it needs WRITE on CDAP instance. Does ADMIN implies WRITE?

  2. In the namespaces matrix, someone who has WRITE on CDAP instance can only create but not delete namespace? Seems asymmetrical?

  3. In the namespaces matrix, granting privilege on namespace requires ADMIN on the namespace, then who can grant the ADMIN privilege first?

  4. In the application matrix, it said "READ (on the artifact): If application is being deployed from an existing artifact". An application is always deployed from an existing artifact. The convenient REST endpoint actually first deploy the artifact, then create the application from the artifact that just get deployed.

  5. For the "Retrieve runtime arguments" in the Program matrix, if a user has READ on program only (no READ for application nor namespace), does that mean the runtime arguments that the program gets will only contain the one set at the program scope? Just want to make sure it is consistent with what the REST API response (hence what the UI shows), as well as what the program execution actually gets.

  6. but dataset write does not need namespace WRITE.

    I don't think this is an inconsistency

  7. Access flows top-down i.e. if an user has READ on namespace it implies that the user has READ on all entities inside the namespace. 

    I thought we did not want that any more. Doesn't it mean that if Joe wants to deploy an app in namespace A, he needs WRITE on A, but that would imply that he has WRITE on all other entities in the namespace? In other words, you cannot grant granular access to an entity.

  8. DeleteADMIN (on the namespace)

    If a namespace is owned by user X, and it contains an app owned by user Y, does deleting the namespace require ADMIN on the app?

    The reason for asking is the scenario of app-level impersonation. In this scenario, a single namespace (that is owned by an unprivileged user) contains multiple apps, each owned by (and impersonating) a different user. Each app owns its data, and no other user than the app's owner has privileges on the data. 

    Is this scenario supported? It seems that it isn't: the owner of the namespace would be ADMIN for all apps (inherited from the namespace), and thus could grant any privileges to itself. 

     

  9.  READ (on namespace) | READ (on the artifact):

    This is implied, because READ on namespace flows down to the artifact. 

  10. A lot of the auth matrix can be simplified by changing all occurrences of <privilege>(namespace) | <same privilege>(entity) to simply say <privilege>(entity). This is because privileges flow down the hierarchy. 

     

  11. Any of READ, WRITE, EXECUTE, or ADMIN on the namespace or application or any of its descendants.

    This is the app's descendants, right?

  12. I think the "View" row can be removed from the matrix. It is always the same: You need any privilege (no matter which) on the entity, or on any of its descendants.

  13. In some cases, we are keeping the previous required privileges, alternative to the newly proposed ones. For example, to add an application:

    • WRITE (on the namespace) | ADMIN (namespace) | ADMIN (application)

    I assume that is for backward-compatibility. Two questions:

    • Are all of the new rules backward-compatible?
    • How do we deprecate a required privilege in order to remove it in the next major release?