Page tree
Skip to end of metadata
Go to start of metadata

Use Cases:

User has configured a pipeline with multiple plugins , User wants to see the plugins used in the app along with programs, streams and datasets used by the pipeline when getting application detail.

User has configured DBSource plugin with macros for the fields "connectionString",  "username" and uses macro function secure for the "password" field. UI wants to get all the configured macro properties for that plugin, in this case "connectionString" and "username", so its easy for user to provide the values for those macros as runtime arguments before starting the run.

User Story:


1) User wants to see list of plugins in the app when calling application detail REST endpoint.
2) For a plugin, user wants to get the details on plugin, including plugin properties and list of macros used in properties.

3) UI wants to get properties macro used by a plugin.

Design:


 
class PluginDetail {
	String pluginName;
	String pluginType;
}
 
Example : PluginDetail {pluginName : "source1", pluginType: "DBSource"}

 

PluginDetail for plugins will be added to co.cask.cdap.proto.ApplicationDetail

public class ApplicationDetail {
  // existing
  private final String name;
  private final String version;
  private final String description;
  private final String configuration;
  private final List<StreamDetail> streams;
  private final List<DatasetDetail> datasets;
  private final List<ProgramRecord> programs;
  private final ArtifactSummary artifact;
  // new field
  private final List<PluginDetail> plugins;
}


REST Endpoint :

GET: /v3/namespaces/{namespace-id}/apps/{app-id}/plugins/
 

this will return list of Plugin object's used by the app.

 

 

// No changes to existing Plugin class in CDAP-API.
public final class Plugin {
  private final ArtifactId artifactId;
  private final PluginClass pluginClass;
  private final PluginProperties properties;
  	Plugin(ArtifactId artifactId, PluginClass pluginClass, PluginProperties) {
	...
  }
  ...
}
/**
 * Plugin instance properties.
 */
@Beta
public class PluginProperties implements Serializable {

  private static final long serialVersionUID = -7396484717511717753L;

  // Currently only support String->String map.
  private final Map<String, String> properties;
  // new addition in PluginProperties to add macro
  private final Set<String> macros;

  public static Builder builder() {
    return new Builder();
  }

  private PluginProperties(Map<String, String> properties) {
    this(properties, null);
  }
 
   private PluginProperties(Map<String, String> properties, @Nullable Set<String> macros) {
    this.properties = properties;
    this.macros = macros;
  }
  
  /**
   * return the set of macros used by the plugin
   */ 
  @Nullable 
  public Set<String> getMacros() {
   return macros;
  }
  
  /**
   * internal method used by platform to set macros used in properties. its not advisable for users to use this. 
   * returns new PluginProperties instance with macros set
   */  
  public PluginProperties setMacros(Set<String> macros) {
    return new PluginProperties(getProperties(), macros);
  }
 
  // no changes to existing methods or Builder
  public Map<String, String> getProperties() {
    return properties;
  }

  /**
   * A builder to create {@link PluginProperties} instance.
   */
  public static final class Builder {

    private final Map<String, String> properties;

    private Builder() {
      this.properties = new HashMap<>();
    }

    /**
     * Adds multiple properties.
     *
     * @param properties map of properties to add.
     * @return this builder
     */
    public Builder addAll(Map<String, String> properties) {
      this.properties.putAll(properties);
      return this;
    }

    /**
     * Adds a property
     * @param key the name of the property
     * @param value the value of the property
     * @return this builder
     */
    public Builder add(String key, String value) {
      this.properties.put(key, value);
      return this;
    }

    /**
     * Creates a new instance of {@link PluginProperties} with the properties added to this builder prior to this call.
     */
    public PluginProperties build() {
      return new PluginProperties(Collections.unmodifiableMap(new HashMap<>(properties)));
    }
  }
}

 

 

  • No labels

20 Comments

  1. why is the user using a REST endpoint? Wouldn't he expect a UI for this?

    1. he will expect a UI for this, but if he wants to do it manually via REST we support that too. also UI is a user of REST endpoint i guess. 

      1. please update the use case not to say that the user expects a REST endpoint. REST is our way of implementing the use case, UI is another to implement it. What is the scope for 3.5? Only REST or also UI?

  2. I don't see how the plugin name is unique in a pipeline. I can use the same plugin on several places. Perhaps it should be the stage name, not the plugin?

    1. i will use plugin-id as the parameter. as that's agnostic of stages (hydrator-specific) and used in our ApplicationSpecification API.

      https://github.com/caskdata/cdap/blob/develop/cdap-api/src/main/java/co/cask/cdap/api/app/ApplicationSpecification.java#L127

      1. I am still not sure I get it. If I have a pipeline that uses a JDBC plugin in two different places, say, once for reading and once for writing. Then that are two instances of the same plugin, but with different properties. How can it be sufficient to identify a plugin by its plugin id? 

        1. so those two jdbc plugin will use different stage name, the stage name is the plugin-id. 

          if jdbc-source and jdbc-sink are the stage-name they would be pluginId's representing corresponding stages.  

          1. Oh my. That is what I first said in this comment: "... stage name ...". Ok that makes sense then. I do want to question why that is called the plugin id? That seems a misnomer. 

              1. The correct term is plugin id. This is the API in PluginConfigurer:

                <T> T usePlugin(String pluginType, String pluginName, String pluginId, PluginProperties properties);

                This is the API in PluginContext:

                <T> T newPluginInstance(String pluginId) throws InstantiationException;

                A plugin id is attached to an application. A plugin name is attached to the actual plugin class. In Hydrator, the stage name is always used as the plugin id because Hydrator enforces that stage names are unique.

  3. I would not call the class "Plugin", it is more a PluginInfo, right?

    1. Plugin is an existing class. while only the set of macros is the new addition in that class. 

      1. Oh really. Another misnomer. 

        1. Hi Andreas, I was thinking that we can add an internal PluginInfo class that will be returned as response to get plugin REST endpoint. by this way we don't have to change the existing Plugin class as that's in cdap-api.  I have updated the document with this change. 

  4. Better have an endpoint to retrieve all plugins of an app at once. Then the UI only needs to make a single request.

  5. I think instead of addMacro() it should be a setMacros(Set<String> macros), and that method should return a new instance of PluginProperties. This will keep the class immutable. 

  6. Also, getMacros() should be @Nullable because it returns null if setMacros() has not been called.

  7. other than that, looks good now. 

  8. oh, can the endpoint return a list? Or does it have to be a Map? How else do you represent the plugin id?

    1. Map makes more sense as we can identify by plugin Id, else if we want to return list to maintain consistency with other endpoints, we have to add a class including plugin-id and plugin and return a list of that class. let me know which is better.