Page tree
Skip to end of metadata
Go to start of metadata

Goals

  • The market will provide an application store for CDAP users to quickly and efficiently add capabilities into the platform.   
  • The Cask Market will be a read only for 4.0 and include provisions for working offline.   

Checklist

  • User stories documented(Todd)
  • Requirements documented(Todd)
  • Requirements Reviewed
  • Mockups Built
  • Design Built
  • Design Accepted

User Stories

  • As a CDAP User I want a plus button on the navbar so that I can an add new capabilities from anywhere in the platform
  • As a CDAP User I want to be able to add new artifacts from the market into the system to speed up development and improve the overall developer experience.
  • As a Hydrator User I want to be able to add plugins from the market to expand the capabilities of my pipeline.  
  • As a Hydrator User from within the Hydrator Studio I want to be able to add new pipelines or applications from within Hydrator, and upon adding, remain in the Hydrator context.
  • As a CDAP user I want to be able to add examples from the market.  
  • As a CDAP user from within the market I want to be able to know what I've added into the platform and have those special cased when browsing the market.  
  • AS a CDAP user I want to have access to the market in offline mode, so that I can add capabilities into the system from where-ever my cluster is deployed.  

Use Cases

Scenario 1
[1] I'm in hydrator studio, in CDAP 4.0
[2] I click the + button
[3] From the market I choose pipeline, and add "SFDC Lead dump" 
[4] The wizard walks me through the rep defined steps to add the pipeline
[5] I return to the hydrator context, and it's in the pipeline studio, AND NOT deployed as an app
Scenario 2
[1] In CDAP, and I click the + button
[2] From the market I choose pipeline, and add "SFDC Lead dump" 
[3] The wizard walks me through the repo defined steps to add the pipeline
[4] Hydrator is opened and it's in the pipeline studio, AND NOT deployed as an app
Scenario 3
[1] In CDAP, and I click the + button
[2] From the market I choose application, "Stock Market Trainer" 
[3] I will see the option "clone in hydrator".
[4] If I click "clone in hydrator" the wizard walks me through the repo defined steps to add the pipeline
[5] Hydrator is opened and it's in the pipeline studio, AND NOT deployed as an app
Scenario 4
[1] In CDAP, and I click the + button
[2] From the market I choose application, "Stock Market Trainer" 
[3] I will see the option "clone in hydrator".
[4] If I click "add", the wizard walks me through the repo defined steps to deploy the application
[5] The app will be deployed and I will see in detailed pipeline view.  
Scenario 5
[1] In CDAP, and I click the + button
[2] From the market I choose plugin, "SFDC Source" 
[3] The wizard walks me through the steps to add the plugin, including the option to have it added as plugin template.  
[4] The plugin is added as a resource, and the resource center shows this as included, with the specific entity version.
Scenario 6
[1] In hydrator studio, and I click the + button
[2] From the market I choose plugin, "SFDC Source" 
[3] The wizard walks me through the steps to add the plugin, including the option to have it added as plugin template.  
[4] The plugin is added and will show up on in the plugin library for immediate use
Scenario 7

[1] In hydrator studio, and I click the + button
[2] From the market I choose add a third party plugin, "MYSQL JDBC"
[3] If multiple versions are available, I should be able to choose which version I want from a dropdown.
[4] The wizard walks me through the steps to add the plugin.
[5] In the case of an external dependency, the license agreement and download should be incorporated into the wizard.
[6] The option to have it added as plugin template should be part of the wizard.
[7] The plugin is added and will show up on in the plugin library for immediate use.

Scenario 8

[1] In CDAP, and I click the + button
[2] From the market I choose add "Traffic Incidents"  mysql data 
[3] If multiple versions are available, I should be able to choose which version I want from a dropdown.
[4] The wizard walks me through the steps to add the datapack locally and into my development environment 

Scenario 8 Workflow Diagram

Requirements

Market Requirements

  • The market should be accessible from anywhere within CDAP.
  • The market should support adding:
    • Artifacts
      • Applications
      • Native Plugins
      • Third Party Plugins
    • Pipelines
    • DataPacks
  • The market should provide context when browsing to know if I've already added an entity.  I should still be able to create another instance and the market should keep track.  
  • The market should instantiate the new entity wizard where applicable.  
  • The market should be browsable independently of the product, from the website.  
  • The market on the website should link to localhost and the cdap port to indicate "open in cdap" to add entities.  
  • The market should be supported by default in both Distributed and Standalone, with the option to be disabled.  
  • The market should work in offline mode after syncing the repository onto the cluster.  
  • The market should be an independent entity outside of CDAP.
  • CDAP should track what has been added previously
  • The market should be read only for the first iteration.   

Market Repository Requirements

  • The repo should include metadata only.
  • The contract between the repo and the entities needs to be versioned to support changes in repo behavior or meta data specifications for entities.
  • The repo should be updatable/installable from the UI, REST API or CMD Line.  
  • The repo should include versioning information for entities.
  • The repo should include a checksum for entities.
  • The repo should include information for third party entities.
  • The repo should include author and licensing information for entities.
  • The repo should include dependency information for entities
  • The repo should include abstraction information (service descriptor) for how an entity will interact with the market wizard workflow.   The wizard will consist of 1 to N steps and must be able to abstract all information necessary from the entity descriptor for how an entity is added including dependencies (internal/external) precedence of steps.

Market Datapack Requirements

Datapacks are sources of data that will improve the bootstrapping and getting started experience on the platform.  

  • Datapacks should be able to be added for:
    • SQL datapacks 
      • MYSQL .sql files
      • SQLITE .sqlite
    • S3 datapacks
      • Text files
      • csv examples
    • HTTP (for hydrator realtime pipelines)
      • Endpoints for various supported web services
  • Datapacks should be available to be added as new sources plugin templates in hydrator.  
  • Datapacks should be able available for custom applications.  

 

  • No labels

23 Comments

  1. First user story: what does "capability" mean? Needs a more concrete definition. 

  2. Second user story. Are we talking a dataset type? Or an instance config for a "standard" (out-of-the-box in cdap-api) dataset type?

  3. 3rd user story: What kind of "artifacts"? Hydrator plugins? Or application artifacts? Is the intention to add reusable components/libaries (that would speed up development). Or is the intention to add apps or plugins that can be configured (but not used on app development)?

  4. 5th user story: Why would a Hydrator user add an application? Wouldn't he only be interested in hydrator plugins?

    1. I was thinking of using the context to present the right things. Which to me made sense, but I think some people didn't want that. It would address this requirement. 

  5. Scenario 4 [3]: Should it be "add application" instead "clone in hydrator"?

    1. Also, what would be an example for an app that a user would add without modification/configuration? Wouldn't she always have to adjust the source or sink configuration to her own environment? 

  6. Scenario 3: How can I clone an app in hydrator? Hydrator can only clone pipelines....

  7. All the scenarios 1-6 are about adding Hydrator pipelines or plugins. So is this going to be a Hydrator-only feature?

    1. That is not true. '+' is a generic way for adding Hydrator Apps + other apps. It's not Hydrator only feature. 

      1. I am just saying all scenarios documented here add hydrator pipelines or plugins. Not saying that the plus button cannot add custom apps. But a custom app cannot be cloned in Hydrator, and it cannot be configured through hydrator. We would need to add a studio-like screen for configuring a custom app (which we don't have today, we can only do that through CLI/REST). Perhaps this document needs some additional sections/scenarios.

        1. Andreas Neumann saying the requirement needs to be adjusted. U are correct (smile)

  8. Last requirement: How does it make sense to support this only in SDK? That means I can get started easily in a playground, but there no way at all to go to production. So what is the point?

    1. Don't know when this changed. But, when I had mocked it was for both Standalone and Distributed and the distributed had option to turn it off. 

      1. I updated the requirement to reflect this correctly.

  9. Main take-aways:

    • It is important to know what's in scope for 4.0 and what's not in scope.

    • We need more requirements around overall experience. Please see comments below

    Questions/Comments

    1. Is adding dataset in scope? If so the requirements are not flushed out
      > expand capabilities within the community by allowing users to publish capabilities for other CDAP users to experience.  
    2. Do we need to cover how to publish the artifacts/plugins/datasets to the repository. 
    3. What are the security requirements. Is the app store open for everyone to browse and add? 
      1. As a sub-section, if a user-a has added an applicationX in a namepsace, and did not give access to user-b. User-b while adding the same application will get an error. 
    4. What will be the offline experience? Say I am not connected to internet. How should CDAP app store functionality behave? 

      >The app store should be supported by default in both Distributed and Standalone, with the option to be disabled.  
    5. Why do need a disable option? Will restarting CDAP be acceptable for disabling?
    6. Does the repo needs to be configurable?

    The app store should have rules for contribution and allow user submissions.

          7. What's the experience for user-submission? 

    The app store should instantiate the new entity wizard where applicable.  

          8. What's the new entity wizard?

          9. What are the requirements that goes around the code/artifacts - documentation of how to use it, use-cases etc. 

     

    1. I have updated the requirements to address the discussion this morning.  

  10. Are there requirements around gathering download metrics if not now if the future? I think that the repo should have access logs or some other mechanism to collect information like: how many users downloaded the LogAnalytics app in 3.5

    1. As the repo will be a package in the first iteration, we will be able to track downloads from the source.  As we move towards a broader solution we will need a mechanism to track entities that are created.  

  11. What does this requirement mean?

    • The repo should be versioned, and be updatable/installable from the UI, REST API or CMD Line. 

     

    I would argue that the repo should NOT be updatable from any of those methods. Also not sure what it means for the repo to be versioned.

    1. Correct, this is badly worded.  I will update to reflect this is versioning the contract (as Albert Shau suggested) and what is included in the meta data for an entity so we can address

      • Changes in behavior of the repo.
      • Changing specifications for what is included in the meta data for an entity.
  12. Regarding scenario 8 and several of the requirements, I don't think this should be used to write data to external systems. Shouldn't it only interact with CDAP? I can see loading data into a stream because that is a CDAP operation, but writing to a mysql table or to s3 doesn't make sense to me. 

    1. Albert Shau Agree.  I will change the requirement.