- User Stories Documented
- User Stories Reviewed
- Design Reviewed
- APIs reviewed
- Release priorities assigned
- Test cases reviewed
- Blog post
The current scheduler has limited trigger types and constraints. We wish to add more configuration by chaining pipelines and programs based on the states of programs, as well as impose more scheduler constraints.
- Triggers: Other programs or pipelines should be started based on the program status of another program or pipeline.
- Constraints: Additional scheduling constraints such as a specific time window, when a program has generated a specific amount of data, and number of concurrent runs, should be added.
Note: The "program" denoted below in the user stories could be a pipeline or any custom program. See the Assumptions section below for more details.
- As a user, I would like to be able to start program B if and only if program A successfully completed, and start program C if and only if program A failed.
- As a user, I would like to be able to start program B if and only if program A has failed, with a time window constraint for program B.
- As a user, I would like to be able to start program B if and only if program A successfully completed and dataset C has 100 MB of new data
- As a user, I would like to be able to start program B if and only if program A successfully completed and dataset C has new partitions.
- **As a user, I would like to start program B if program A resulted in a lot of errors. (Error metrics)
The "program" specified in the user stories can be a pipeline in the same namespace, or any custom program.
- We are assuming that that the program state based scheduling can take in all possible program states (Initialized, Running, Completed, Failed, Killed). The UI can later be configured to allow only the appropriate and meaningful program states.
- All combinations of constraints and triggers that are meaningful are possible, where the UI can be configured to restrict a combination as needed.
- Program State Based Scheduling will rely on the existing TMS system and follow a similar architecture to the existing Scheduler.
- UI can populate the program type (One of
- Constraints added to a ProgramStateTrigger will be applied to the current program, not the previous program (as seen in User Story #2).
TMS Message Design:
For a Program State message, the notification will contain:
New Programmatic APIs
New Java APIs introduced (both user facing and internal)
Deprecated Programmatic APIs
New REST APIs
|New or Update Existing API Method||Path||Method||Description||Request Body||Response Code||Response|
To add a schedule for a program to an application
200 - On success
404 - When application is not available
409 - Schedule with the same name already exists
500 - Any internal errors
Deprecated REST API
|/v3/apps/<app-id>||GET||Returns the application spec for a given application|
CLI Impact or Changes
- Add CLI handlers for adding new ProgramSchedule
- Impact #2
- Impact #3
UI Impact or Changes
- Add ProgramSchedule options to the "Schedule" panel, along with dropdowns for pipeline selection and program status selection
- Impact #2
- Impact #3
What's the impact on Authorization and how does the design take care of this aspect
Impact on Infrastructure Outages
System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspect
|Test ID||Test Description||Expected Results|
|1||Configure a schedule to run every 5 minutes and always complete successfully. Configure a second schedule to run if the first schedule has succeeded.||Both schedules run and finish successfully.|
|2||Configure a schedule to fail. Configure a second schedule to run only if the first schedule has succeeded.||The first schedule should run. The second schedule should not run.|
|3||Configure a schedule to run for 10 seconds before it completes. Configure a second schedule to run only if the first schedule has succeeded. 500 milliseconds before the first schedule completes, delete the second schedule.||The first schedule should run and finish successfully.|
- User-facing class to encapsulate a program? Rather than passing in namespace, application, applicationVersion, and programType, and program all to encapsulate a program?
- More granularity with program states? Programs have different states based on the program type. Not directly related as ProgramSchedulerStatus encompasses general states of all programs which is the primary use case, but accurate and detailed program states may still be something to consider.
Name topic in TMS for ProgramSchedulerStates
Send program state notifications through TMS
- At central location for all programs for all ProgramSchedulerStates
- API Specification
- Parse new ProgramStateTrigger
- Subscribe to ProgramStateNotifications from TMS and create schedule
- Add to ProgramScheduleStore and ensure atomicity of events
- Work #1
- Work #2
- Work #3