Page tree
Skip to end of metadata
Go to start of metadata

  • No labels

1 Comment

  1. Mass ingestion is the ingestion or replication of large amounts of data for use or storage in a database or a repository or files. The database can be a data lake, a cloud repository, or a Hadoop cluster. In addition it also provides support incremental/delta changes between batch loads, propagate schema changes/drift to the targets with support for partitioning/bucketing the data based on incoming data sets column definitions. Mass Ingestion service to ingest data at scale from selected database and streaming data sources and transfer the data with low latency to selected messaging systems and cloud targets.

    Mass Ingestion provides the following ingestion solutions:

    • Mass Ingestion Databases: Propagates data from source objects in a database management system (DBMS) to multiple types of targets. A database ingestion

       task can transfer a point-in-time snapshot of all source data in a schema to a target in a batch operation. A database ingestion task can also incrementally propagate DML data changes in near real time from an Oracle or SQL Server source to a target on a continuous basis. If you select the combined initial and incremental load type, the database ingestion task performs an initial load and then automatically switches to incremental load processing of data changes.
    • Mass Ingestion Files: Transfers a large number of files of different types between on-premises and cloud repositories. You can use Mass Ingestion Files to track and monitor file transfers.

    • Mass Ingestion Streaming: Transfers real-time streaming and machine data from selected sources to selected messaging systems and batch targets.

    It aimed read database schemas, tables, columns based on quick metadata load from source database then propagate the results quickly to target stores. Target stores cloud anything like Cloud Object stores like GCS, ADLS, S3 and could any Database/Datawarehouse targets.