Mass ingestion is the ingestion or replication of large amounts of data for use or storage in a database or a repository or files. The database can be a data lake, a cloud repository, or a Hadoop cluster. In addition it also provides support incremental/delta changes between batch loads, propagate schema changes/drift to the targets with support for partitioning/bucketing the data based on incoming data sets column definitions. Mass Ingestion service to ingest data at scale from selected database and streaming data sources and transfer the data with low latency to selected messaging systems and cloud targets.
Mass Ingestion provides the following ingestion solutions:
Mass Ingestion Databases: Propagates data from source objects in a database management system (DBMS) to multiple types of targets. A database ingestion
Mass Ingestion Files: Transfers a large number of files of different types between on-premises and cloud repositories. You can use Mass Ingestion Files to track and monitor file transfers.
Mass Ingestion Streaming: Transfers real-time streaming and machine data from selected sources to selected messaging systems and batch targets.
It aimed read database schemas, tables, columns based on quick metadata load from source database then propagate the results quickly to target stores. Target stores cloud anything like Cloud Object stores like GCS, ADLS, S3 and could any Database/Datawarehouse targets.
Powered by a free Atlassian Confluence Open Source Project License granted to Coopr. Evaluate Confluence today.