Amazon Redshift now supports streaming ingestion from Confluent Managed Cloud and self-managed Apache Kafka clusters on Amazon EC2 instances, expanding its capabilities beyond Amazon Kinesis Data Streams (KDS) and Amazon Managed Streaming for Apache Kafka (MSK).
With this update, customers can ingest data from a wider range of streaming sources directly into their Amazon Redshift data warehouses. Amazon Redshift introduces mTLS (mutual Transport Layer Security) as the authentication protocol for secure communication between Amazon Redshift and the newly supported Kafka streaming sources. This ensures that data ingestion from these new sources maintains the high security standards expected in enterprise data workflows. Additionally, a new SQL identifier ‘KAFKA’ has been introduced to simplify the identification of these newly supported Kafka sources in Amazon Redshift External Schema definitions.
You can start using this expanded streaming ingestion capability immediately, to build more comprehensive and flexible data pipelines that ingest data from various Kafka sources — those offered by AWS (Amazon MSK), those available from partners (Confluent Cloud) or those that are self-managed (Apache Kafka) on Amazon EC2.
To learn more and get started with streaming data into Amazon Redshift from any Kafka source, refer to the Amazon Redshift streaming documentation.
Source:: Amazon AWS