Overview
Spark ClickHouse Connector is a high performance connector build on top of Spark DataSource V2.
Requirements
- Basic knowledge of Apache Spark and ClickHouse.
- An available ClickHouse single node or cluster.
- An available Spark cluster, check the following Compatible Matrix to make sure the Spark version is compatible with this Connector.
- Make sure your network policy satisfies the following requirements, both driver and executor of Spark need to access ClickHouse HTTP port. If you are using it to access ClickHouse cluster, ensure the connectivity between driver and executor of Spark and each node of ClickHouse cluster.
Notes
- Integration tests based on Java 8 & 17, Scala 2.12 & 2.13, Spark 3.4 and ClickHouse v23.8, with both single ClickHouse instance and ClickHouse cluster.
Compatible Matrix
For old versions, please refer the compatible matrix.
| Version | Compatible Spark Versions | ClickHouse JDBC version |
|---|---|---|
| master | Spark 3.3, 3.4, 3.5 | 0.6.0 |
| 0.7.3 | Spark 3.3, 3.4 | 0.4.6 |
| 0.6.0 | Spark 3.3 | 0.3.2-patch11 |
| 0.5.0 | Spark 3.2, 3.3 | 0.3.2-patch11 |
| 0.4.0 | Spark 3.2, 3.3 | Not depend on |
| 0.3.0 | Spark 3.2, 3.3 | Not depend on |
| 0.2.1 | Spark 3.2 | Not depend on |
| 0.1.2 | Spark 3.2 | Not depend on |