1. 安裝Scala, apache-spark, Hadoop

brew install scala
brew install apache-spark
brew install hadoop
pip install pyspark

注意不要自己另外安裝jdk, 會造成版本對不上報錯。因為安裝apache-spark的過程中會自動安裝openjdk。

2. 配置環境變量

JAVA_HOME=/opt/homebrew/Cellar/openjdk@11/11.0.26/libexec/openjdk.jdk/Contents/Home
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=$JAVA_HOME/lib
export JAVA_HOMESCALA_HOME=/opt/homebrew/opt/scala
export PATH=$PATH:$SCALA_HOME/binexport SPARK_PATH=/opt/homebrew/Cellar/apache-spark/3.5.4
export PATH=$PATH:$SPARK_PATH/binexport openjdk_home=/opt/homebrew/Cellar/openjdk@17/17.0.14HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.1
export PATH=$PATH:/$HADOOP_HOME/bin

注意jdk版本不對會造成Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed

3. 下載jdbc driver
   鏈接, 選擇 Platform Independent

將下載的解壓包中的jar文件拷貝到apache-spark的目錄下: /opt/homebrew/Cellar/apache-spark/3.5.4/libexec/jars

4. 使用如下代碼來測試

from pyspark.sql import SparkSession
from pyspark import SparkContext# sc = SparkContext.getOrCreate()sparkdriver = SparkSession.builder.\appName('demo').\master('local[*]').\config('spark.driver.extraClassPath', '/opt/homebrew/Cellar/apache-spark/3.5.4/libexec/jars/mysql-connector-j-9.2.0.jar').\getOrCreate()df_mysql = sparkdriver.read.format('jdbc').\option('url', 'jdbc:mysql://localhost:3306').\option('driver', 'com.mysql.jdbc.Driver').\option('user', 'root').\option('password', '123').\option('query', 'select * from tablename').\load()df_mysql.show(10)

Connect MySQL with PySpark using JDBC Connector (Mac 2022)