Spark3读S3上文件

一、启动Spark-shell

• 本地启动spark-shell

./spark-shell

二、读文件

val rdd =spark.sparkContext.textFile(“s3a://opticsprod/dwh/ods_prod/software_1h/datetime=2020070919/test.parquet”);

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0
      /_/

Using Scala version 2.12.10 (Java HotSpot(TM