[정보보호 R&D 2019] Pyspark 설치

1. 자바 설치 

- https://java.com/ko/download/ 

2. Spark 설치 

- http://spark.apache.org/downloads.html

3. winutils.exe 설치 

- http://github.com/steveloughran/winutils

4. Pyspark 설치

- python -m pip install findspark

5. 설치 확인 

import findspark

findspark.init()

import pyspark

from pyspark.sql import SparkSession

spark = SparkSession.buider.getOr.Create()

df = spark.sql("select 'spark' as hello") 

df.show() 

만약, 위와 같은 과정을 거쳐도 설치가 되지 않는다면 경로의 문제일 가능성이 높음. Java 및 Spark의 경로가 맞게 설정되었는지 확인해야 함. 

확인코드 

import os 

print(os.environ['SPARK_HOME'])

print(os.environ['JAVA_HOME'])

print(os.environ['PATH'])

환경변수 

- SPARK_HOME = C:\spark\spark-2.3.2-bin-hadoop2.7

- HADOOP_HOME = C:\spark\spark-2.3.2-bin-hadoop2.7

- JAVA_HOME = C:\Program Files\Java\jdk1.8.0_201

경로변수 

- C:\spark\spark-2.3.2-bin-hadoop2.7\bin

- C:\Program Files\Java\jdk1.8.0_201\bin

참고자료 

- https://medium.com/@naomi.fridman/install-pyspark-to-run-on-jupyter-notebook-on-windows-4ec2009de21f

- https://bigdata-madesimple.com/guide-to-install-spark-and-use-pyspark-from-jupyter-in-windows/