Spark Eclipse mave开发环境（Windows）

1、下载Eclipse
2、配置本地Maven到Eclipse中
1）、配置Maven的安装目录到Eclipse上
在这一步中，我们要配置本地Maven到Eclipse中
步骤：window–>preferences—>maven—>Installations—>add

2）、配置Maven的settings.xml到Eclipse上
步骤：window–>preferences—>maven—>User Settings

3、步骤：window–>preferences—>maven—>Archetype http://repo1.maven.org/maven2/archetype-catalog.xml（右键另存为）

4、创建一个scala的maven项目

5、修改jdk为1.8，scala为2.11.11

6、修改pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>dblab</groupId>
  <artifactId>WordCount</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <inceptionYear>2008</inceptionYear>
  <properties>
    <scala.version>2.11.11</scala.version>
    <spark.version>2.3.3</spark.version>
  </properties>
   <repositories>
    <repository>
      <id>scala-tools.org</id>
      <name>Scala-Tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </repository>
  </repositories>     
  <pluginRepositories>
    <pluginRepository>
      <id>scala-tools.org</id>
      <name>Scala-Tools Maven2 Repository</name>
      <url>http://download.eclipse.org/releases/indigo/</url>
    </pluginRepository>
  </pluginRepositories>
   <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>${spark.version}</version>
</dependency>
<dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>
  </dependencies>
 <build>
        <sourceDirectory>src</sourceDirectory>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.3.1</version>
            </plugin>
        </plugins>
    </build>
</project>

（2）如果第一套无法在Eclipse里运行代码，请使用第二套（运行代码时打开注释）

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>dblab</groupId>
  <artifactId>WordCount</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <inceptionYear>2008</inceptionYear>
  <properties>
    <scala.version>2.11.11</scala.version>
    <spark.version>2.3.3</spark.version>
  </properties>
  
   <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>${spark.version}</version>
</dependency>
<dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>
  </dependencies>
 <!--
 <build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <testSourceDirectory>src/test/scala</testSourceDirectory>
    <plugins>
      <plugin>
        <groupId>org.scala-tools</groupId>
        <artifactId>maven-scala-plugin</artifactId>
        <version>2.15.0</version>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
            <configuration>
              <args>

                <arg>-dependencyfile</arg>
                <arg>${project.build.directory}/.scala_dependencies</arg>
              </args>
            </configuration>
          </execution>
        </executions>
      </plugin>
      
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.6</version>
        <configuration>
          <useFile>false</useFile>
          <disableXmlReport>true</disableXmlReport>
          <includes>
            <include>**/*Test.*</include>
            <include>**/*Suite.*</include>
          </includes>
        </configuration>
      </plugin>
    </plugins>
  </build>
-->
</project>

7、清空项目，新建WordCount.scala

package huixing.wordcount

import org.apache.spark.SparkContext
 import org.apache.spark.SparkContext._
 import org.apache.spark.SparkConf
 
object WordCount {
    def main(args: Array[String]) {
        val inputFile =  "F:\\sparkdemo.txt"
        val conf = new SparkConf().setAppName("WordCount").setMaster("local")
        val sc = new SparkContext(conf)
                val textFile = sc.textFile(inputFile)
                val wordCount = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)
                wordCount.foreach(println)       
    }
}

8、运行程序测试：

9、导成jar包:
Run as->maven install

10、运行jar包：
条件：1)、配置环境变量：


2)、解决：
Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executablenull\bin\winutils.exe in the Hadoop binaries.
下载hadoop-common-2.2.0-bin-master.zip将bin目录下文件解压到$HADOOP_HOME\bin

11、windows shell
1）切到spark目录f: cd spark-2.3.3-bin-hadoop2.7
2)bin\spark-submit –class huixing.wordcount.WordCount D:\maven\mvnRespo\huixing\wordcount\0.0.1-SNAPSHOT\wordcount-0.0.1-SNAPSHOT.jar

注：

解决办法 : 删除文件即可。.metadata/.plugins/org.eclipse.e4.workbench/workbench.xmi