{"id":2019,"date":"2024-01-12T13:15:21","date_gmt":"2024-01-12T04:15:21","guid":{"rendered":"https:\/\/www.kwonline.org\/memo2\/?p=2019"},"modified":"2024-01-12T13:29:13","modified_gmt":"2024-01-12T04:29:13","slug":"run-spark-on-ubuntu-22_04","status":"publish","type":"post","link":"https:\/\/www.kwonline.org\/memo2\/2024\/01\/12\/run-spark-on-ubuntu-22_04\/","title":{"rendered":"Ubuntu 22.04: Spark \u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb"},"content":{"rendered":"<p>&nbsp;<br \/>\nSpark \u4f7f\u3046\u306e\u3067\u30e1\u30e2\u3002<\/p>\n<p>\u6bce\u5ea6\u4ffa\u3057\u304b\u4f7f\u308f\u306a\u3044\u306e\u3067\u30db\u30fc\u30e0\u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u306b\u5165\u308c\u308b\u3002<br \/>\n\u3068\u308a\u3042\u3048\u305a Hadoop \u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u3066\u304a\u304f\u3002<\/p>\n<p>\u53c2\u8003: <a href=\"https:\/\/www.kwonline.org\/memo2\/2024\/01\/11\/run-hadoop-on-ubuntu-22_04\/\" rel=\"noopener\" target=\"_blank\">Ubuntu 22.04: Hadoop \u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/a><\/p>\n<h2>1. \u30c0\u30a6\u30f3\u30ed\u30fc\u30c9<\/h2>\n<p>Apache \u516c\u5f0f\u30b5\u30a4\u30c8\u3067\u30d0\u30fc\u30b8\u30e7\u30f3\u3092\u9078\u3093\u3067\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3059\u308b\u3002<\/p>\n<p><a href=\"https:\/\/spark.apache.org\/downloads.html\" rel=\"noopener\" target=\"_blank\">Downloads | Apache Spark<\/a><\/p>\n<p>spark-3.4.2-bin-hadoop3.tgz \u3092\u9078\u3093\u3060\u3002<\/p>\n<h2>2. \u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/h2>\n<p>\u7d9a\u3044\u3066\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\ntar -zxf spark-3.4.2-bin-hadoop3.tgz\r\nmv spark-3.4.2-bin-hadoop3 ~\/spark\r\n<\/pre>\n<p>.profile \u306b\u74b0\u5883\u5909\u6570\u3092\u8ffd\u52a0<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# vim ~\/.profile\r\nexport SPARK_HOME=$HOME\/spark\r\nexport PATH=$PATH:$SPARK_HOME\/bin:$SPARK_HOME\/sbin\r\nexport PYSPARK_PYTHON=\/usr\/bin\/python3\r\n<\/pre>\n<p>\u8ffd\u8a18\u3057\u305f\u3089 source \u3067\u8aad\u307f\u8fbc\u3080<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\nsource ~\/.profile\r\n<\/pre>\n<h2>3. \u8d77\u52d5<\/h2>\n<p>Master \u3068 Worker \u3092\u8d77\u52d5\u3059\u308b\u3002<\/p>\n<p>start-master.sh<br \/>\nstart-worker.sh spark:\/\/localhost:7077<\/p>\n<p>\u3053\u308c\u3067\u8d77\u52d5\u5b8c\u4e86\u3002<br \/>\n\u3042\u3068\u306f\u3001 spark-shell \u307e\u305f\u306f pyspark \u30b3\u30de\u30f3\u30c9\u3067\u5bfe\u8a71\u30b7\u30a7\u30eb\u3092\u8d77\u52d5\u3057\u3066\u304a\u624b\u8efd\u306b\u4f7f\u3046\u3002<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; Spark \u4f7f\u3046\u306e\u3067\u30e1\u30e2\u3002 \u6bce\u5ea6\u4ffa\u3057\u304b\u4f7f\u308f\u306a\u3044\u306e\u3067\u30db\u30fc\u30e0\u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u306b\u5165\u308c\u308b\u3002 \u3068\u308a\u3042\u3048\u305a Hadoop \u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u3066\u304a\u304f\u3002 \u53c2\u8003: Ubuntu 22.04: Hadoop \u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb 1. \u30c0 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,8,10],"tags":[],"class_list":["post-2019","post","type-post","status-publish","format-standard","hentry","category-data-engineering","category-linux","category-spark"],"_links":{"self":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/2019","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/comments?post=2019"}],"version-history":[{"count":5,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/2019\/revisions"}],"predecessor-version":[{"id":2024,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/2019\/revisions\/2024"}],"wp:attachment":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/media?parent=2019"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/categories?post=2019"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/tags?post=2019"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}