{"id":2102,"date":"2024-02-19T13:43:52","date_gmt":"2024-02-19T04:43:52","guid":{"rendered":"https:\/\/www.kwonline.org\/memo2\/?p=2102"},"modified":"2024-02-19T13:43:52","modified_gmt":"2024-02-19T04:43:52","slug":"connect-from-apache-spark-to-adls2","status":"publish","type":"post","link":"https:\/\/www.kwonline.org\/memo2\/2024\/02\/19\/connect-from-apache-spark-to-adls2\/","title":{"rendered":"Spark \u304b\u3089ADLS2 \u306b\u30a2\u30af\u30bb\u30b9\u3059\u308b"},"content":{"rendered":"<p>&nbsp;<br \/>\nSpark \u304b\u3089 Azure Data Lake Storage Gen2 \u306e\u30d5\u30a1\u30a4\u30eb\u3092\u958b\u304d\u305f\u304b\u3063\u305f\u306e\u3067\u30e1\u30e2<\/p>\n<p>Pyspark \u5b9f\u884c\u6642\u306b\u4f9d\u5b58\u30e9\u30a4\u30d6\u30e9\u30ea\u3092\u6307\u5b9a\u3059\u308c\u3070\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u3066\u304f\u308c\u308b\u3002<br \/>\n\u4f7f\u3063\u3066\u308b Spark\/Hadoop \u306e\u30d0\u30fc\u30b8\u30e7\u30f3\u306b\u5408\u308f\u305b\u3066\u4e0b\u8a18\u3092\u5b9f\u884c\u3002<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\npyspark --packages org.apache.hadoop:hadoop-azure:3.3.4\r\n<\/pre>\n<p>\u5fc5\u8981\u306a jar \u30d5\u30a1\u30a4\u30eb\u306e\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u304c\u59cb\u307e\u308b\u306e\u3067\u3057\u3070\u3089\u304f\u5f85\u3063\u305f\u3089\u4f7f\u3048\u308b\u3002<\/p>\n<p>\u3067\u3001\u30b3\u30fc\u30c9\u3092\u5b9f\u884c<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\nfrom pyspark.sql import SparkSession\r\n\r\nspark = SparkSession.builder.appName(&quot;ReadFromADLS2&quot;) \\\r\n    .config(&quot;fs.azure.account.auth.type.{STORAGE-ACCOUNT}.dfs.core.windows.net&quot;, &quot;SharedKey&quot;) \\\r\n    .config(&quot;fs.azure.account.key.{STORAGE-ACCOUNT}.dfs.core.windows.net&quot;, &quot;{STORAGE-ACCOUNT-ACCESS-KEY}&quot;) \\\r\n    .getOrCreate()\r\n\r\ndf = spark.read.csv(&quot;abfss:\/\/{CONTAINER}@{STORAGE-ACCOUNT}.dfs.core.windows.net\/orders.csv&quot;)\r\ndf.show()\r\n<\/pre>\n<p>\u4e0a\u8a18\u306e <strong>{STORAGE-ACCOUNT}<\/strong>, <strong>{STORAGE-ACCOUNT-ACCESS-KEY}<\/strong>, <strong>{CONTAINER}<\/strong> \u306e\u7b87\u6240\u306f\u66f8\u304d\u63db\u3048\u308b\u3002<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; Spark \u304b\u3089 Azure Data Lake Storage Gen2 \u306e\u30d5\u30a1\u30a4\u30eb\u3092\u958b\u304d\u305f\u304b\u3063\u305f\u306e\u3067\u30e1\u30e2 Pyspark \u5b9f\u884c\u6642\u306b\u4f9d\u5b58\u30e9\u30a4\u30d6\u30e9\u30ea\u3092\u6307\u5b9a\u3059\u308c\u3070\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u3066\u304f\u308c\u308b\u3002 \u4f7f\u3063\u3066\u308b Spark [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,21,8,29,10],"tags":[],"class_list":["post-2102","post","type-post","status-publish","format-standard","hentry","category-azure","category-data-engineering","category-linux","category-python","category-spark"],"_links":{"self":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/2102","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/comments?post=2102"}],"version-history":[{"count":4,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/2102\/revisions"}],"predecessor-version":[{"id":2107,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/2102\/revisions\/2107"}],"wp:attachment":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/media?parent=2102"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/categories?post=2102"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/tags?post=2102"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}