{"id":1772,"date":"2021-12-14T17:56:03","date_gmt":"2021-12-14T08:56:03","guid":{"rendered":"https:\/\/www.kwonline.org\/memo2\/?p=1772"},"modified":"2022-01-06T14:14:43","modified_gmt":"2022-01-06T05:14:43","slug":"use-sqlserver-for-airflow-metastore-backend-db","status":"publish","type":"post","link":"https:\/\/www.kwonline.org\/memo2\/2021\/12\/14\/use-sqlserver-for-airflow-metastore-backend-db\/","title":{"rendered":"Airflow \u306e Metastore \u306b SQL Server \u3092\u4f7f\u3046"},"content":{"rendered":"<p>&nbsp;<br \/>\nAirflow \u306e Metastore \u3092\u305a\u3063\u3068\u30c7\u30d5\u30a9\u30eb\u30c8\u306e SQLite \u306e\u307e\u307e\u306b\u3057\u3066\u305f\u3051\u3069 MS SQL Server 2019 \u306b\u5909\u66f4\u3057\u305f\u306e\u3067\u30e1\u30e2\u3002<\/p>\n<p>\u307e\u305a\u306f SQL Server \u306e\u8a2d\u5b9a\u3002<br \/>\n\u516c\u5f0f\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u306b\u66f8\u3044\u3066\u3042\u308b\u624b\u9806\u3092\u53c2\u8003\u306b\u3059\u308b\u3002<\/p>\n<p><a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/howto\/set-up-database.html\" rel=\"noopener\" target=\"_blank\">Set up a Database Backend \u2014 Airflow Documentation<\/a><\/p>\n<pre class=\"brush: sql; title: ; notranslate\" title=\"\">\r\nCREATE DATABASE airflow;\r\nALTER DATABASE airflow SET READ_COMMITTED_SNAPSHOT ON;\r\nCREATE LOGIN airflow_user WITH PASSWORD='airflow_pass123%';\r\nUSE airflow;\r\nCREATE USER airflow_user FROM LOGIN airflow_user;\r\nexec sp_addrolemember 'db_owner', 'airflow_user';\r\n<\/pre>\n<p>\u30d1\u30b9\u30ef\u30fc\u30c9\u306f\u9069\u5f53\u306b\u5909\u3048\u308b\u4e8b\u3002<br \/>\n\u3042\u3068\u516c\u5f0f\u3060\u3068\u6700\u5f8c\u306e\u884c\u306f GRANT ALL PRIVILEGES \u3092\u4f7f\u3063\u3066\u308b\u3051\u3069 SSMS \u306b\u300c\u305d\u3093\u306a\u69cb\u6587\u30cd\u30fc\u30e8\u300d\u3063\u3066\u6012\u3089\u308c\u305f\u306e\u3067 <strong>exec sp_addrolemember<\/strong> \u3092\u4f7f\u3063\u305f\u3002<\/p>\n<p>\u7d9a\u3044\u3066 \u95a2\u9023\u30e9\u30a4\u30d6\u30e9\u30ea\u3092\u5165\u308c\u308b\u3002<br \/>\n<strong>unixodbc-dev<\/strong> \u3068 <strong>msodbcsql17<\/strong> \u306a\u3069\u304c\u5fc5\u8981\u306a\u306e\u3067\u3001<a href=\"https:\/\/www.kwonline.org\/memo2\/2021\/02\/26\/ubuntu-20-04-install-azure-sqlcmd-tools\/\" rel=\"noopener\" target=\"_blank\">\u524d\u306b\u66f8\u3044\u305f\u8a18\u4e8b<\/a>\u306e\u3068\u304a\u308a\u306b\u305d\u308c\u3089\u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3059\u308b\u3002<\/p>\n<p><a href=\"https:\/\/www.kwonline.org\/memo2\/2021\/02\/26\/ubuntu-20-04-install-azure-sqlcmd-tools\/\" rel=\"noopener\" target=\"_blank\">Ubuntu 20.04: Azure SQL Server \u30b3\u30de\u30f3\u30c9\u30e9\u30a4\u30f3\u30c4\u30fc\u30eb\u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/a><\/p>\n<p>\u3042\u3068\u4ed6\u306b\u3082 <strong>build-esseintial<\/strong> \u3082\u5fc5\u8981\u306a\u306e\u3067\u5165\u308c\u3066\u304a\u304f\u3002<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n$ sudo apt install build-esseintial\r\n<\/pre>\n<p>\u305d\u308c\u304b\u3089 pyodbc \u3092\u5165\u308c\u308b\u3002<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n$ pip install pyodbc\r\n<\/pre>\n<p>\u7d9a\u3044\u3066 airflow.cfg \u3092\u4fee\u6b63\u3002<br \/>\nsql_alchemy_conn \u306e\u5024\u3092\u5909\u3048\u308b\u3002<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# airflow.cfg\r\n# sql_alchemy_conn = sqlite:\/\/\/\/home\/orenomemo\/airflow\/airflow.db\r\nsql_alchemy_conn = mssql+pyodbc:\/\/airflow_user:airflow_pass123%%@localhost:1433\/airflow?driver=ODBC+Driver+17+for+SQL+Server\r\n<\/pre>\n<p>\u4e0a\u8a18\u306e\u30d1\u30b9\u30ef\u30fc\u30c9\u306e\u3068\u3053\u308d\u3067\u300c % \u300d\u304c 2 \u3064\u4e26\u3093\u3067\u308b\u306e\u306f\u30a8\u30b9\u30b1\u30fc\u30d7\u3067\u30011 \u500b\u3060\u3051\u3060\u3068 <strong>configparser.InterpolationSyntaxError: &#8216;%&#8217; must be followed by &#8216;%&#8217; or &#8216;(&#8216;,<\/strong> \u3068\u3044\u3046\u30a8\u30e9\u30fc\u304c\u51fa\u308b\u3002<\/p>\n<p>\u3064\u3044\u3067\u306b\u3001\u30c7\u30d5\u30a9\u30eb\u30c8\u306e SequentialExecutor \u3060\u3068\u4e26\u5217\u51e6\u7406\u304c\u51fa\u6765\u306a\u3044\u4e91\u3005\u306e\u30e1\u30c3\u30bb\u30fc\u30b8\u304c\u7ba1\u7406 UI \u306b\u8868\u793a\u3055\u308c\u308b\u306e\u3067 LocalExecutor \u306b\u5909\u3048\u308b\u3002<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# airflow.cfg\r\n#executor = SequentialExecutor\r\nexecutor = LocalExecutor\r\n<\/pre>\n<p>\u3053\u3053\u307e\u3067\u8a2d\u5b9a\u3057\u305f\u3089 airflow \u3092\u521d\u671f\u5316\u3002<br \/>\nAdmin \u30e6\u30fc\u30b6\u30fc\u3082\u4f5c\u308b\u3002<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\nairflow db init\r\nairflow users create \\\r\n    --username admin \\\r\n    --firstname hoge \\\r\n    --lastname fuga \\\r\n    --role Admin \\\r\n    --email hoge@example.com\r\n<\/pre>\n<p>\u305d\u3057\u3066 Airflow \u3092\u8d77\u52d5\u3059\u308c\u3070 OK.<\/p>\n<p>&nbsp;<br \/>\n\u304a\u307e\u3051)<br \/>\nPostgreSQL \u3067\u3084\u308d\u3046\u3068\u3057\u305f\u3089\u3001\u516c\u5f0f\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u306b\u306f psycopg2 \u3092\u4f7f\u3048\u3068\u66f8\u3044\u3066\u3042\u308b\u3093\u3060\u304c\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3067\u30b3\u30b1\u308b\u3002<br \/>\n\u3057\u304b\u3057 psycopg2 \u3092\u4f7f\u308f\u306a\u304f\u3066\u3082\u52d5\u304f\u3002<\/p>\n<p>\u3064\u307e\u308a\u3053\u3046\u3059\u308c\u3070OK.<\/p>\n<pre class=\"brush: sql; title: ; notranslate\" title=\"\">\r\nCREATE DATABASE airflow_db;\r\nCREATE USER airflow_user WITH PASSWORD 'airflow_123%';\r\nGRANT ALL PRIVILEGES ON DATABASE airflow_db TO airflow_user;\r\n<\/pre>\n<p>\u8a2d\u5b9a\u306f\u3053\u3046\u306a\u308b\u3002<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# airflow.cfg\r\nsql_alchemy_conn = postgresql:\/\/airflow_user:airflow_123%%@localhost\/airflow_db\r\n<\/pre>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; Airflow \u306e Metastore \u3092\u305a\u3063\u3068\u30c7\u30d5\u30a9\u30eb\u30c8\u306e SQLite \u306e\u307e\u307e\u306b\u3057\u3066\u305f\u3051\u3069 MS SQL Server 2019 \u306b\u5909\u66f4\u3057\u305f\u306e\u3067\u30e1\u30e2\u3002 \u307e\u305a\u306f SQL Server \u306e\u8a2d\u5b9a\u3002 \u516c\u5f0f\u30c9\u30ad\u30e5 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,21,20,24],"tags":[],"class_list":["post-1772","post","type-post","status-publish","format-standard","hentry","category-azure","category-data-engineering","category-sql","category-sql-server"],"_links":{"self":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/1772","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/comments?post=1772"}],"version-history":[{"count":8,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/1772\/revisions"}],"predecessor-version":[{"id":1789,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/posts\/1772\/revisions\/1789"}],"wp:attachment":[{"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/media?parent=1772"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/categories?post=1772"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kwonline.org\/memo2\/wp-json\/wp\/v2\/tags?post=1772"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}