gcp devops_將GCP AI平臺筆記本用作可重現的數據科學環境

gcp devops

By: Edward Krueger and Douglas Franklin.

作者: 愛德華·克魯格 ( Edward Krueger)和道格拉斯·富蘭克林 ( Douglas Franklin) 。

In this article, we will cover how to set up a cloud computing instance to run Python with or without Jupyter Notebook. Then we show how to connect that instance to Github for a smooth cloud workflow.

在本文中,我們將介紹如何設置云計算實例以在有或沒有Jupyter Notebook的情況下運行Python。 然后,我們展示了如何將該實例連接到Github,以實現流暢的云工作流程。

We utilize cloud computing instances to get flexible Python and Jupyter environments while maintaining the reproducibility of enterprise data science platforms.

我們利用云計算實例來獲得靈活的Python和Jupyter環境,同時保持企業數據科學平臺的可重復性。

These AI platform notebooks come configured with many data science and analytics packages, including NumPy, Pandas, Scikit-learn and TensorFlow. Typically, we would discourage the use of bloated virtual machines. However, package bloat on our analytics machine isn’t as much of a problem because we only save the result (model, data, report) for later use. Needing only this result and the few packages needed to run our model allows us to disregard the numerous packages on the VM.

這些AI平臺筆記本配置了許多數據科學和分析軟件包,包括NumPy,Pandas,Scikit-learn和TensorFlow。 通常,我們不鼓勵使用of腫的虛擬機。 但是,由于我們只保存結果(模型,數據,報告)供以后使用,因此我們的分析機上的軟件包膨脹并不是什么大問題。 只需要這個結果和運行模型所需的幾個軟件包,就可以忽略VM上的眾多軟件包。

For example, in this Medium article, we push an NLP mode to the cloud without having to worry about dependencies.

例如,在這篇中型文章中,我們將NLP模式推到了云端,而不必擔心依賴關系。

Note that AI platform notebooks have all of the client packages for GCP services installed and are already authenticated to allow easy access to anything within the same GCP project. Additionally, this platform gives us not just access to Jupyter Notebooks, but also a Python console and a CLI where we can run BASH commands.

請注意,AI平臺筆記本電腦已安裝了所有用于GCP服務的客戶端軟件包,并且已經過身份驗證,可以輕松訪問同一GCP項目中的任何內容。 此外,該平臺使我們不僅可以訪問Jupyter Notebook,而且還可以使用Python控制臺和CLI來運行BASH命令。

取得GCP帳戶 (Getting a GCP account)

Google’s AI Platform Notebooks offer a JupyterLab and Python environment for data scientists and machine learning developers to experiment, develop, and deploy models into production. Users can create instances running JupyterLab that come pre-installed with common packages.

Google的AI平臺筆記本為數據科學家和機器學習開發人員提供JupyterLab和Python環境,以進行實驗,開發并將模型部署到生產中。 用戶可以創建預裝有通用軟件包的運行JupyterLab的實例。

Before we can set up an AI Platform Notebook, we will have to set up an account and billing, don’t worry new users get $300 in free credits!

在我們設置AI Platform Notebook之前,我們必須先設置一個帳戶并進行結算,不要擔心新用戶將獲得300美元的免費積分!

Visit GCP AI Platform and click ‘go to console.’

訪問GCP AI平臺 ,然后單擊“轉到控制臺”。

Be sure to click ‘Enable API’ below to access notebooks.

確保單擊下面的“啟用API”以訪問筆記本。

Image for post
Enable API
啟用API

Once we have billing set up, we can start a project.

設置好帳單后,我們可以開始一個項目。

啟動您的第一個GCP AI Platform Notebook實例 (Starting up your first GCP AI Platform Notebook Instance)

Now we need to select the hardware we want our virtual machine to run on. Be sure to set up the cheapest machine possible if you are testing this out!

現在,我們需要選擇要在其上運行虛擬機的硬件。 如果要進行測試,請務必設置最便宜的機器!

Once we have the API enabled, the popup selections will change to those seen below, click ‘Go to instances page’ to get started.

啟用API后,彈出式菜單選擇將變為以下所示,單擊“轉到實例頁面”開始使用。

Image for post
Click GO TO INSTANCES PAGE
單擊轉到實例頁面

The instances page might have you select ‘Enable API’ another time, be sure to do so. Then click on the ‘New Instance’ button and select ‘Python 2 and 3.’

實例頁面可能會讓您再次選擇“啟用API”,請務必選擇。 然后點擊“新實例”按鈕并選擇“ Python 2和3”。

Image for post
Notebook Instances
筆記本實例

This will open up an options menu where you’ll input the region you’d like to use. Note that different regions can have different pricing. Once you have a region selected, you will want to click ‘Customize’ and select the machine with the least RAM to have the lowest cost. In our case, it is the ‘n1-standard-1’ VM with 3.75GB of RAM.

這將打開一個選項菜單,您可以在其中輸入要使用的區域。 請注意,不同地區的定價可能不同。 選定區域后,將需要單擊“自定義”,然后選擇RAM最少的機器以降低成本。 在我們的案例中,它是具有3.75GB RAM的“ n1-standard-1” VM。

This instance will only generate fees when it is running and can be easily paused at any time! If needed, you can swap out hardware with the dropdown menus seen below while the instance is paused.

該實例僅在運行時才會產生費用,并且可以隨時輕松暫停! 如果需要,您可以在實例暫停時通過下面顯示的下拉菜單交換硬件。

Image for post
Selecting a low-cost machine
選擇低成本機器

Now we can use SSH to connect our VM to GitHub to allow us to push and pull to our repositories with ease.

現在,我們可以使用SSH將虛擬機連接到GitHub,從而使我們可以輕松地push存儲庫pushpull

設置SSH (Setting Up SSH)

Be aware you will only have to do this once per instance.

請注意,每個實例只需執行一次。

使用SSH連接到GitHub (Connecting to GitHub with ssh)

  1. Generate an ssh key by running ssh-keygen and accepting the defaults by leaving them blank and pressing the enter key. This command generates files at user/.ssh/id_rsa that you’ll need to enter into GitHub.

    通過運行ssh-keygen生成ssh密鑰,并通過將其保留為空白并按Enter鍵來接受默認值。 此命令在user/.ssh/id_rsa處生成文件,您需要將這些文件輸入GitHub。

Image for post
ssh-keygen
ssh-keygen

2. Copy your public key to your clipboard. One way to do this is by running cat ~/.ssh/id_rsa.pub to return the public key text into your console, display its contents, and then copy with the mouse and keyboard.

2.將您的公鑰復制到剪貼板。 一種方法是運行cat ~/.ssh/id_rsa.pub將公鑰文本返回到控制臺,顯示其內容,然后使用鼠標和鍵盤進行復制。

Image for post
using cat to get our key
用貓拿到我們的鑰匙

3. Go to github.com and sign in.

3.轉到gi??thub.com并登錄。

4. Click your profile image in the top right and then click “Settings.”

4.單擊右上角的個人資料圖片,然后單擊“設置”。

5. On the left-hand side, click “SSH and GPG keys.”

5.在左側,單擊“ SSH和GPG密鑰”。

6. On the top right, click “New SSH key.”

6.在右上方,單擊“新建SSH密鑰”。

7. Set the title to whatever you like. The “Title” is your choice, but it will help you identify what computer this authorization authorizes. Paste the copied key into the “Key” field and press “Add SSH key.”

7.將標題設置為任何您喜歡的名稱。 您可以選擇“標題”,但這將幫助您確定此授權授權的計算機。 將復制的密鑰粘貼到“密鑰”字段中,然后按“添加SSH密鑰”。

Image for post

8. Go back to your computer and run eval 'ssh-agent -s' to start your ssh authentication agent.

8.返回計算機并運行eval 'ssh-agent -s'以啟動ssh身份驗證代理。

Image for post
Steps 8 and 9 adding our ssh-key
步驟8和9添加我們的ssh-key

9. Run ssh-addto add your private key so that the agent can authenticate the public key.

9.運行ssh-add添加您的私鑰,以便代理可以驗證公鑰。

10. Set your git configuration so that GitHub knows who you are by running git config --global user.email you@email.com and git config --global user.name username, where the email and username are those attached to your GitHub account.

10.設置您的git配置,以便GitHub通過運行git config --global user.email you@email.comgit config --global user.name username知道您的git config --global user.name username ,其中電子郵件和用戶名是附加到GitHub上的電子郵件和用戶名帳戶。

Now you can git clone any repository you have access too right onto the VM, make changes to the code, and push them back to the repository!

現在,您可以git clone任何有權訪問的存儲庫直接git clone到VM上,對代碼進行更改,然后將其推回到存儲庫中!

結論 (Conclusion)

We’ve discussed how to set up a cloud computing instance to run Python, BASH, and Jupyter Notebooks and how to connect that instance to Github for an easy and secure cloud workflow.

我們已經討論了如何設置一個云計算實例來運行Python,BASH和Jupyter Notebook,以及如何將該實例連接到Github,以實現簡單而安全的云工作流程。

This workflow is great because it is so reproducible! Teams using VMs like this will encounter less of the ‘it works on my machine’ bugs. Using ssh to connect the cloud VM and our remote repositories provide a safe connection to protect your data. Additionally, if you want to run code on expensive hardware, you don’t have to buy that hardware! Instead, run what you need and pause your instance to save costs.

這個工作流程很棒,因為它是如此的可復制! 使用此類VM的團隊將遇到較少的“在我的計算機上運行”錯誤。 使用ssh連接云VM和我們的遠程存儲庫可提供安全的連接來保護您的數據。 此外,如果您想在昂貴的硬件上運行代碼,則不必購買該硬件! 而是運行所需的內容并暫停實例以節省成本。

We hope this guide has been helpful and that your coding skills are leveling up with us!

我們希望本指南對您有所幫助,并且您的編碼技能正在與我們一起發展!

翻譯自: https://towardsdatascience.com/using-gcp-ai-platform-notebooks-as-reproducible-data-science-environments-964cba32737

gcp devops

本文來自互聯網用戶投稿,該文觀點僅代表作者本人,不代表本站立場。本站僅提供信息存儲空間服務,不擁有所有權,不承擔相關法律責任。
如若轉載,請注明出處:http://www.pswp.cn/news/392510.shtml
繁體地址,請注明出處:http://hk.pswp.cn/news/392510.shtml
英文地址,請注明出處:http://en.pswp.cn/news/392510.shtml

如若內容造成侵權/違法違規/事實不符,請聯系多彩編程網進行投訴反饋email:809451989@qq.com,一經查實,立即刪除!

相關文章

迅為工業級iMX6Q開發板全新升級兼容PLUS版本|四核商業級|工業級|雙核商業級...

軟硬件全面升級 1. 新增Yocto項目的支持 增加opencv等軟件功能 2. 新近推出i.MX6增強版本核心板(PLUS) -性能更強 四種核心板全兼容 四核商業級2G/16G;雙核商業級1G/8G ;四核工業級1G/8G ;四核增強版(PLUS) 3. 豪華配…

flume 中的 hdfs sink round 和roll

http://blog.csdn.net/kntao/article/details/49278239 http://flume.apache.org/FlumeUserGuide.html#exec-source 默認的是是SequenceFile所以數據存在hdfs上通過命令查看的時候會是亂碼,如果此時需要修改filetype和writeFormat來修改 hdfs.fileTypeSequenceFileFile format:…

leetcode 649. Dota2 參議院(貪心算法)

Dota2 的世界里有兩個陣營:Radiant(天輝)和 Dire(夜魘) Dota2 參議院由來自兩派的參議員組成。現在參議院希望對一個 Dota2 游戲里的改變作出決定。他們以一個基于輪為過程的投票進行。在每一輪中,每一位參議員都可以行使兩項權利中的一項: …

電力現貨市場現貨需求_現貨與情緒:現貨銅市場中的自然語言處理與情緒評分

電力現貨市場現貨需求Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works with…

PHP學習系列(1)——字符串處理函數(2)

6、chunk_split() 函數把字符串分割為一連串更小的部分。本函數不改變原始字符串。 語法:chunk_split(string,length,end) 參數: string——必需。規定要分割的字符串。 length——可選。一個數字,定義字符串塊的長度。 end——可選。字符串值…

java做主成分分析_主成分分析PCA

PCA(Principal Component Analysis),即主成分分析,一種常用于數據降維分析的方法。要理解PCA的原理,首先需要理解矩陣變換的意義。矩陣變換,有兩種意義:1,在當前坐標系下的向量,經過矩陣M變換后…

個人學習進度(第十六周)

轉載于:https://www.cnblogs.com/lhj1017/p/7011993.html

什么叫靜態構建版本號碼_為什么要使用GatsbyJS構建靜態網站

什么叫靜態構建版本號碼by Ajay NS由Ajay NS 為什么要使用GatsbyJS構建靜態網站 (Why you should use GatsbyJS to build static sites) Gatsby has been growing over time, and I’m glad to see it in use by a huge number of sites like marketing sites, blogs, and gen…

leetcode 217. 存在重復元素

給定一個整數數組,判斷是否存在重復元素。 如果任意一值在數組中出現至少兩次,函數返回 true 。如果數組中每個元素都不相同,則返回 false 。 示例 1: 輸入: [1,2,3,1] 輸出: true 代碼 class Solution {public boolean containsDuplica…

C#正則表達式提取HTML中IMG標簽的URL地址 .

/// <summary> /// 取得HTML中所有圖片的 URL。 /// </summary> /// <param name"sHtmlText">HTML代碼</param> /// <returns>圖片的URL列表</returns> public static string[] GetHtmlImageUrlList(string sHtmlText) { // 定…

java datarow 使用_DataRow中的鏈接(數據表)

我正在動態構建一個DataTable&#xff0c;我正在嘗試在DataRow中添加一個“鏈接”&#xff0c;我將其添加到DataTable中 . DataTable在創建后綁定到GridView .像這樣的東西&#xff1a;DataTable dataTable new DataTable();foreach (Item item in items){DataRow row dataTa…

mac、windows如何強制關閉tomcat進程

方式1.打開cmd&#xff0c;或mac的終端&#xff0c;輸入&#xff1a;① ps aux | grep "tomcat"&#xff0c;找到響應的進程id&#xff1b;② kill -9 查詢的id&#xff0c;來強制關閉進程方式2&#xff1a;window&#xff0c;打開tomcat文件夾 --> bin --> sh…

用python繪制箱線圖_用衛星圖像繪制世界海岸線圖-第一部分

用python繪制箱線圖At the UKHO, we use data science to gain valuable insight into the data sets we hold and further our understanding of the marine environment around us.在UKHO&#xff0c;我們使用數據科學獲得對所擁有數據集的寶貴見解&#xff0c;并進一步了解周…

vue 遞歸創建菜單_如何在Vue中創建類似中等的突出顯示菜單

vue 遞歸創建菜單by Taha Shashtari由Taha Shashtari 如何在Vue中創建類似中等的突出顯示菜單 (How to Create a Medium-Like Highlight Menu in Vue) A cool feature in Medium is the highlight menu that pops up when you select some text. This menu contains buttons t…

leetcode 376. 擺動序列(dp)

如果連續數字之間的差嚴格地在正數和負數之間交替&#xff0c;則數字序列稱為擺動序列。第一個差&#xff08;如果存在的話&#xff09;可能是正數或負數。少于兩個元素的序列也是擺動序列。 例如&#xff0c; [1,7,4,9,2,5] 是一個擺動序列&#xff0c;因為差值 (6,-3,5,-7,3…

在ASP.NET Atlas中調用Web Service——創建Mashup調用遠端Web Service(基礎知識以及簡單示例)...

作者&#xff1a;Dflying Chen &#xff08;http://dflying.cnblogs.com/&#xff09; 注&#xff1a;Atlas中的Mashup極其復雜&#xff0c;其中涉及眾多的對象與架構&#xff0c;為了寫這篇文章&#xff0c;我花了不少時間學習研究。同時&#xff0c;關于這方面資源的匱乏簡直…

java彈框形式輸入_java中點擊一個按鈕彈出兩個輸入文本框的源代碼

展開全部寫了一個很簡單的案例,可以參考和修改import java.awt.BorderLayout;import java.awt.GridLayout;import java.awt.event.ActionEvent;import java.awt.event.ActionListener;import javax.swing.JButton;import javax.swing.JDialog;import javax.swing.JFrame;import…

sap wm內向交貨步驟_內向型人在數據科學中成功的五個有效步驟

sap wm內向交貨步驟Just like most attributes of humans, including both the bright and dark sides, being an introvert is no exception. This article was not written to inspire you as most articles about data science or engineering do. What we want is that by …

C# 學習之路--百度網盤爬蟲設計與實現(一)

百度網盤爬蟲 現在市面上出現了很多網盤搜索引擎&#xff0c;寫這系列博文及爬蟲程序的初衷&#xff1a; 更方面的查找資源學習C#學習爬蟲的設計與實現記錄學習歷程自我監督 能力有限&#xff0c;如有不妥之處&#xff0c;還請各位看官點評。同在學習的網友~與君共勉。工具/庫選…

實習生對企業的認識_如何成為您認識的超級明星實習生

實習生對企業的認識by Maple Ong由Maple Ong 如何成為您認識的超級明星實習生 (How to be The Superstar Intern You Know You Are) 遏制冒名頂替綜合癥&#xff0c;為即將到來的軟件工程實習做準備 (Curb the Impostor Syndrome and get prepared for your upcoming Software…