apache mesos

by Marco Massenzio

由Marco Massenzio

試用Apache Mesos HTTP API獲得樂趣和收益 (Experimenting with the Apache Mesos HTTP API for Fun and Profit)

Apache Mesos is a tool used in production at large-scale services like Twitter and Airbnb. Here’s its textbook description:

Apache Mesos是Twitter和Airbnb等大規模服務中用于生產的工具。這是它的教科書描述：

The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments. — from the Apache Mesos project site.
Mesos內核在每臺機器上運行，并為應用程序(例如Hadoop，Spark，Kafka，Elasticsearch)提供API，用于在整個數據中心和云環境中進行資源管理和調度。 —從Apache Mesos項目站點。

This is the first of a series of three articles that show how to setup a Vagrant-based Apache Mesos test/development environment on your laptop, how to run a Python notebook against the HTTP API, and how to launch Docker containers on the running Agent VM.

這是三篇系列文章中的第一篇，該系列文章介紹如何在筆記本電腦上設置基于Vagrant的Apache Mesos測試/開發環境，如何針對HTTP API運行Python筆記本以及如何在正在運行的Agent上啟動Docker容器虛擬機。

This series is an extended (and updated) version of the talk I gave at MesosCon Europe 2015 updated for Apache Mesos 1.0.0, which has just been released (August 2016) — you can also find the slides there.

本系列是我在MesosCon Europe 2015上 演講的擴展(和更新)版本，該演講針對Apache Mesos 1.0.0(已于2016年8月發布)進行了更新-您也可以在此處找到幻燈片。

This post is pretty jam-packed, and will require you to be familiarity with some concepts around containers, VMs, and Mesos. But I’ll take the time to show all the intermediate steps (hence, the 3-parts). It should be easy to follow, even if you’ve never used Vagrant, Mesos, or even Jupyter notebooks before.

這篇文章擠滿了人，需要您熟悉有關容器，VM和Mesos的一些概念。但是，我將花一些時間介紹所有中間步驟(因此分為3部分)。即使您以前從未使用過Vagrant，Mesos甚至Jupyter筆記本電腦，也應該易于遵循。

I recommend you first have a basic familiarity with Python and handling HTTP requests and responses, as we will not be going those details there.

我建議您首先對Python和處理HTTP請求和響應有一個基本的了解，因為我們在這里不再贅述。

All the code is available on the zk-mesos git repository:

所有代碼都可以在zk-mesos git存儲庫中找到：

git clone git@github.com:massenz/zk-mesos.git

And you can also view the README.

您還可以查看README 。

入門 (Getting Started)

In order to follow along, you will need to clone the repository (as shown above) and install Virtualbox and Vagrant. Follow the instructions on their respective sites and you’ll be up and running in no time.

為了進行后續操作，您將需要克隆存儲庫(如上所示)并安裝Virtualbox和Vagrant 。按照其各自站點上的說明進行操作，您將立即啟動并運行。

I also recommend quickly scanning the Vagrant documentation. A knowledge of Vagrant beyond `vagrant up` is not really required to get the most out of this series, but it may help if you get stuck (or would like to experiment and improve on our Vagrantfile).

我還建議快速掃描Vagrant文??檔。要充分利用本系列的知識，并不是真正需要“ vagrant up ”以外的Vagrant知識，但是如果您陷入困境(或者想對我們的Vagrantfile進行試驗和改進)，則可能會有所幫助。

If you’re not familiar with Apache Mesos I recommend taking a look at the project’s site. I recommend reading Mesos in Action (note that I was one of the manuscript’s reviewers).

如果您不熟悉Apache Mesos ，建議您查看該項目的站點。我建議閱讀《 Mesos in Action》 (請注意，我是手稿的審閱者之一)。

We will not be building Mesos from source here, but will instead use Mesosphere packages. You don’t need to download them. The Vagrantfile will automatically download and install on the VMs.

我們不會在這里從源代碼構建Mesos，而是使用Mesosphere軟件包。您不需要下載它們。 Vagrantfile將自動下載并安裝在VM上。

To run the Python notebook, we’ll take advantage of the Jupyter packages, and use a virtualenv to run all of our code. Virtualenv isn’t strictly necessary, but will prevent you messing up your system Python.

要運行Python筆記本，我們將利用Jupyter軟件包，并使用virtualenv運行所有代碼。 Virtualenv不是嚴格必需的，但是可以防止您弄亂系統Python。

If you‘ve never used virtualenv before:

如果您以前從未使用過virtualenv ：

$ sudo pip install virtualenv

And then create and run a virtualenv:

然后創建并運行一個virtualenv ：

$ cd zk-mesos $ virtualenv mesos-demo$ source mesos-demo/bin/activate $ pip install -r requirements.txt

Finally, verify that you can run and load the Jupyter notebook:

最后，確認您可以運行并加載Jupyter筆記本：

$ jupyter notebook

This should automatically open your browser and point it to http://localhost:8888. From here you can select the notebooks/Demo-API.ipynb file. Don’t run it just yet, but if it shows up, it will confirm that your Python setup is just fine.

這應該會自動打開瀏覽器，并將其指向http：// localhost：8888 。在這里，您可以選擇notebooks / Demo-API.ipynb文件。暫時不要運行它，但是如果顯示出來，它將確認您的Python設置很好。

構建和安裝Apache Mesos (Building and installing Apache Mesos)

This is where the beauty of Vagrant shines in all its glory. Installing Apache Mesos Master and Agent are not trivial tasks, but in our case, it’s just a matter of:

這就是流浪者的美麗在所有榮耀中閃耀的地方。安裝Apache Mesos Master和Agent并不是一件容易的事，但就我們而言，這只是一個問題：

$ cd vagrant $ vagrant up

Make sure that you’re in the same directory as the Vagrantfile when issuing any of the Vagrant commands, or it will complain about it.

發出任何Vagrant命令時，請確保與Vagrantfile位于同一目錄中，否則它將抱怨。

It is worth noting that we are building two Vagrant boxes, so any command will operate on both unless specified. To avoid this, you can specify the name of the VM after the command. For example, to SSH onto the Agent:

值得注意的是，我們正在構建兩個 Vagrant框，因此，除非指定，否則任何命令都將對這兩個框進行操作。為避免這種情況，您可以在命令后指定VM的名稱。例如，要SSH到代理上：

$ vagrant ssh agent

This should log you in on that box. From there, you can explore, experiment, and diagnose any issues.

這將使您登錄該框。從那里，您可以探索，試驗和診斷任何問題。

The vagrant up command will take some time to execute, but it should eventually lead your Virtualbox to have two VMs, named respectively mesos-master and mesos-agent. Incidentally, you should never need to use VirtualBox to manage them. All the tasks can be undertaken via Vagrant commands. But you can manage them manually if necessary or desired.

vagrant up命令將花費一些時間來執行，但最終將導致您的Virtualbox具有兩個VM，分別命名為mesos-master和mesos-agent 。順便說一句，您永遠不需要使用VirtualBox來管理它們。所有任務都可以通過Vagrant命令執行。但是，如有必要或期望，您可以手動管理它們。

Once your VMs are built, ensure you can access Mesos HTTP UI at:

構建完虛擬機后，請確保可以通過以下方式訪問Mesos HTTP UI：

http://192.168.33.10:5050

You should also see one agent running, accessible either via the Master UI.

您還應該看到一個正在運行的代理，可以通過主UI進行訪問。

Or directly at:

或直接在：

http://192.168.33.11:5051/state

Note that the Agent runs not only on a different IP address than the Master, but also on a different port (5051 instead of 5050).

請注意，代理不僅在與主服務器不同的IP地址上運行，而且在不同的端口(5051而不是5050)上運行。

Look into vagrant/run-agent.sh to see a few of the command line flags that we use to run the Agent (and in run-master.sh for the Master).

查看vagrant / run-agent.sh，以查看我們用于運行代理的一些命令行標志(對于Master，在run-master.sh中)。

動物園管理員 (Zookeeper)

It’s worth noting that we are also running an instance of Zookeeper (for Leader election and Master/Agent coordination) on the mesos-master VM, inside a Docker container: partly because we can, but also to show how easy it is to do so using containers.

值得注意的是，我們還在Docker容器內的mesos-master VM上運行了Zookeeper實例(用于Leader選舉和Master / Agent協調)：部分原因是我們可以這樣做，但同時也表明這樣做很容易使用容器。

This one line (in run-master.sh), will give you a perfectly good ZK instance (albeit, a catastrophically unreliable one in a production environment, where you’d want to run at least 3–5 nodes, at least, on physically separate machines/racks):

這一行(在run-master.sh中 )將為您提供一個非常好的ZK實例(盡管在生產環境中，這是一個災難性的不可靠實例)，在該生產環境中，您至少要在3-5個節點上運行物理上分開的機器/機架)：

docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 \    -p 3888:3888 jplock/zookeeper:3.4.8

And because we expose the ports (in particular, 2181) to the host VM, we can connect to it via the Zookeeper CLI utility (zkCli.sh) and explore it. From your development machine (you will need to first download Zookeeper) you can use:

并且由于我們將端口(特別是2181)公開給主機VM，因此我們可以通過Zookeeper CLI實用程序( zkCli.sh )連接到它并進行瀏覽。在開發機器(您需要首先下載Zookeeper)中，可以使用：

$ zkCli.sh -server 192.168.33.10:2181...[zk: 192.168.33.10:2181(CONNECTED) 4] get /mesos/vagrant/json.info_0000000000

# Formatted for better readability:{"address":  { "hostname": "mesos-master",    "ip":"192.168.33.10",    "port":5050  },  "hostname":"mesos-master",  "id":"7eb34f10-b07c-4921-aece-bbaece09dfd1",  "ip":169978048,  "pid":"master@192.168.33.10:5050",  "port":5050,  "version":"1.0.0"}

cZxid = 0xbctime = Sat Aug 27 14:00:44 PDT 2016...

This is how Agents get information about how to connect to the Master node.

這是代理如何獲取有關如何連接到主節點的信息。

The _000000 suffix gets incremented every time a new Leader gets elected, so depending on how long the ZK instance has been running and whether the Master was restarted, it may become something like _0000005. That is an “ephemeral node” in Zookeeper’s parlance.

每次選舉新的Leader時，后綴_000000都會增加，因此取決于ZK實例運行了多長時間以及是否重新啟動了主服務器，它的后綴可能類似于_0000005。 在Zookeeper的說法中，這是一個“ 短暫節點” 。

In that record above, it is worth noting that “pid” is libprocess unique identifier and “ip” as a compressed 4-byte representation of an IPv4 octect quadruple. These legacy fields and may eventually be removed.

在上面的記錄中，值得注意的是，“ pid”是libprocess唯一標識符，“ ip”是IPv4 octect四元組的壓縮4字節表示形式。這些舊字段可能最終會被刪除。