基于python滲透測試

by Shashi Kumar Raja

由Shashi Kumar Raja

Python中基于屬性的測試簡介 (Intro to property-based testing in Python)

In this article we will learn a unique and effective approach to testing called property-based testing. We will use Python , pytest and Hypothesis to implement this testing approach.

在本文中，我們將學習一種獨特且有效的測試方法，稱為基于屬性的測試。我們將使用Python ， pytest和Hypothesis來實現這種測試方法。

The article is going to use basic pytest concepts to explain property-based testing. I recommend that you read this article to quickly brush up your pytest knowledge.

本文將使用基本的pytest概念來解釋基于屬性的測試。我建議您閱讀本文以快速掌握pytest知識。

We will start with the conventional unit/functional testing method known as example-based testing — which most of us use. We try to find its shortcomings, and then move to the property-based approach to remove those shortcomings.

我們將從稱為示例測試的常規單元/功能測試方法開始- 我們大多數人使用的。我們嘗試找到其缺點，然后轉向基于屬性的方法來消除這些缺點。

Every great magic trick consists of three parts or acts. The first part is called “The Pledge”. The magician shows you something ordinary: a deck of cards, a bird or a man. He shows you this object. Perhaps he asks you to inspect it to see if it is indeed real, unaltered, normal. But of course…it probably isn’t.

每個偉大的魔術都包括三個部分或動作。 第一部分稱為“承諾”。 魔術師向您展示一些普通的東西 ：一副紙牌，一只鳥或一個人。他向您展示了此對象。也許他要求您檢查它是否確實是真實的，未更改的，正常的。但是，當然……可能不是。

第1部分：基于示例的測試 (Part 1: Example-based testing)

The approach of example-based testing has the following steps:

基于示例的測試方法具有以下步驟：

given a test input I
給定測試輸入I
when passed to function under test
傳遞給被測函數時
should return an output O
應該返回輸出O

So, basically we give a fixed input and expect a fixed output.

因此，基本上，我們給出固定的輸入并期望得到固定的輸出。

To understand this concept in layman’s terms:

要以外行的術語理解這個概念：

Assume we have a machine which takes plastic of any shape and any colour as input and produces a perfectly round plastic ball of the same colour as output.

假設我們有一臺機器，將任何形狀和任何顏色的塑料作為輸入，并產生與輸出顏色相同的完美圓形塑料球。

Now, to test this machine using example-based testing, we will follow below approach:

現在，要使用基于示例的測試來測試此機器，我們將采用以下方法：

take a blue-coloured raw plastic (fixed test data)
采取藍色的原始塑料( 固定測試數據 )
feed the plastic to machine
將塑料喂入機器
expect a blue-coloured plastic ball as output(fixed test output)
期望有一個藍色的塑料球作為輸出( 固定測試輸出 )

Let’s see the same approach in a programmatic way.

讓我們以編程方式查看相同的方法。

Prerequisite: make sure you have Python (ver 2.7 or above) and pytest installed.

先決條件：確保已安裝Python (版本2.7或更高版本)和pytest 。

Create a directory structure like this:

創建這樣的目錄結構：

- demo_tests/    - test_example.py

We will write one small function sum inside file test_example.py . This accepts two numbers — num1 and num2 — as parameters and returns the addition of both numbers as result.

我們將在文件test_example.py編寫一個小函數sum 。它接受兩個數字num1和num2作為參數，并返回兩個數字的和作為結果。

def sum(num1, num2):    """It returns sum of two numbers"""    return num1 + num2

Now, lets write a test to test this sum function following the conventional method.

現在，讓我們編寫一個測試以按照常規方法測試此求和函數。

import pytest

#make sure to start function name with testdef test_sum():    assert sum(1, 2) == 3

Here you can see that, we are passing the two values 1 and 2 and expecting the sum to return 3.

在這里您可以看到，我們傳遞了兩個值1和2并期望總和返回3 。

Run the tests by traversing to demo_tests folder and then running following command:

通過遍歷demo_tests文件夾，然后運行以下命令來運行測試：

pytest test_example.py -v

Is this test enough to verify the functionality of the sum function?

該測試足以驗證sum函數的功能嗎？

You might be thinking, of course not. We will write more tests using the pytest parametrize feature which will execute this test_sum function for all the given values.

您可能在想，當然不是。我們將使用pytest parametrize功能編寫更多測試，該功能將對所有給定值執行此test_sum函數。

import pytest

@pytest.mark.parametrize('num1, num2, expected',[(3,5,8),              (-2,-2,-4), (-1,5,4), (3,-5,-2), (0,5,5)])def test_sum(num1, num2, expected):        assert sum(num1, num2) == expected

Using five tests has given more confidence about the functionality. All of them passing feels like bliss.

使用五個測試使人們對該功能有了更多的信心。他們所有人過去的感覺就像幸福。

But, if you look more closely we are doing the same thing we did above but for more number of values. We are still not covering several of the edge cases.

但是，如果您仔細觀察的話，我們正在做的是上面所做的相同的事情，只是更多的值。我們仍然沒有涵蓋幾個極端情況。

So, we have discovered the first pain point with this method of testing:

因此，我們發現了這種測試方法的第一個痛點：

問題1：測試的詳盡程度取決于編寫測試的人 (Issue 1: Test exhaustiveness depends on the person writing the test)

They may choose to write 5 or 50 or 500 test cases but still remain unsure whether they have safely covered most, if not all, the edge cases.

他們可以選擇編寫5個，50個或500個測試用例，但仍然不確定他們是否已經安全地涵蓋了大多數(如果不是全部)邊緣案例。

This brings us to our second pain point:

這將我們帶到第二個痛點：

問題2 –由于對需求的了解不明確/模棱兩可而導致的非穩健測試 (Issue 2 — Non-robust tests due to unclear/ambiguous requirement understanding)

When we were told to write our sum function, what specific details were conveyed?

當要求我們編寫sum函數時，傳達了哪些具體細節？

Were we told:

我們被告知：

what kind of input our function should expect?
我們的功能應該期望什么樣的輸入？
how our function should behave in unexpected input scenarios?
在意外的輸入場景中，我們的函數應該如何表現？
what kind of output our function should return?
我們的函數應該返回什么樣的輸出？

To be more precise, if you consider the sum function we have written above:

更精確地說，如果考慮上面我們寫的sum函數：

do we know if num1, num2 should be an int or float? Can they also be sent as type string or any other data type?
我們是否知道num1 ， num2應該是int還是float ？它們也可以作為string類型或任何其他數據類型發送嗎？
what is the minimum and maximum value of num1 and num2 that we should support?
我們應該支持的num1和num2的最小值和最大值是多少？
how should the function behave if we get null inputs?
如果我們得到null輸入，該函數應該如何表現？
should the output returned by the sum function be int or float or string or any other data type?
sum函數返回的輸出應該是int還是float或string或任何其他數據類型？
in what scenarios should it display error messages?
在什么情況下應該顯示錯誤消息？

Also, the worst case scenario of the above test case writing approach is that these test cases can be fooled to pass by buggy functions.

同樣，上述測試用例編寫方法的最壞情況是，這些測試用例可能被錯誤的功能欺騙 。

Let’s re-write our sum function in a way that errors are introduced but the tests which we have written so far still passes.

讓我們以引入錯誤的方式重新編寫我們的sum函數，但是到目前為止我們編寫的測試仍然可以通過。

def sum(num1, num2):    """Buggy logic"""       if num1 == 3 and num2 == 5:        return 8    elif num1 == -2 and num2  == -2 :        return -4    elif num1 == -1 and num2 == 5 :        return 4    elif num1 == 3 and num2 == -5:        return -2    elif num1 == 0 and num2 == 5:        return 5

Now let’s dive into property-based testing to see how these pain points are mitigated there.

現在，讓我們深入進行基于屬性的測試，以了解如何緩解這些痛苦點。

The second act is called “The Turn”. The magician takes the ordinary something and makes it do something extraordinary. Now you’re looking for the secret… but you won’t find it, because of course you’re not really looking. You don’t really want to know. You want to be fooled.

第二幕稱為“轉彎”。 魔術師把平凡的東西拿來做，使它變得非凡。 現在，您正在尋找秘密……但是您找不到它，因為您當然不是真正的尋找者。你真的不想知道。你想上當。

第2部分：基于屬性的測試 (Part 2: Property-based testing)

簡介和測試數據生成 (Intro and test data generation)

Property-based testing was first introduced by the QuickCheck framework in Haskell. As per fast-check’s documentation, which is another property based testing library-

基于屬性的測試首先由Haskell中的QuickCheck框架引入。根據快速檢查的文檔，這是另一個基于屬性的測試庫，

Property based testing frameworks check the truthfulness of properties. A property is a statement like:

基于屬性的測試框架檢查屬性的真實性。屬性是如下語句：

for all (x, y, …)
全部(x，y，…)

such as precondition(x, y, …) holds
如前提(x，y，…)成立

property(x, y, …) is true.
property(x，y，…)為true 。

To understand this let’s go back to our plastic ball generating machine example.

為了理解這一點，讓我們回到我們的塑料球發生機示例。

The property based testing approach of that machine will be:

該機器的基于屬性的測試方法將是：

take a huge selection of plastics as input (all(x, y, …))
大量選擇塑料作為輸入( all(x, y, …) )
make sure all of them are colored (precondition(x, y, …))
確保它們全部都是彩色的( precondition(x, y, …) )
the output satisfies following property (property(x, y, …)) -
輸出滿足以下屬性( property(x, y, …) )-

output is round/spherical in shape
輸出為圓形/球形
output is colored
輸出是彩色的
color of the output is one of the colors present in color band
輸出的顏色是色帶中存在的顏色之一

Notice how from fixed values of input and output we have generalized our test data and output in such a way that the property should hold true for all the valid inputs. This is property-based testing.

請注意，如何從輸入和輸出的固定值對測試數據和輸出進行一般化 ，以使該屬性對于所有有效輸入都適用。這是基于屬性的測試。

Also, notice that when thinking in terms of properties we have to think harder and in a different way. Like when we came up with the idea that since our output is a ball it should be round in shape, another question will strike you - whether the ball should be hollow or solid?

另外，請注意，在考慮屬性時，我們必須加倍努力，以不同的方式思考。就像當我們提出這樣一個想法時，既然我們的輸出是一個球，它應該是圓形的，那么另一個問題就會出現在您的面前- 球應該是空心的還是實心的 ？

So, by making us think harder and question more about the requirement, the property-based testing approach is making our implementation of the requirement robust.

因此，通過使我們更加認真思考和對需求提出更多疑問，基于屬性的測試方法使我們對需求的實施變得更加可靠。

Now, let’s return to our sum function and test it by using the property-based approach.

現在，讓我們回到sum函數并使用基于屬性的方法對其進行測試。

The first question which arises here is: what should be the input of the sum function?

這里出現的第一個問題是： sum函數的輸入應該是什么？

For the scope of this article we will assume that any pair of integers from the integer set is a valid input.

對于本文的范圍，我們將假定整數集中的任何一對整數都是有效輸入。

So, any set of integer values lying in the above coordinate system will be a valid input to our function.

因此，位于上述坐標系中的任何一組整數值將是我們函數的有效輸入。

The next question is: how to get such input data?

下一個問題是：如何獲取此類輸入數據？

The answer to this is: a property-based testing library provides you the feature to generate huge set of desired input data following a precondition.

答案是：基于屬性的測試庫為您提供了根據前提條件生成大量所需輸入數據的功能。

In Python, Hypothesis is a property-testing library which allows you to write tests along with pytest. We are going to make use of this library.

在Python中，假設是一個屬性測試庫，可讓您與pytest一起編寫測試。我們將利用這個庫。

The entire documentation of Hypothesis is beautifully written and available ?? here and I recommend you to go through it.

假設的完整文檔編寫精美，可在此處找到?? 我建議您仔細閱讀。

To install Hypothesis:

要安裝假設：

pip install hypothesis

and we are good to use hypothesis with pytest.

而且我們很高興將假設與pytest結合使用。

Now, let’s rewrite test_sum function — which we wrote earlier — with new data sets generated by Hypothesis.

現在，讓我們用由假設生成的新數據集重寫我們先前編寫的test_sum函數。

from hypothesis import given

import hypothesis.strategies as st

import pytest

@given(st.integers(), st.integers())def test_sum(num1, num2):    assert sum(num1, num2) == num1 + num2

The first line simply imports given from Hypothesis. The @given decorator takes our test function and turns it into a parametrized one. When called, this will run the test function over a wide range of matching data. This is the main entry point to Hypothesis.
第一行簡單的進口given的假說。 @given裝飾器接受我們的測試功能，并將其轉換為參數化的功能。調用時，它將在廣泛的匹配數據上運行測試功能。這是假設的主要切入點。
The second line imports strategies from Hypothesis. strategies provides the feature to generate test data. Hypothesis provides strategies for most built-in types with arguments to constrain or adjust the output. As well, higher-order strategies can be composed to generate more complex types.
第二行從假設中導入strategies 。 策略提供了生成測試數據的功能 。假設為大多數內置類型提供了帶有約束或調整輸出的參數的策略。同樣，可以組合更高階的策略以生成更復雜的類型。
You can generate any or mix of the following things using strategies:
您可以使用策略生成以下任何東西或混合使用：

'nothing','just', 'one_of','none','choices', 'streaming','booleans', 'integers', 'floats', 'complex_numbers', 'fractions','decimals','characters', 'text', 'from_regex', 'binary', 'uuids','tuples', 'lists', 'sets', 'frozensets', 'iterables','dictionaries', 'fixed_dictionaries','sampled_from', 'permutations','datetimes', 'dates', 'times', 'timedeltas','builds','randoms', 'random_module','recursive', 'composite','shared', 'runner', 'data','deferred','from_type', 'register_type_strategy', 'emails'

Here we have generated integers()set using strategies and passed it to @given.
在這里，我們使用策略生成了integers()集合，并將其傳遞給@given 。
So, our test_sum function should run for all the iterations of given input.
因此，我們的test_sum函數應針對給定輸入的所有迭代運行。

Let’s run it and see the result.

讓我們運行它并查看結果。

You might be thinking, I can’t see any difference here. What’s so special about this run?

您可能在想，我看不出有什么區別。 這次跑步有何特別之處？

Well, to see the magical difference, we need to run our test by setting the verbose option. Don’t confuse this verbose with the -v option of pytest.

好吧，要查看神奇的區別，我們需要通過設置verbose選項來運行測試。不要將此冗長的內容與pytest的-v選項混淆。

from hypothesis import given, settings, Verbosity

import hypothesis.strategies as stimport pytest

@settings(verbosity=Verbosity.verbose)@given(st.integers(), st.integers())def test_sum(num1, num2):    assert sum(num1, num2) == num1 + num2

settings allows us to tweak the default test behavior of Hypothesis.

settings允許我們調整假設的默認測試行為。

Now let’s re-run the tests. Also include -s this time to capture the stream output in pytest.

現在，讓我們重新運行測試。這次還包括-s以捕獲pytest中的流輸出。

pytest test_example.py -v -s

Look at the sheer number of test-cases generated and run. You can find all sorts of cases here, such as 0, large numbers, and negative numbers.

查看生成并運行的大量測試用例。您可以在這里找到各種情況，例如0，大數和負數。

You might be thinking, it’s impressive, but I can’t find my favorite test case pair (1,2 ) here. What if I want that to run?

您可能會想，這令人印象深刻，但是我在這里找不到我最喜歡的測試用例對(1,2) 。如果我要運行該怎么辦？

Well, fear not, Hypothesis allows you to run a given set of test cases every time if you want by using the @example decorator.

好吧，不用擔心，假設可以讓您每次使用@ example裝飾器運行給定的測試用例集。

from hypothesis import given, settings, Verbosity, example

import hypothesis.strategies as stimport pytest

@settings(verbosity=Verbosity.verbose)@given(st.integers(), st.integers())@example(1, 2)def test_sum(num1, num2):    assert sum(num1, num2) == num1 + num2

Also, notice that each run will always generate a new jumbled up test case following the test generation strategy, thus randomizing the test run.

另外，請注意，每次運行將始終遵循測試生成策略來生成新的混亂測試用例，從而使測試運行隨機化。

So, this solves our first pain point- the exhaustiveness of test cases.

因此，這解決了我們的第一個痛點-測試用例的詳盡性。

努力思考要測試的屬性 (Thinking hard to come up with properties to test)

So far, we saw one magic of property-based testing which generates desired test data on the fly.

到目前為止，我們看到了基于屬性的測試的一種魔力，它可以動態生成所需的測試數據。

Now let’s come to the part where we need to think hard and in a different way to create such tests which are valid for all test inputs but unique to sum function.

現在讓我們進入需要認真思考的部分，以另一種方式來創建對所有測試輸入均有效但 sum函數唯一的測試。

1 + 0 = 10 + 1 = 15 + 0 = 5-3 + 0 = -38.5 + 0 = 8.5

Well, that’s interesting. It seems like adding 0 to a number results in the same number as sum. This is called the identity property of addition.

好吧，那很有趣。似乎在數字上加上0會得到與sum相同的數字。這稱為加法的標識屬性。

Let’s see one more:

讓我們再看一個：

2 + 3 = 53 + 2 = 5

5 + (-2) = 3-2 + 5 = 3

It looks like we found one more unique property. In addition the order of parameters doesn’t matter. Placed left or right of the + sign they give the same result. This is called the commutative property of addition.

看來我們找到了另一個獨特的屬性。另外，參數的順序無關緊要。在+號的左側或右側放置它們可得到相同的結果。這稱為加法的交換性質。

There is one more, but I want you to come up with it.

還有一個，但我希望您提出。

Now, we will re-write our test_sum to test these properties:

現在，我們將重新編寫test_sum以測試以下屬性：

from hypothesis import given, settings, Verbosity

import hypothesis.strategies as stimport pytest

@settings(verbosity=Verbosity.verbose)@given(st.integers(), st.integers())def test_sum(num1, num2):    assert sum(num1, num2) == num1 + num2

# Test Identity property    assert sum(num1, 0) = num1     #Test Commutative property      assert sum(num1, num2) == sum(num2, num1)

Our test is now exhaustive — we have also converted the tests to make them more robust. Thus, we solved our second pain point: non-robust test cases.

現在，我們的測試是詳盡無遺的-我們還對測試進行了轉換，以使其更加可靠。因此，我們解決了第二個痛點： 非健壯的測試用例 。

Just for curiosity’s sake, let’s try to fool this test with that buggy code we used before.

只是出于好奇，讓我們嘗試用之前使用的錯誤代碼來欺騙該測試。

As an old proverb says- fool me once, shame on you, fool me twice, shame on me.

就像一句古老的諺語所說：愚弄我一次，羞辱你，愚弄我兩次，羞辱我。

You can see that it caught an error. Falsifying example: test_sum(num1=0, num2=0). It simply means that our expected property didn't hold true for these pairs of test cases, thus the failure.

您可以看到它捕獲了一個錯誤。 Falsifying example: test_sum(num1=0, num2=0). 這僅表示我們的預期屬性不適用于這對測試用例，從而導致失敗。

But you wouldn’t clap yet. Because making something disappear isn’t enough; you have to bring it back. That’s why every magic trick has a third act, the hardest part, the part we call “The Prestige”.

但是您還不會鼓掌。因為使某事消失是不夠的。你必須把它帶回來。 這就是為什么每個魔術都有第三幕，最難的部分，我們稱之為“威望”的部分。

第3部分：收縮失敗 (Part 3: Shrinking failures)

Shrinking is the process by which Hypothesis tries to produce human-readable examples when it finds a failure. It takes a complex example and turns it into a simpler one.

縮小是假設在發現故障時試圖產生易于理解的示例的過程。它采用了一個復雜的示例，并將其變成一個簡單的示例。

To demonstrate this feature, let’s add one more property to our test_sum function which says num1 should be less than or equal to 30.

為了演示此功能，讓我們在test_sum函數中再添加一個屬性，該屬性表示num1應該小于或等于30.

from hypothesis import given, settings, Verbosity

import hypothesis.strategies as stimport pytest

@settings(verbosity=Verbosity.verbose)@given(st.integers(), st.integers())def test_sum(num1, num2):    assert sum(num1, num2) == num1 + num2

# Test Identity property    assert sum(num1, 0) = num1     #Test Commutative property      assert sum(num1, num2) == sum(num2, num1)    assert num1 <= 30

After running this test, you will get an interesting output log on the terminal here:

運行此測試后，您將在此處的終端上獲得有趣的輸出日志：

collected 1 item

test_example.py::test_sum Trying example: test_sum(num1=0, num2=-1)Trying example: test_sum(num1=0, num2=-1)Trying example: test_sum(num1=0, num2=-29696)Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=-1763, num2=47)Trying example: test_sum(num1=6, num2=1561)Trying example: test_sum(num1=-24900, num2=-29635)Trying example: test_sum(num1=-13783, num2=-20393)

#Till now all test cases passed but the next one will fail

Trying example: test_sum(num1=20251, num2=-10886)assert num1 <= 30AssertionError: assert 20251 <= 30

#Now the shrinking feature kicks in and it will try to find the simplest value for which the test still fails

Trying example: test_sum(num1=0, num2=-2)Trying example: test_sum(num1=0, num2=-1022)Trying example: test_sum(num1=-165, num2=-29724)Trying example: test_sum(num1=-14373, num2=-29724)Trying example: test_sum(num1=-8421504, num2=-8421376)Trying example: test_sum(num1=155, num2=-10886)assert num1 <= 30AssertionError: assert 155 <= 30

# So far it has narrowed it down to 155

Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=64, num2=0)assert num1 <= 30AssertionError: assert 64 <= 30

# Down to 64

Trying example: test_sum(num1=-30, num2=0)Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=31, num2=0)

# Down to 31

Trying example: test_sum(num1=-30, num2=0)Falsifying example: test_sum(num1=31, num2=0)FAILED

# And it finally concludes (num1=31, num2=0) is the simplest test data for which our property doesn't hold true.

One more good feature — its going to remember this failure for this test and will include this particular test case set in the future runs to make sure that the same regression doesn’t creep in.

另一個好功能- 它會記住此測試的失敗，并將在將來運行時包含此特定的測試用例集，以確保不會出現相同的回歸。

This was a gentle introduction to the magic of property based testing. I recommend all of you try this approach in your day to day testing. Almost all major programming languages have property based testing support.

這是對基于屬性的測試的神奇介紹。我建議大家在日常測試中嘗試這種方法。幾乎所有主要的編程語言都具有基于屬性的測試支持。

You can find the entire code used here in my ? github repo.

您可以在我的？中找到此處使用的完整代碼。 g ithub回購。

If you liked the content show some ??

如果您喜歡內容，請顯示一些??