Introduction
In this article, we will try to understand Magento2 Functional Testing Framework(MFTF), developing magento2 extensions is one thing and making sure it is ready to deliver to the customer is another, it can be a time taking process of testing manually and raising issues then after the fix again testing the whole extension. To solve this issue we can use some third-party tools(like selenium, phantomjs, codeception etc) to automate the system, but still, you will have to code a lot and need to update that code each time a new feature added or any issue fixed but to overcome all these issues Magento has provided its own functional testing framework, although it is nothing new in Magento, it has recently launched it’s latest functional testing framework which is based on selenium, codeception, allure etc, we will try to understand how it works and how to get started with it.
The new MFTF is totally based on XML, you don’t need to write a single line of PHP, so you only need to understand the XML tags and you can create automated test cases on the fly, you can find the new MFTF on git https://github.com/magento/magento2-functional-testing-framework/releases/tag/2.2.0.
Getting Started
You can find the new MFTF module at {Magento_Root}/dev/tests/acceptance, this folder is only available in magento2 latest version(2.2.4 and above). Magento2 has already provided a good documentation on getting started with this, you can learn all the configuration and settings from here:
So I am not going to explain how to set up MFTF, you can learn it from the above link, I will explain the test structure, all the XML files that we create in MFTF test.
I will explain the test case example for Magento Customer module customer login.
But before getting into the actual code, let’s understand it’s structure, You can put the MFTF test cases inside your custom module at this location:
{Magento_Root}/app/code/Company/TestModule/Test/Mftf
Magento default test cases reside in :{Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/
So for creating functional tests for your module, you will need to create a maximum of six folders in your MFTF module, let’s understand all the six folders one by one:
Test:
Inside ‘Test’ folder, we will put our test cases, the test cases are nothing but simple XML files each file can contain multiple tests or a single test, convention says that you should put one test in each file and file name and test name must be the same and the name should follow camel casing. A test case contains step by step actions and some setup and teardown, like any other test case you can set up some data, or some environment to initiate the test, setup is done inside a before tag, and if we set up some environment for test sometimes we have to remove it too after the test, that we can do inside an after tag. After that we can write actions like filling a field, loading a URL, clicking a button, submitting a form, adding an attachment then some assertions to make sure tests are being successful, like after clicking the save button asserting that being on this page, or to check to see if success message has appeared inside particular HTML tag. Inside the test case, we can also define single actions or specify a group of actions(will reside inside ActionGroup Folder) for reusability. All the test cases written in XML are converted into codeception php classes. You can learn more about test cases here:
below is the simple example of a test case, in customer module to test customer login Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Test/StorefrontPersistedCustomerLoginTest.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!--
/**
* Copyright © Magento, Inc. All rights reserved.
* See COPYING.txt for license details.
*/
-->
<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Test/etc/testSchema.xsd">
<test name="StorefrontPersistedCustomerLoginTest">
<annotations>
<features value="Persisted customer can login from storefront."/>
<stories value="Persisted customer can login from storefront."/>
<title value="Persisted customer can login from storefront."/>
<description value="Persisted customer can login from storefront."/>
<severity value="CRITICAL"/>
<testCaseId value="MAGETWO-72103"/>
<group value="customer"/>
</annotations>
<before>
<createData stepKey="customer" entity="Simple_US_Customer"/>
</before>
<after>
<deleteData stepKey="deleteCustomer" createDataKey="customer" />
</after>
<amOnPage stepKey="amOnSignInPage" url="{{StorefrontCustomerSignInPage.url}}"/>
<fillField stepKey="fillEmail" userInput="$customer.email$" selector="{{StorefrontCustomerSignInFormSection.emailField}}"/>
<fillField stepKey="fillPassword" userInput="$customer.password$" selector="{{StorefrontCustomerSignInFormSection.passwordField}}"/>
<click stepKey="clickSignInAccountButton" selector="{{StorefrontCustomerSignInFormSection.signInAccountButton}}"/>
<see stepKey="seeFirstName" userInput="$customer.firstname$" selector="{{StorefrontCustomerDashboardAccountInformationSection.ContactInformation}}" />
<see stepKey="seeLastName" userInput="$customer.lastname$" selector="{{StorefrontCustomerDashboardAccountInformationSection.ContactInformation}}" />
<see stepKey="seeEmail" userInput="$customer.email$" selector="{{StorefrontCustomerDashboardAccountInformationSection.ContactInformation}}" />
</test>
</tests>
Let’s understand the tags and attributes in the above example:
tests: Container for test cases.
- test: test case a sequence of actions and assertions.
- test “name” attribute: name of the test case for identity.
- annotations: annotations in each test case are used to add some explanation about the test, so anyone can easily understand the test case when the test cases are converted to codeception test case it is converted to a PHP comment preceding @.
- annotations “testCaseId” attribute: this attribute is used to set test case id it must be unique for all test cases.
- annotation “group” attribute: this attribute is used to execute the test cases by a group like if two or many test cases have the same group “customer”, we can execute only those test cases which have a group name “customer” with this command:
vendor/bin/codecept run functional --group customer
- before: this tag is used to create test data to set up any initial environment.
- createData: operation to create data using REST API.
- createData “entity” attribute: data entity that will be used in the creation of the data defined in Data folder, and the entity type will be used to get the operation defined in the MetaData folder meta files.
- createData “stepKey” attribute: name of the step so it can be referenced in other actions or operations.
- createData: operation to create data using REST API.
- after: this tag is used to clean the data created in the before tag or remove any environment settings.
- deleteData: delete operation for data created in before tag.
- deleteData “createDataKey” attribute: reference the stepKey in the createData key in the before tag.
- deleteData: delete operation for data created in before tag.
- amOnPage: used to navigate pages.
- amOnPage “url” attribute: URL of the page to open.
- fillField: used to fill form fields on the page.
- fillField “userInput” attribute: provide user input to fill in the field, in this case, we are taking data from the customer entity created in before tag using its step key.
- fillField “selector” attribute: provide a selector to find the field which needs to be filled.
- see: this is an assertion tag to check if expected and actual output is equal if: equal test is passed else: test fails.
- see “userInput” attribute: this attribute is used to provide the expected output.
- see “selector” attribute: this is used to provide the element selector to get the actual output in the HTML page.
Page:
Inside page folder we can define pages that will be visited during test case execution, the naming convention is similar to test cases, file name and page name must be the same, and the name will be following camel casing and all the names must end with “Page” suffix. A page can have multiple sections and each section will have multiple elements(like div, anchor, form, button, checkbox etc). A page example, from customer module Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Page/StorefrontCustomerSignInPage.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!--
/**
* Copyright © Magento, Inc. All rights reserved.
* See COPYING.txt for license details.
*/
-->
<pages xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/PageObject.xsd">
<page name="StorefrontCustomerSignInPage" url="/customer/account/login/" area="storefront" module="Magento_Customer">
<section name="StorefrontCustomerSignInFormSection" />
</page>
</pages>
Let’s understand the tags and attributes in the above example:
pages: page container.
- page: a page which can be referenced with a URL.
- page “name” attribute: page to uniquely identify the page.
- page “url” attribute: relative URL for the page.
- page “area” attribute: an area where the page exists, adminhtml or frontend.
- page “module” attribute: name the module in which the page is defined.
learn more about pages:
Section:
A section is a part of a page like a header, footer, content, sidebar etc. You can divide the page into as many sections as you like, although you can create one section and define all the elements there but, it will not be readable and understandable. A section looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<!--
/**
* Copyright © Magento, Inc. All rights reserved.
* See COPYING.txt for license details.
*/
-->
<sections xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/SectionObject.xsd">
<section name="StorefrontCustomerCreateFormSection">
<element name="firstnameField" type="input" selector="#firstname"/>
<element name="lastnameField" type="input" selector="#lastname"/>
<element name="emailField" type="input" selector="#email_address"/>
<element name="passwordField" type="input" selector="#password"/>
<element name="confirmPasswordField" type="input" selector="#password-confirmation"/>
<element name="createAccountButton" type="button" selector="button.action.submit.primary" timeout="30"/>
</section>
</sections>
Above is an example you can find: Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Section/StorefrontCustomerCreateFormSection.xml. It has various tags and attributes:
sections: a container for the section.
- section: individual section on the page.
- section “name” attribute: name of the section same as file name, it is used to uniquly identify the section.
- element: element is individual elements on the page like div, button, form fields(input, textarea, select etc) etc.
- element “selector” attribute: this is a CSS selector of the element.
- element “name” attribute: name of the element to uniquely identify the element.
- element “type” attribute: type of HTML element it can be:
<xs:simpleType name="uiElementType">
<xs:restriction base="xs:string">
<xs:enumeration value="text"/>
<xs:enumeration value="textarea"/>
<xs:enumeration value="input"/>
<xs:enumeration value="button"/>
<xs:enumeration value="checkbox"/>
<xs:enumeration value="radio"/>
<xs:enumeration value="checkboxset"/>
<xs:enumeration value="radioset"/>
<xs:enumeration value="date"/>
<xs:enumeration value="file"/>
<xs:enumeration value="select"/>
<xs:enumeration value="multiselect"/>
<xs:enumeration value="wysiwyg"/>
<xs:enumeration value="iframe"/>
<xs:enumeration value="block"/>
</xs:restriction>
</xs:simpleType>
according to Magento_Base_Dir/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/SectionObject.xsd this is used to validate XML schema of section.
ActionGroup:
An action group is the group of actions (like click on the button, page load etc), it is very useful to create group of actions for better reusability, like LoginToStorefrontActionGroup, SignUpNewUserFromStorefrontActionGroup in customer test module, in many test cases it must require the customer to login or register, if you don’t have specific way to define a group of actions you will end up writing the same series of actions in all the test cases to resolve this Magento provides Acton Groups. An action group XML resides in Magento_Base_Dir/app/code/CompanyName/ModuleName/Test/Mftf/ActionGroup, an action group must end with suffix “ActionGroup”. Here is an example of Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/ActionGroup/LoginToStorefrontActionGroup.xml :
<?xml version="1.0" encoding="UTF-8"?>
<!--
/**
* Copyright © Magento, Inc. All rights reserved.
* See COPYING.txt for license details.
*/
-->
<actionGroups xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Test/etc/actionGroupSchema.xsd">
<actionGroup name="LoginToStorefrontActionGroup">
<arguments>
<argument name="Customer"/>
</arguments>
<amOnPage stepKey="amOnSignInPage" url="{{StorefrontCustomerSignInPage.url}}"/>
<fillField stepKey="fillEmail" userInput="{{Customer.email}}" selector="{{StorefrontCustomerSignInFormSection.emailField}}"/>
<fillField stepKey="fillPassword" userInput="{{Customer.password}}" selector="{{StorefrontCustomerSignInFormSection.passwordField}}"/>
<click stepKey="clickSignInAccountButton" selector="{{StorefrontCustomerSignInFormSection.signInAccountButton}}"/>
</actionGroup>
</actionGroups>
Let’s understand the tags and attributes in the above example:
actionGroups: container for action group.
- actionGroup: container for actions.
- actionGroup “name” attribute: name to uniquely identify the action group.
- arguments: container for any data passed in to the “actionGroup”.
- argument: data to use in this action.
- argument “name” attribute: name of the data entity, defined inside Data folder, explained below.
- argument: data to use in this action.
- amOnPage: amOnPage tag is the name of the action to goto a page.
amOnPage “stepKey” attribute: it is used as reference of the action.- amOnPage “url” attribute: used to define which page to load, url leads to a page we can get the page url simple addressing it like this: “{{StorefrontCustomerSignInPage.url}}”, StorefrontCustomerSignInPage is the page name and URL is its attribute.
- fillField: this tag is used to fill any form field.
- fillField “userInput” attribute: the data need to be filled in the field, data can be referenced from the data entity that is passed as an argument like this: “{{Customer.email}}”, here Customer is the data entity and email is the attribute.
- fillField “selector” attribute: this attribute helps select the element in which data is to be filled, in above example we have passed the selector like this: “{{StorefrontCustomerSignInFormSection.emailField}}”, here StorefrontCustomerSignInFormSection is the section name and emailField is the element name.
Data:
Any test case will need some dummy data for completing the test case, like dummy data for a product, to test product create flow, dummy data for the customer for registration and login feature test. All the data XML files must end with Data suffix. let’s take an example for customer entity data Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Data/CustomerData.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!--
/**
* Copyright © Magento, Inc. All rights reserved.
* See COPYING.txt for license details.
*/
-->
<entities xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/etc/dataProfileSchema.xsd">
<entity name="CustomerEntityOne" type="customer">
<data key="group_id">0</data>
<data key="default_billing">defaultBillingValue</data>
<data key="default_shipping">defaultShippingValue</data>
<data key="confirmation">confirmationData</data>
<data key="created_at">12:00</data>
<data key="updated_at">12:00</data>
<data key="created_in">createdInData</data>
<data key="dob">01-01-1970</data>
<data key="email" unique="prefix">[email protected]</data>
<data key="firstname">John</data>
<data key="lastname">Doe</data>
<data key="middlename">S</data>
<data key="password">pwdTest123!</data>
<data key="prefix">Mr</data>
<data key="suffix">Sr</data>
<data key="gender">0</data>
<data key="store_id">0</data>
<data key="taxvat">taxValue</data>
<data key="website_id">0</data>
<requiredEntity type="address">CustomerAddressSimple</requiredEntity>
<data key="disable_auto_group_change">0</data>
<!--requiredEntity type="extension_attribute">ExtensionAttributeSimple</requiredEntity-->
</entity>
<entity name="Simple_US_Customer" type="customer">
<data key="group_id">0</data>
<data key="default_billing">true</data>
<data key="default_shipping">true</data>
<data key="email" unique="prefix">[email protected]</data>
<data key="firstname">John</data>
<data key="lastname">Doe</data>
<data key="fullname">John Doe</data>
<data key="password">pwdTest123!</data>
<data key="store_id">0</data>
<data key="website_id">0</data>
<requiredEntity type="address">US_Address_TX</requiredEntity>
</entity>
<entity name="Simple_US_Customer_For_Update" type="customer">
<var key="id" entityKey="id" entityType="customer"/>
<data key="firstname">Jane</data>
</entity>
</entities>
Above is the example for customer data entity, let’s understand the XML tags and attributes:
entities: it is a container for data entity.
- entity: individual data entity of the customer.
- entity “name” attribute: entity name attribute is to identify data entity uniquely.
- entity “type” attribute: entity type is the type of entity which is used in metadata to identify the entity operation.
- data: the data tag is to set dummy data value which will be used in test cases.
- data “key” attribute: key attribute is used to define the data key name.
- requiredEntity: for referencing another data entity inside a data entity.
- var: var tag is used to hold return data value after a certain action like customer saves, delete etc is performed.
- var “key” attribute: name of the data key of the current entity.
- var “entityKey” attribute: name of the data key from the referenced entity.
- var “entityType” attribute: name of the data type from the referenced entity.
Metadata:
While executing test cases sometimes tester might need some data entities to be created at runtime and used in the test or after the test complete delete all the created data, this can be done using metadata. You can create operations like create, delete, update, get. Each metadata can be only created for each data entity type defined, for example above we understood by the example for “customer” data entity type, for customer data entity we can create metadata in customer-meta.xml, let’s understand it by looking at Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Metadata/customer-meta.xml
<?xml version="1.0" encoding="UTF-8"?>
<!--
/**
* Copyright © Magento, Inc. All rights reserved.
* See COPYING.txt for license details.
*/
-->
<operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/etc/dataOperation.xsd">
<operation name="CreateCustomer" dataType="customer" type="create" auth="adminOauth" url="/V1/customers" method="POST">
<contentType>application/json</contentType>
<object dataType="customer" key="customer">
<field key="group_id">integer</field>
<field key="default_billing">string</field>
<field key="default_shipping">string</field>
<field key="confirmation">string</field>
<field key="created_at">string</field>
<field key="updated_at">string</field>
<field key="created_in">string</field>
<field key="dob">string</field>
<field key="email">string</field>
<field key="firstname">string</field>
<field key="lastname">string</field>
<field key="middlename">string</field>
<field key="prefix">string</field>
<field key="suffix">string</field>
<field key="gender">integer</field>
<field key="store_id">integer</field>
<field key="taxvat">string</field>
<field key="website_id">integer</field>
<array key="addresses">
<value>address</value>
</array>
<field key="disable_auto_group_change">integer</field>
<field key="extension_attributes">customer_extension_attribute</field>
<array key="custom_attributes">
<value>custom_attribute</value>
</array>
</object>
<field key="password">string</field>
<field key="redirectUrl">string</field>
</operation>
<operation name="DeleteCustomer" dataType="customer" type="delete" auth="adminOauth" url="/V1/customers/{id}" method="DELETE">
<contentType>application/json</contentType>
</operation>
</operations>
let’s understand the tags and attributes in the above example:
operations: this tag is the container of all the operations on the data entity( in the above case customer type data entity)
- operation: an operation can be: create, update, delete or get.
- operation “name” attribute: name of the operation to uniquely identify the operation.
- operation “dataType” attribute: data entity type required to perform the operation.
- operation “type” attribute: type of operation “create”, “delete”, “update” or “get”.
- operation “auth” attribute: Authentication required to perform the operation values can be “adminOauth”, “adminFormKey”, “customerFormKey”, “anonymous”
- operation “url” attribute: REST API endpoint.
- operation “method” attribute: Method type for REST API endpoint values can be POST, DELETE, PUT, GET etc.
- contentType: data request header contentType value.
- object: object tag is to define the data to be posted in the REST API endpoint.
- object “dataType” attribute: data type of the object.
- object “key” attribute: object name since there can be multiple objects in the operation.
- field: field is the data field type that needs to send in the REST API call.
- field “key” attribute: to identify the key.
- array: to send array data type in the request.
- array “key” attribute: name to identify the array.
- value: contains the type of data entity type.
Hope the above explanation, could have added some idea about the new functional testing framework how it manages everything with XML files, the relation between the XML files.
Next time I will post how to create the test cases for your custom module.
If you have any doubts about the above explanation please ask in the comments.
2 comments