Introduction

Developing Magento 2 Extensions is one thing, but ensuring they are ready for delivery to the customer is another.

Testing can be a time-consuming process that involves manual testing, raising issues, and then retesting the entire extension after each fix.

To address this challenge, third-party tools like Selenium, PhantomJS, and Codeception can be used to automate the testing process.

However, this still requires significant coding and regular updates whenever a new feature is added or an issue is fixed.

To overcome these challenges, Magento 2 provides its own functional testing framework. which is based on Selenium, Codeception, Allure, and other tools.

The MFTF is totally based on XML, so you only need to understand the XML tags and you can create automated test cases on the fly, you can check the MFTF Repository.

Getting Started

You can find the Magento Functional Testing Framework (MFTF) module located at {Magento_Root}/dev/tests/acceptance. This folder is available in Magento 2.2.4 and later versions.

Magento 2 already provides comprehensive documentation to help you get started with MFTF, so I won’t dive into the setup process here.

I’ll focus on explaining the structure of MFTF tests, particularly the XML files that define the tests.

To illustrate how everything works, I’ll walk you through an example using the Magento Customer module, specifically a customer login test case.

Before we jump into the actual code, let’s first take a look at the overall structure. You can organize and place your MFTF test cases within your custom module at the following location:

{Magento_Root}/app/code/Company/TestModule/Test/Mftf

Magento 2 default test cases reside in:{Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/

So for creating functional tests for your module, you will need to create a maximum of six folders in your MFTF module, let’s understand them one by one

Test:

Inside the ‘Test’ folder, we place our test cases. These test cases are simple XML files, with each file containing either multiple tests or just a single test.

Convention suggests placing one test per file, and the file name should match the test name, following camel casing.

A test case contains step-by-step actions, as well as setup and teardown processes. Setup is done inside a before tag, where you can prepare data or set up the environment for the test.

If changes were made to the environment, they can be cleaned up in the after tag.

After setup, the test case includes actions such as filling fields, loading URLs, clicking buttons, submitting forms, or adding attachments.

Assertions are then made to confirm that the test is successful, like verifying the page or checking for a success message in a specific HTML tag.

Within the test case, you can also define individual actions or group them together in ActionGroups for reusability.

All test cases written in XML are eventually converted into Codeception PHP classes. You can learn more about writing test cases in the official documentation.

Below is a simple example of a test case, in the customer module to test customer login

Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Test/StorefrontPersistedCustomerLoginTest.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!--
 /**
  * Copyright © Magento, Inc. All rights reserved.
  * See COPYING.txt for license details.
  */
-->

<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Test/etc/testSchema.xsd">
    <test name="StorefrontPersistedCustomerLoginTest">
        <annotations>

            <features value="Persisted customer can login from storefront."/>
            <stories value="Persisted customer can login from storefront."/>
            <title value="Persisted customer can login from storefront."/>
            <description value="Persisted customer can login from storefront."/>
            <severity value="CRITICAL"/>
            <testCaseId value="MAGETWO-72103"/>
            <group value="customer"/>
        </annotations>
        <before>
            <createData stepKey="customer" entity="Simple_US_Customer"/>
        </before>
        <after>
            <deleteData stepKey="deleteCustomer" createDataKey="customer" />
        </after>

        <amOnPage stepKey="amOnSignInPage"  url="{{StorefrontCustomerSignInPage.url}}"/>
        <fillField  stepKey="fillEmail" userInput="$customer.email$" selector="{{StorefrontCustomerSignInFormSection.emailField}}"/>
        <fillField  stepKey="fillPassword" userInput="$customer.password$" selector="{{StorefrontCustomerSignInFormSection.passwordField}}"/>
        <click stepKey="clickSignInAccountButton" selector="{{StorefrontCustomerSignInFormSection.signInAccountButton}}"/>
        <see stepKey="seeFirstName" userInput="$customer.firstname$" selector="{{StorefrontCustomerDashboardAccountInformationSection.ContactInformation}}" />
        <see stepKey="seeLastName" userInput="$customer.lastname$" selector="{{StorefrontCustomerDashboardAccountInformationSection.ContactInformation}}" />
        <see stepKey="seeEmail" userInput="$customer.email$" selector="{{StorefrontCustomerDashboardAccountInformationSection.ContactInformation}}" />
    </test>
</tests>

Let’s understand the tags and attributes in the above example:

tests: Container for test cases.

• test: test case a sequence of actions and assertions.
  • test “name” attribute: name of the test case for identity.
• annotations: annotations in each test case are used to add some explanation about the test, so anyone can easily understand the test case when the test cases are converted to codeception test case it is converted to a PHP comment preceding @.
  • annotations “testCaseId” attribute: this attribute is used to set test case id it must be unique for all test cases.
  • annotation “group” attribute: this attribute is used to execute the test cases by a group like if two or many test cases have the same group “customer”, we can execute only those test cases which have a group name “customer” with this command: vendor/bin/codecept run functional --group customer
• before: this tag is used to create test data to set up any initial environment.
  • createData: operation to create data using REST API.
    • createData “entity” attribute: data entity that will be used in the creation of the data defined in Data folder, and the entity type will be used to get the operation defined in the MetaData folder meta files.
    • createData “stepKey” attribute: name of the step so it can be referenced in other actions or operations.
• after: this tag is used to clean the data created in the before tag or remove any environment settings.
  • deleteData: delete operation for data created in before tag.
    • deleteData “createDataKey” attribute: reference the stepKey in the createData key in the before tag.
• amOnPage: used to navigate pages.
  • amOnPage “url” attribute: URL of the page to open.
• fillField: used to fill form fields on the page.
  • fillField “userInput” attribute: provide user input to fill in the field, in this case, we are taking data from the customer entity created in before tag using its step key.
  • fillField “selector” attribute: provide a selector to find the field which needs to be filled.
• see: this is an assertion tag to check if expected and actual output is equal if: equal test is passed else: test fails.
  • see “userInput” attribute: this attribute is used to provide the expected output.
  • see “selector” attribute: this is used to provide the element selector to get the actual output in the HTML page.

Page:

Inside the “Page” folder, we define the pages that will be visited during the test case execution.

The naming convention for these page files is similar to test cases: the file name and page name must be the same, follow camel casing, and end with the “Page” suffix.

A page can have multiple sections, and each section will contain several elements, such as <div>, <a>, <form>, <button>, <checkbox>, and so on.

These elements represent the various components or controls on the page that we will interact with during the test.

For example, in the customer module, a page might represent the customer login form.

This page will have sections for the username and password fields, a login button, and any error messages that might appear.

Each of these elements will be defined within the page to facilitate easy interaction during test execution.

Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Page/StorefrontCustomerSignInPage.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!--
 /**
  * Copyright © Magento, Inc. All rights reserved.
  * See COPYING.txt for license details.
  */
-->

<pages xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/PageObject.xsd">
    <page name="StorefrontCustomerSignInPage" url="/customer/account/login/" area="storefront" module="Magento_Customer">
        <section name="StorefrontCustomerSignInFormSection" />
    </page>
</pages>

Let’s understand the tags and attributes in the above example:

pages: page container.

• page: a page which can be referenced with a URL.
  • page “name” attribute: page to uniquely identify the page.
  • page “url” attribute: relative URL for the page.
  • page “area” attribute: an area where the page exists, adminhtml or frontend.
  • page “module” attribute: name the module in which the page is defined.

Section:

A section is a part of a page like a header, footer, content, sidebar etc.

You can divide the page into as many sections as you like, although you can create one section and define all the elements there but, it will not be readable and understandable.

A section looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!--
 /**
  * Copyright © Magento, Inc. All rights reserved.
  * See COPYING.txt for license details.
  */
-->

<sections xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/SectionObject.xsd">
    <section name="StorefrontCustomerCreateFormSection">
        <element name="firstnameField" type="input" selector="#firstname"/>
        <element name="lastnameField" type="input" selector="#lastname"/>
        <element name="emailField" type="input" selector="#email_address"/>
        <element name="passwordField" type="input" selector="#password"/>
        <element name="confirmPasswordField" type="input" selector="#password-confirmation"/>
        <element name="createAccountButton" type="button" selector="button.action.submit.primary" timeout="30"/>
    </section>
</sections>

Above is an example you can find:

Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Section/StorefrontCustomerCreateFormSection.xml. It has various tags and attributes:

sections: a container for the section.

• section: individual section on the page.
  • section “name” attribute:  name of the section same as file name, it is used to uniquly identify the section.
• element: element is individual elements on the page like div, button, form fields(input, textarea, select etc) etc.
  • element “selector” attribute: this is a CSS selector of the element.
  • element “name” attribute: name of the element to uniquely identify the element.
  • element “type” attribute: type of HTML element it can be:

<xs:simpleType name="uiElementType">
        <xs:restriction base="xs:string">
            <xs:enumeration value="text"/>
            <xs:enumeration value="textarea"/>
            <xs:enumeration value="input"/>
            <xs:enumeration value="button"/>
            <xs:enumeration value="checkbox"/>
            <xs:enumeration value="radio"/>
            <xs:enumeration value="checkboxset"/>
            <xs:enumeration value="radioset"/>
            <xs:enumeration value="date"/>
            <xs:enumeration value="file"/>
            <xs:enumeration value="select"/>
            <xs:enumeration value="multiselect"/>
            <xs:enumeration value="wysiwyg"/>
            <xs:enumeration value="iframe"/>
            <xs:enumeration value="block"/>
        </xs:restriction>
</xs:simpleType>

according to

Magento_Base_Dir/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/SectionObject.xsd

this is used to validate XML schema of section.

ActionGroup:

An action group is a group of actions  (like click on the button, page load, etc).

It is very useful to create a group of actions for better reusability, like LoginToStorefrontActionGroup, SignUpNewUserFromStorefrontActionGroup in customer test module.

Many test cases require the customer to log in or register. Without action groups, you’d end up repeating the same series of actions in every test case.

To resolve this Magento 2 provides Acton Groups.

An action group XML resides in

Magento_Base_Dir/app/code/CompanyName/ModuleName/Test/Mftf/ActionGroup, an action group must end with the suffix “ActionGroup”.

Here is an example of

Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/ActionGroup/LoginToStorefrontActionGroup.xml :

<?xml version="1.0" encoding="UTF-8"?>
<!--
 /**
  * Copyright © Magento, Inc. All rights reserved.
  * See COPYING.txt for license details.
  */
-->
<actionGroups xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Test/etc/actionGroupSchema.xsd">
    <actionGroup name="LoginToStorefrontActionGroup">
        <arguments>
            <argument name="Customer"/>
        </arguments>
        <amOnPage stepKey="amOnSignInPage"  url="{{StorefrontCustomerSignInPage.url}}"/>
        <fillField  stepKey="fillEmail" userInput="{{Customer.email}}" selector="{{StorefrontCustomerSignInFormSection.emailField}}"/>
        <fillField  stepKey="fillPassword" userInput="{{Customer.password}}" selector="{{StorefrontCustomerSignInFormSection.passwordField}}"/>
        <click stepKey="clickSignInAccountButton" selector="{{StorefrontCustomerSignInFormSection.signInAccountButton}}"/>
    </actionGroup>
</actionGroups>

Let’s understand the tags and attributes in the above example:

actionGroups: container for action group.

• actionGroup: container for actions.
  • actionGroup “name” attribute: name to uniquely identify the action group.
• arguments: container for any data passed in to the “actionGroup”.
  • argument: data to use in this action.
    • argument “name” attribute: name of the data entity, defined inside Data folder, explained below.
• amOnPage: amOnPage tag is the name of the action to goto a page.
  amOnPage “stepKey” attribute: it is used as reference of the action.
  • amOnPage “url” attribute: used to define which page to load, url leads to a page we can get the page url  simple addressing it like this: “{{StorefrontCustomerSignInPage.url}}”, StorefrontCustomerSignInPage is the page name and URL is its attribute.
• fillField: this tag is used to fill any form field.
  • fillField “userInput” attribute: the data need to be filled in the field, data can be referenced from the data entity that is passed as an argument like this: “{{Customer.email}}”, here Customer is the data entity and email is the attribute.
  • fillField “selector” attribute: this attribute helps select the element in which data is to be filled, in above example we have passed the selector like this: “{{StorefrontCustomerSignInFormSection.emailField}}”, here StorefrontCustomerSignInFormSection is the section name and emailField is the element name.

Data:

Any test case will need some dummy data for completing the test case, like dummy data for a product, to test product create flow, dummy data for the customer for registration and login feature test.

All the data XML files must end with a Data suffix.

Let’s take an example for customer entity data

Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Data/CustomerData.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!--
 /**
  * Copyright © Magento, Inc. All rights reserved.
  * See COPYING.txt for license details.
  */
-->

<entities xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/etc/dataProfileSchema.xsd">
    <entity name="CustomerEntityOne" type="customer">
        <data key="group_id">0</data>
        <data key="default_billing">defaultBillingValue</data>
        <data key="default_shipping">defaultShippingValue</data>
        <data key="confirmation">confirmationData</data>
        <data key="created_at">12:00</data>
        <data key="updated_at">12:00</data>
        <data key="created_in">createdInData</data>
        <data key="dob">01-01-1970</data>
        <data key="email" unique="prefix">test@email.com</data>
        <data key="firstname">John</data>
        <data key="lastname">Doe</data>
        <data key="middlename">S</data>
        <data key="password">pwdTest123!</data>
        <data key="prefix">Mr</data>
        <data key="suffix">Sr</data>
        <data key="gender">0</data>
        <data key="store_id">0</data>
        <data key="taxvat">taxValue</data>
        <data key="website_id">0</data>
        <requiredEntity type="address">CustomerAddressSimple</requiredEntity>
        <data key="disable_auto_group_change">0</data>
        <!--requiredEntity type="extension_attribute">ExtensionAttributeSimple</requiredEntity-->
    </entity>
    <entity name="Simple_US_Customer" type="customer">
        <data key="group_id">0</data>
        <data key="default_billing">true</data>
        <data key="default_shipping">true</data>
        <data key="email" unique="prefix">John.Doe@example.com</data>
        <data key="firstname">John</data>
        <data key="lastname">Doe</data>
        <data key="fullname">John Doe</data>
        <data key="password">pwdTest123!</data>
        <data key="store_id">0</data>
        <data key="website_id">0</data>
        <requiredEntity type="address">US_Address_TX</requiredEntity>
    </entity>
    <entity name="Simple_US_Customer_For_Update" type="customer">
        <var key="id" entityKey="id" entityType="customer"/>
        <data key="firstname">Jane</data>
    </entity>
</entities>

Above is an example for a customer data entity, let’s understand the XML tags and attributes:

entities: it is a container for data entity.

• entity: individual data entity of the customer.
  • entity “name” attribute: entity name attribute is to identify data entity uniquely.
  • entity “type” attribute: entity type is the type of entity which is used in metadata to identify the entity operation.
• data: the data tag is to set dummy data value which will be used in test cases.
  • data “key” attribute: key attribute is used to define the data key name.
• requiredEntity: for referencing another data entity inside a data entity.
• var: var tag is used to hold return data value after a certain action like customer saves, delete etc is performed.
  • var “key” attribute: name of the data key of the current entity.
  • var “entityKey” attribute: name of the data key from the referenced entity.
  • var “entityType” attribute: name of the data type from the referenced entity.

Metadata:

While executing test cases sometimes the tester might need some data entities to be created at runtime and used in the test or after the test is complete or delete all the created data.

This can be done using metadata.

You can create operations like create, delete, update, get.

Each metadata can be only created for each data entity type defined, for customer data entity we can create metadata in customer-meta.xml.

Let’s understand it by looking at Magento_Base_Dir/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Customer/Metadata/customer-meta.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
 /**
  * Copyright © Magento, Inc. All rights reserved.
  * See COPYING.txt for license details.
  */
-->

<operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/etc/dataOperation.xsd">
    <operation name="CreateCustomer" dataType="customer" type="create" auth="adminOauth" url="/V1/customers" method="POST">
        <contentType>application/json</contentType>
        <object dataType="customer" key="customer">
            <field key="group_id">integer</field>
            <field key="default_billing">string</field>
            <field key="default_shipping">string</field>
            <field key="confirmation">string</field>
            <field key="created_at">string</field>
            <field key="updated_at">string</field>
            <field key="created_in">string</field>
            <field key="dob">string</field>
            <field key="email">string</field>
            <field key="firstname">string</field>
            <field key="lastname">string</field>
            <field key="middlename">string</field>
            <field key="prefix">string</field>
            <field key="suffix">string</field>
            <field key="gender">integer</field>
            <field key="store_id">integer</field>
            <field key="taxvat">string</field>
            <field key="website_id">integer</field>
            <array key="addresses">
                <value>address</value>
            </array>
            <field key="disable_auto_group_change">integer</field>
            <field key="extension_attributes">customer_extension_attribute</field>
            <array key="custom_attributes">
                <value>custom_attribute</value>
            </array>
        </object>
        <field key="password">string</field>
        <field key="redirectUrl">string</field>
    </operation>
    <operation name="DeleteCustomer" dataType="customer" type="delete" auth="adminOauth" url="/V1/customers/{id}" method="DELETE">
        <contentType>application/json</contentType>
    </operation>
</operations>

let’s understand the tags and attributes in the above example:

operations: this tag is the container of all the operations on the data entity( in the above case customer type data entity)

• operation: an operation can be: create, update, delete or get.
  • operation “name” attribute: name of the operation to uniquely identify the operation.
  • operation “dataType” attribute: data entity type required to perform the operation.
  • operation “type” attribute: type of operation “create”, “delete”, “update” or “get”.
  • operation “auth” attribute:  Authentication required to perform the operation values can be “adminOauth”, “adminFormKey”, “customerFormKey”, “anonymous”
  • operation “url” attribute: REST API endpoint.
  • operation “method” attribute: Method type for REST API endpoint values can be POST, DELETE, PUT, GET etc.
• contentType: data request header contentType value.
• object: object tag is to define the data to be posted in the REST API endpoint.
  • object “dataType” attribute: data type of the object.
  • object “key” attribute:  object name since there can be multiple objects in the operation.
  • field: field is the data field type that needs to send in the REST API call.
  • field “key” attribute: to identify the key.
  • array: to send array data type in the request.
    • array “key” attribute: name to identify the array.
    • value: contains the type of data entity type.

Hope the above explanation, could have added some idea about the new functional testing framework how it manages everything with XML files, the relation between the XML files.

Next time I will post how to create the test cases for your custom module.

If you have any doubts about the above explanation please ask in the comments.