Magento 2: How to handle huge result sets

Hello friends !!!

We often use collections to fetch and work with data from our database in Magento 2 and mostly these work perfectly fine,
but when we talk to fetch and iterate a huge number of records in a single process it becomes extremely difficult because memory usage becomes a problem.
we can face most of the problems while dealing with large collections. so we need to look for an alternative solution. In this article, I’m going to talk about how to handle huge result sets in Magento2.

Querying and loading a huge number of records…

(Magento\Framework\Model\ResourceModel\Iterator) that enables us to iterate huge database result sets by applying callbacks to each row in the result. This is perfect when you have millions of records that you need to process without loading them all into memory.

The purpose here is to only ever load a single row into memory at any time (unless you assign it to a variable outside the scope of the callback function passed to $iterator->walk()). If we achieve this then it does not matter how many rows we need to process, it should not increase our memory consumption.

Let’s understand this in an example:

<?php

use Magento\Framework\App\ResourceConnection;
use Magento\Framework\Model\ResourceModel\IteratorFactory;

class IteratorExample
{
    /** @var ResourceConnection */
    private $resourceConnection;
    
    /** @var IteratorFactory */
    private $iteratorFactory;

    public function __construct(ResourceConnection $resourceConnection, IteratorFactory $iteratorFactory)
    {
        $this->resourceConnection = $resourceConnection;
        $this->iteratorFactory = $iteratorFactory;
    }
    
    public function processResults(): void
    {
        $connection = $this->resourceConnection->getConnection();
        $iterator = $this->iteratorFactory->create();
        
        $query = $connection->select()->from('table_name');

        $iterator->walk((string) $query, [function (array $result) {
            // do something with $result
        }], [], $connection);
    }
}

Hope this will be helpful.
Thanks