Fixed Thread Cache Rebuild - Died without 1024MB

[Ryan McAdams]

So I was doing a cache rebuild on a forum with just over 3M threads and it died repeatedly telling me it broke the cap of 128MB for php's memory limit.

Error here:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /public_html/community/library/Zend/Db/Statement/Mysqli.php on line 295

So I did some digging because my php memory limit was set to 256MB on my server and found that XenForo was overriding the php memory limit in it's Application.php file:

        public function beginApplication($configDir = '.', $rootDir = '.', $loadDefaultData = true)
        {
                if ($this->_initialized)
                {
                        return;
                }

                if (function_exists('get_magic_quotes_gpc') && get_magic_quotes_gpc())
                {
                        self::undoMagicQuotes($_GET);
                        self::undoMagicQuotes($_POST);
                        self::undoMagicQuotes($_COOKIE);
                        self::undoMagicQuotes($_REQUEST);
                }
                if (function_exists('set_magic_quotes_runtime'))
                {
                        set_magic_quotes_runtime(false);
                }

                @ini_set('memory_limit', 128 * 1024 * 1024);
                ignore_user_abort(true);

                @ini_set('output_buffering', false);
                while (@ob_end_clean());

                error_reporting(E_ALL | E_STRICT & ~8192);
                @ini_set('display_errors', true);
                set_error_handler(array('XenForo_Application', 'handlePhpError'));
                set_exception_handler(array('XenForo_Application', 'handleException'));

                date_default_timezone_set('UTC');

                self::$time = time();

                self::$host = (empty($_SERVER['HTTP_HOST']) ? '' : $_SERVER['HTTP_HOST']);

                require(XenForo_Autoloader::getInstance()->autoloaderClassToFile('Lgpl_utf8'));

                $this->_configDir = $configDir;
                $this->_rootDir = $rootDir;
                $this->addLazyLoader('requestPaths', array($this, 'loadRequestPaths'));

                if ($loadDefaultData)
                {
                        $this->loadDefaultData();
                }

                $this->_initialized = true;
        }

The line in question is:

@ini_set('memory_limit', 128 * 1024 * 1024);

Now normally this would probably work and be no big deal... however my cache rebuild (again threads) would not complete with this limit. It broke on 256MB also so I decided to set it to 1024.

When PHP's limit was 1 gig it worked fine and the rebuild ran all the way through... (please note I did not try 512).

I figured I should report this

Thanks,
Ryan

 

[Mike]

Was this with debug mode on?

It's possibly related to you mentioning that you have a thread with 250,000 posts. That wasn't really an expectation of the system. It's worth looking into.

 

[Paul B]

I think debug might have been on a I suggested it to try and catch any errors related to the original import.

As it turns out, it wasn't stuck, just processing that huge thread.

 

[Ryan McAdams]

Was this with debug mode on?

It's possibly related to you mentioning that you have a thread with 250,000 posts. That wasn't really an expectation of the system. It's worth looking into.

Debug mode was on Mike - Sorry for not answering that.

I'm more than willing to help any way I can Just tell me what you'd like me to do!

 

[Mike]

A fix to part of this issue is here:
http://xenforo.com/community/thread...g-multiple-posts-in-very-large-threads.18286/
It may help, though with a 250,000 post thread, you will probably still need to increase the limit a bit (256M would probably be ok).

I've also made a change to the rebuilding code when rebuilding position counters to use significantly less memory.