How-to Handle Large XML Files in PHP
Sooner or later, when you are processing, importing and ingesting data from large suppliers such as Amazon, iTunes Music Store and Virgin Megastore, to name but 3, you are likely to hit a scenario where you are need to handle, process, import and output XML files that are serveral hundred megabytes or even gigabytes in size. In the course of the past year, doing just that, I have come up with some of my own tips and tools for actually processing the suckers, and here I try to document some of these for the benefit of others.
Splitting the XML into Smaller Chunk Files
After preparing the XML you are ready to start processing it. PHP has horrendous memory usage skills and therefore it is hardly the best idea to try and read the file into memory and parse it using the XML tree-based parsers that are built in. To avoid having to switch to using Perl stream-based parsers, I devised an ingenious means by which to split the XML file into variable sized chunks, thus making it much easier to both process the XML with PHP but also to stop and start the process.
The process of chunking the file essentially take advantage of the line breaks in the XML and reads through the XML line by line. It is then programmed to perform certain actions as it reaches certain parts of the XML. For one it increments a counter as it reaches the end of each record, so that when it hits a designated number it can close the current chunk file and begin a new one. The end result of the process is that you will end up with a decent number of bite-sized xml files that are well-formed and readable by any parser.
To start with just initialise all the required variables:
-
<?php
-
-
//initialize vars
-
$minutes=1; // intervals for gates
-
$filenum = 1; // start chunk file number at 1
-
$recordnum = 1; // start at record 1
-
-
//file settings
-
$basefilename = ""; // the base file name for the chunks
-
$xmlfile = ""; // the xml file name to be processed
-
$xmldatadelimiter = ""; // core data delimiter
-
$xmlitemdelimiter = ""; // record delimiter
-
$chunksize = ""; // number of records in each chunk file
-
$xmlstring ="<?xml version="1.0" encoding="UTF-8">\n";
-
$xmlstring.="<$xmldatadelimiter\n>"; // xmlchunk file header
-
-
//dirs and files
-
$dir = "/path/to/directory"; // path to where splits will be stored
-
$exportfile = "$dir"."/splits/$basefilename-$filenum.xml";
Once this is done you are ready to start processing the initial XML file:
This essentially loops through the file line by line. This means that you need to ensure that the XML file has line breaks at the end of each element. If it does not, the entire XML file will be read into memory by PHP!
No that you have the current line held in $buffer, you can start processing the contents of the line and act accordingly in any or all of 4 possible ways:
- increment the record number for the item
- write the line to the chunk file.
- increment the chunk file number, close the xml and start on a new export file
And so each of these actions can be acheived as follows:
-
// if item delimiter reached
-
// increment record number iterator
-
$recordnum++;
-
}
-
//write line to chunk file
-
// if chunk limit reached then start to
-
// close the file with well formed xml
-
if ($recordnum>chunksize) {
-
-
// post feed end tag
-
-
// and increment file number to start new log file chunk
-
//reset record counter number for new chunk file
-
$recordnum=0;
-
$filenum++;
-
-
//update export file name
-
$exportfile = "$dir"."/splits/$basefilename-$filenum.xml";
-
-
//echo status report to STDOUT
-
-
// write new chunk xml file header
-
-
}
After this just run through some internal script time logging and script killing code to (1) keep track of the time it takes and (2) to kill it in event of the script running out of control.
Then just close the file handle for the original XML file, and then as part of the initial check to see if the XML file exists, add an error message to the else clause:
To finish up the process just echo out some time based stats:
Don't Eat With Your Mouthful!
And so in turn, once you have these files, you can use a PHP script to read the contents of the directory containing the split files and then read then into whatever parser you chouse, be it something like simpleXML or a custom php XML handler.
Find out more over at the PHP documentation site
