I have this php script that I’ve have in production since 2007, running from a cron several times a day that just started to cause problems today. I’ve tried various ways of echoing results back to me, and adding more execution time, but it looks like I need more ideas because each time I run the script I get a different number of records.
The script deletes all the records in a mySQL table, then reads in data from a BOR url, chops up the lines with curl, then inserts the records into the mySQL table. Then it repeats the url, curl, insert steps for nine more BOR sites.
The cron job runs the script with lynx like lynx -dump http://sitehome/php/BOR/getFlow6.php, but if I run the lynx line in a ssh shell, I get internal error messages. If I run the script in a browser I get 500 Internal server error messages. But I get a different number of records and sites inserted each time I run it.
When I enable the echo $contents line; in the first site’s block, a browser test shows the file contents dumped until it gives “Content Encoding Error The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression.”
[undefined=undefined][undefined=undefined]require_once("…/ChartDirector/lib/phpchartdir.php");
set_time_limit(1090);
//retrieve Prosser flow data from BOR database
$theurl=“http://www.usbr.gov/pn-bin/yak/arc3.pl”
."?station=YRPW&year=2000&month=10&day=1&year=2015&month=5&day=20&pcode=QD";
$ch = curl_init();
$timeout = 5; // set to zero for no timeout
curl_setopt ($ch, CURLOPT_URL, $theurl);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$contents = curl_exec($ch);
curl_close($ch);
// display file
//echo $contents;
// The feed appears to have useful parsing point identifiers.
// I just used them to get everything between them (including flags)
$cStartStr = “BEGIN DATA”;
//$cEndStr = “END DATA”;
$cEndStr = “Error: file access opening fab”;
$cPageTail = stristr($contents, $cStartStr);
$nUsefulDataEndPos = strpos($cPageTail, $cEndStr);
$cUsefulData = substr($cPageTail, 0, $nUsefulDataEndPos);
// explode the content using newlines as delimeters
$aContents = explode(chr(10), $cUsefulData);
//print_r($aContents);
//echo ‘line 34’;
// Two new arrays initialized to empty
//$myDates = array();
//$myTemps = array();
//echo ‘line 38’;
// Set up and test database connection
@ $db = new mysqli(‘bor****’, ‘username’, ‘pw’, ‘table’);
//echo ‘Line 41’;
if (mysqli_connect_errno())
{
echo ‘Error: Could not connect to database. Please try again later.’;
exit;
}
//Echo ‘line 47’;
// Empty the flow table in preparation for filling it up with revised data
$query = “DELETE from Flow”;
$result = $db->query($query);
// skip the leading and trailing junk
// Prolly don’t want to do all these assignments in the loop. They’re just used for readability
for ($i=3; $i<count($aContents)-1; $i++) {
// Dates are formatted as 10 characters
$cDateStr = substr($aContents[$i],0,10);
#convert date string (a date in human language) to a date in programming language
$myDates = strtotime($cDateStr);
//echo $cDateStr, $myDates;
// QD is everything in the trimmed value after the last space
$nQDVal = substr($aContents[$i], strrpos(trim($aContents[$i]), chr(32))+1);
//$query = "insert into Flow values ('".$myDates."', '".$nQDVal."', '".YRPW."')";
$nQDVal == "MISSING " ?
$query = "insert into Flow values ('".$myDates."', null, '".YRPW."')" :
$query = "insert into Flow values ('".$myDates."', '".$nQDVal."', '".YRPW."')";
$result = $db->query($query);
}
//peep scene
//echo(’
’);’);
//print_r($nQDVal);
//print_r($myDates);
//echo(’
//--------------------------------------------------------------------------
//retrieve Roza flow data from BOR database
$theurl=“http://www.usbr.gov/pn-bin/yak/arc3.pl”
."?station=RBDW&year=2000&month=9&day=25&year=2015&month=4&day=15&pcode=QD";
$ch = curl_init();
$timeout = 5; // set to zero for no timeout
curl_setopt ($ch, CURLOPT_URL, $theurl);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$contents = curl_exec($ch);
curl_close($ch);
// display file
//echo $contents;
// The feed appears to have useful parsing point identifiers.
// I just used them to get everything between them (including flags)
$cStartStr = “BEGIN DATA”;
//$cEndStr = “END DATA”;
$cEndStr = “Error: file access opening fab”;
$cPageTail = stristr($contents, $cStartStr);
$nUsefulDataEndPos = strpos($cPageTail, $cEndStr);
$cUsefulData = substr($cPageTail, 0, $nUsefulDataEndPos);
// explode the content using newlines as delimeters
$aContents = explode(chr(10), $cUsefulData);
// skip the leading and trailing junk0
// Prolly don’t want to do all these assignments in the loop. They’re just used for readability
for ($i=3; $i<count($aContents)-1; $i++) {
// Dates are formatted as 10 characters
$cDateStr = substr($aContents[$i],0,10);
#convert date string (a date in human language) to a date in programming language
$myDates = strtotime($cDateStr);
// QD is everything in the trimmed value after the last space
$nQDVal = substr($aContents[$i], strrpos(trim($aContents[$i]), chr(32))+1);
// if ($nQDVal == “MISSING “)
// {
//$query = “insert into Flow values (’”.$myDates.”’, null, '”.RBDW."’)";
//}
//else
//{
//$query = “insert into Flow values (’”.$myDates."’, ‘".$nQDVal."’, ‘".RBDW."’)";
//}
$nQDVal == “MISSING " ?
$query = “insert into Flow values (’”.$myDates.”’, null, ‘".RBDW."’)" :
$query = “insert into Flow values (’”.$myDates."’, ‘".$nQDVal."’, ‘".RBDW."’)";
$result = $db->query($query);
}
//peep scene
//echo(’
’);’);[/undefined]
//print_r($nQDVal);
//print_r($myDates);
//echo(’
[/undefined]
and simile for 8 more sites.
[hr]
By uncommenting echo $contents; farther into the file, it appeared the script wasn’t getting far enough to get to the third site CHCW. Added more to the time_limit, then more, then more. Now I have set_time_limit(1090000); which just seems ridiculous, and I only seem to get to the forth site.