Php scripts stopped working, ask for more time?

software development

#1

I have a few php scripts that have long been in production that I run as crons to screen scrape a Bureau of Reclamation site and stuff data into MySQL databases. Well, last week they all stopped working. For example, http://ykfp.org/php/BOR/getFlow6.php .

First, I had trouble remembering how they work, now I’m having trouble remembering how to diagnose a php problem. When I try running a script from a command prompt, I get an error “An attempt was made to dynamically load ChartDirector, but was not successful. Please refer to your ChartDirector for PHP Documentation.” But I think this is misleading, since the library is still present and loads fine to draw my charts: http://ykfp.org/php/BOR/prodbflowtempchartalt.php and the library’s demo charts http://ykfp.org/php/ChartDirector/phpdemo/

The cron job runs these php scripts with lynx, but running one from a browser like Firefox, it runs for a long time, so I think it gets past the dynamic load of the library. But I get a “The server encountered an internal error or misconfiguration and was unable to complete your request.” I verified that the url I’m calling from the BOR is still the same. Do you suppose I just need to tell my host somehow that I need more time to run these screen scrapers?


#2

No, I was barking up the wrong tree, thinking I needed to ask for more execution time. On Monday, I started pulling blocks of code from the top into a new script and inserted echo commands down through the program, then accessed with Firefox until I was able to narrow down the hang up to the first database call:
//echo ‘Line 41’;
if (mysqli_connect_errno())
{
echo ‘Error: Could not connect to database. Please try again later.’;
exit;
}
//Echo ‘line 47’;
// Empty the flow table in preparation for filling it up with revised data
$query = “DELETE from Flow”;
$result = $db->query($query);

Didn’t see the “Could not connect” line, so the Delete query must have been hanging.

Wasn’t able to work on it Tuesday, but when I got back to the office on Wednesday, I found that the original program seemed to be working perfectly again from the cron jobs. What was going on Monday with MySQL?


#3

Hi, I was wondering how you’re screen scraping. I’m looking to see if php_http is supported out of the box but it doesn’t appear that way. I’m a windows guy but I’m trying to figure out if it’s possible to install/configure it. Any help would be appreciated!

Thanks!

/Rob


#4

As noted elsewhere, the curl PHP extension is installed by default, and will probably do what you’re after. I’ve never actually seen the PECL http extension being used in applications — the fact that it’s got the more obvious name is kind of weird and unfortunate. (In fact, some parts of the http extension are even implemented so that they make use of the curl extension.)


#5

Thank you!!


#6

For screen scraping I first tried file_get_contents as described by Welling and Thompson in “PHP and MySQL Web Development” but with the help of the nice people on this forum I learned that now file_get_contents is commonly unavailable to use on external urls because of security problems. I had to then learn curl. http://discussion.dreamhost.com/post-62841.html
pangea33 was very helpful in getting me started. http://discussion.dreamhost.com/thread-62849.html


#7

Thanks for your help MajorGeek and andrewf. I switched over to cURL and was able to make a http request.