POST with PHP/CURL => Dead End

software development

#1

Good Morning

I’m sorry for opening a new post on the same subject but the formatting of my previous message used a PRE tag which ended in annoyingly large windows. That’s why I preferred to open a new one.

Concerning my initial question, i learned quite a lot and my initial question is obsolete anyway. But there is still a problem I can’t solve after spending almost three days on it.

1- I created a script to display the results of an image request on “dreamstime.fotonumerix.com”. You can see the hard coded request (“everest” used for keyword) at http://www.fotonumerix.com/api-dreamstime/dreamstimeapi30.php.

This perfectly working script is the following:

(Please note that in order to display correctly on the forum, I inserted some carriage returns that may look weird)

<?php function My_simplexml_load_file($URL) { $ch = curl_init($URL); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HEADER, 0); $xml = simplexml_load_string(curl_exec($ch)); curl_close($ch); return $xml; } $xml = My_simplexml_load_file( 'http://www.dreamstime.com/api.xml? username=[b]MYUSERNAME[/b]& password=[b]MYENCRYPTEDPASSWORD[/b]& type=get& request=search& srh_field=everest'); $pages = $xml->totalResults/40; if ($xml->totalResults <= 0) { print "

Your images search query gave no results


Try with different keywords.
"; } else { print "

$xml->totalResults search results for your query
$xml->partialResults are shown on this page

page 1 / $pages"; foreach ($xml->items->item as $item){ print ""; } // end of the foreach bracket print "
Dreamstime Image N° imageURL\" alt=\"Image labeled $item->title on Dreamstime\" target=\"_blank\">$item->imageID Label: $item->title Purchased $item->downloads times imageURL\" alt=\"Image labeled $item->title on Dreamstime\" target=\"_blank\">mediumThumb\" title =\"Image labeled $item->title on Dreamstime\">
"; } // end of the else bracket ?>

But then I finally understood the whole idea behind the POST method and created another script where I could modify my keywords which, you’ll admit it, is far more useful. Here it is at http://www.fotonumerix.com/api-dreamstime/dreamstimeapi50.php.

As you can see, nothing ever comes back from any request. The simplified transcription of this script (http://www.fotonumerix.com/api-dreamstime/dreamstimeapi60.php) is the following:


Keywords: <?php function My_simplexml_load_file($url, $srh_field) { $ch = curl_init(); // Begining cURL Action curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, URL_SUFFIX.$srh_field); curl_setopt($ch, CURLOPT_HEADER, 0); $xml = simplexml_load_string(curl_exec($ch)); curl_close($ch); // End cURL Action return $xml; } define('URL_PREFIX', 'http://www.dreamstime.com/api.xml?'); define('URL_SUFFIX', 'username=[b]MYUSERNAME[/b]& password=[b]MYENCRYPTEDPASSWORD[/b]& type=get&request=search&srh_field='); // Fixed part of the request parameters if($_SERVER['REQUEST_METHOD']==='POST') { // Require POST or DIE if(isset($_POST['srh_field'])) $srh_field=$_POST['srh_field']; // If the POST issues Keywords, it goes in the VAR $srh_field $xml = My_simplexml_load_file(URL_PREFIX, $srh_field); // Call uf the cURL ans simplexml() function if ($xml->totalResults <= 0) { print "

No images matched your \"$srh_field\" query.


Try with different keywords.
"; } else { print "

$xml->totalResults search results for \"$srh_field\"
$xml->partialResults are shown on this page

"; foreach ($xml->items->item as $item){ print ""; } // end of the foreach bracket print "
Dreamstime Image N° imageURL\" alt=\"Image labeled $item->title on Dreamstime\" target=\"_blank\">$item->imageID Label: $item->title Purchased $item->downloads times imageURL\" alt=\"Image labeled $item->title on Dreamstime\" target=\"_blank\">mediumThumb\" title =\"Image labeled $item->title on Dreamstime\">
"; } // end of the else bracket } else die('Please enter keywords.'); exit; ?> ----------

I’ve put all sorts of tracers everywhere to make sure that all my variables and constant were ok and they are. The correct URL is well forged with this script. $xml is TRUE after the CURL Close. But nothing is ever shown.

I tried everything I could but I really do not understand where the problem might be.

So if anyone could help me on this one, I would really appreciate.

Thanks many times in advance.

Friendly

MajusC00L


#2

In order to bump this unsolved and almost desperate post, I created new scripts, simpler version to better illustrate what my problem is. Hoping to find some help on this issue, thanks in advance for reading what follows.

I’ve spent the lats two weeks battling with PHP scripts where I try to post keywords to XML engine and get some results back on my site. I went quite far but I encounter problems for which I apparently can’t find any solutions. So yesterday, I thought that I probably lack some basics and I recreated the script in a simpler version to try to identify where I’m doing something wrong. It was a good idea because even with this very basic approach of fetching a page with PHP/CURL, there are things that do not work. Hence this post in a a desperate hope to get back on the right tracks.

1- The idea is the following: A simple form (with a POST method) ask for a keyword which is used by the remaining code to post a PHP/CURL request to an image database RSS search engine. You can see this script in action at the following address: http://www.fotonumerix.com/misc/form01_002.php.

The code for this page is:

[quote]

PHP/CURL Search form Keywords: <?php $result = ""; $url_prefix = "http://www.crestock.com/rss/keyword.aspx?"; $url_suffix = "keyword=".$_REQUEST["keywords"]; $url = $url_prefix.$url_suffix; $ch = curl_init(); [b] curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);[/b] $result=curl_exec($ch); curl_close($ch); echo $result; ?> [/quote] This page works almost correctly. But I don't understand why I'm sometimes presented the form to fill and sometimes the results are displayed directly and there is no way I can have the form displayed ([u]first mystery[/u])

The other page is a more complex version of this first script and it shows exactly where my problem of the last two weeks lies. It’s almost the same script exept that it uses the CURLOPT_POST and CURLOPT_POSTFIELD options. As soon as I use these options, there is no way I can have get results. PHP/CURL usage seems to be so simple to use. I really can’t see what I’m doing wrong. This second script is at http://www.fotonumerix.com/misc/form01_003.php and the code is:

[quote]

PHP/CURL Search form Keywords: <?php $result = ""; $url_prefix = "http://www.crestock.com/rss/keyword.aspx?"; $url_suffix = "keyword=".$_REQUEST["keywords"]; //create a string type like "[i]keyword=anykeywords[/i]" appended to $url_prefix // $url = $url_prefix.$url_suffix; // not used in this version $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url_prefix); [b] curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $url_suffix);[/b] curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $result=curl_exec($ch); curl_close($ch); echo $result; ?> [/quote] "How the heck can I make this second script work?" would be my main question on this post

Thanks in advance for any help provided. I’m totally blinded on this one.

Friendly

MajusC00L


#3

Incorrect markup. You didn’t close the TITLE in your HEAD, and so it wasn’t getting closed until the browser parser sees it being closed in the XML results.

:cool: openvein.org -//- One-time [color=#6600CC]$50.00 discount[/color] on [color=#0000CC]DreamHost[/color] plans: Use ATROPOS7


#4

Atropos7

Thanks so much for this one. Drowning in the php/curl issue, I did not have that close look at the obvious. I just made the correction and the first script displays correctly now.

I can now concentrate on the problem i have on the second script as soon as I start using CURLOPT_POST and CURLOPT_POSTFIELDS.

Would you also have an idea on this one? It just “echo” nothing at all.

Thanks again

Friendly


#5

You’re not using urlencode on the value of the keywords string

You don’t do any error handling. You might want to check curl_error or curl_getinfo

:cool: openvein.org -//- One-time [color=#6600CC]$50.00 discount[/color] on [color=#0000CC]DreamHost[/color] plans: Use ATROPOS7


#6

Heres code to fetch feed subscribers from google reader.
View Code

And heres a script I hacked up to automatically login to google adsense. Of course the above feed subscriber fetcher also does that, both using SSL and POST
View Code

More official examples

But what are you trying to do anyway? why don’t you just request yoursite.com/form01_003.php?keywords=computer and do it like this? Why do you need POST?

$keywords=urlencode($_GET['keywords']); $ch = curl_init("http://www.crestock.com/rss/keyword.aspx?keyword=$keywords"); curl_setopt($ch,CURLOPT_HTTPGET,1); curl_setopt($ch,CURLOPT_RETURNTRANSFER,1); curl_setopt($ch,CURLOPT_HEADER,1); $g=curl_exec($ch); curl_close($ch); header("Content-type: text/plain"); echo $g;

_____
[color=#00CC00] _ _| _ _ _ _|_ _ (_|_\|<(_||_)(_|(_| |(/_ | [/color]
:~


#7

Hi askapache.

Thank you very much for your quick response. I will have a closer look at the links you provide later today.

The correction you bring here to the script make it look like the first one (more like a GET request) which already works. In the final script, I will of course use the urlencode() you suggest here.

The reason why I want to use the POST method is because in the final script, the fields are going to be a lot longer than the sole “keyword” field I’m using in this example script. I don’t know if true but I think that the GET method is not really suited for very long POST.

Why did you include the Header management? I do not yet understand when the header should be processed or not.

Friendly


#8

Hello Atropos7

I did try on my side to see if the urlencode() was making a difference. I traced my variables to witness the change. I understand the use of urlencode() and I will use it in my final script. In this case, with a single word keyword, it does not make a difference. This simple script has just been created to explain my POST problem in this forum.

On the other hand, I did not try the error handling very hard. I’ll put effort in trying to implement it on the script and see what result I get. I’ll get back on this later tonight.

Thanks for your help

Friendly


#9

the problem is that http://www.crestock.com/rss/keyword.aspx does not respond with anything when you send it a post.

So you can try all you like but nothing will ever come back.

What url are you trying to post to? Where is the form you are trying to copy?

If this doesn’t work than I can’t see what would.

[code]$gacookie="/tmp/curl-1.txt";
if(file_exists($gacookie)){ $g=file_get_contents($gacookie); echo $g; }
@touch($gacookie);
@chmod($gacookie,0666);

$FF=array(“Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/;q=0.5”,
“Accept-Language: en-us,en;q=0.5”,“Accept-Encoding: gzip,deflate”,“Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7”,“Connection: Keep-Alive”,“Pragma:”);

if($fp = tmpfile()){
$ch = curl_init(“http://www.crestock.com/rss/keyword.aspx”);
curl_setopt ($ch, CURLOPT_STDERR, $fp);
curl_setopt ($ch, CURLOPT_VERBOSE, 2);
curl_setopt ($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt ($ch, CURLOPT_MAXCONNECTS, 4);
curl_setopt ($ch, CURLOPT_MAXREDIRS, 3);
curl_setopt ($ch, CURLOPT_REFERER, “http://www.crestock.com/rss/keyword.aspx”);
curl_setopt ($ch, CURLOPT_ENCODING, 0);
curl_setopt ($ch, CURLOPT_USERAGENT, ‘Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11’);
curl_setopt ($ch, CURLOPT_HTTPHEADER, $FF);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $gacookie);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $gacookie);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_FAILONERROR, 1);
curl_setopt ($ch, CURLOPT_HEADER, 1);
curl_setopt ($ch, CURLINFO_HEADER_OUT, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 30);
$r=curl_exec($ch);$ch_info=curl_getinfo($ch);
if (curl_errno($ch)) return false;
else curl_close($ch);
header(“Content-type: text/plain”);print_r($ch_info);echo $r;sleep(2);

$ch = curl_init(“http://www.crestock.com/rss/keyword.aspx”);
curl_setopt ($ch, CURLOPT_STDERR, $fp);
curl_setopt ($ch, CURLOPT_VERBOSE, 2);
curl_setopt ($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt ($ch, CURLOPT_MAXCONNECTS, 4);
curl_setopt ($ch, CURLOPT_MAXREDIRS, 3);
curl_setopt ($ch, CURLOPT_REFERER, “http://www.crestock.com/rss/keyword.aspx”);
curl_setopt ($ch, CURLOPT_ENCODING, 0);
curl_setopt ($ch, CURLOPT_USERAGENT, ‘Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7’);
curl_setopt ($ch, CURLOPT_HTTPHEADER, $FF);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $gacookie);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $gacookie);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_FAILONERROR, 1);
curl_setopt ($ch, CURLOPT_HEADER, 1);
curl_setopt ($ch, CURLINFO_HEADER_OUT, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 30);
$r=curl_exec($ch);$ch_info=curl_getinfo($ch);
if (curl_errno($ch)) return false;
else curl_close($ch);
header(“Content-type: text/plain”);print_r($ch_info);echo $r;sleep(2);

$postdata=‘keywords=’.$_GET[‘k’];

$ch1 = curl_init(“http://www.crestock.com/rss/keyword.aspx”);
curl_setopt ($ch1, CURLOPT_STDERR, $fp);
curl_setopt ($ch1, CURLOPT_VERBOSE, 2);
curl_setopt ($ch1, CURLOPT_AUTOREFERER, 1);
curl_setopt ($ch1, CURLOPT_MAXCONNECTS, 4);
curl_setopt ($ch1, CURLOPT_MAXREDIRS, 3);
curl_setopt ($ch1, CURLOPT_ENCODING, 0);
curl_setopt ($ch1, CURLOPT_USERAGENT, ‘Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7’);
//curl_setopt ($ch1, CURLOPT_HTTPHEADER, $FF);
curl_setopt ($ch1, CURLOPT_COOKIEJAR, $gacookie);
curl_setopt ($ch1, CURLOPT_COOKIEFILE, $gacookie);
curl_setopt ($ch1, CURLOPT_POSTFIELDS, $postdata);
curl_setopt ($ch1, CURLOPT_POST, 1);
curl_setopt ($ch1, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch1, CURLOPT_FAILONERROR, 1);
curl_setopt ($ch1, CURLOPT_HEADER, 1);
curl_setopt ($ch1, CURLINFO_HEADER_OUT, 1);
curl_setopt ($ch1, CURLOPT_CONNECTTIMEOUT, 30);
$r=curl_exec ($ch1);$ch1_info=curl_getinfo($ch1);
if (curl_errno($ch1)) return false;
else curl_close($ch1);
header(“Content-type: text/plain”);print_r($ch1_info);echo $r;sleep(2);

}fclose($fp);
@unlink($gacookie);

[/code]_____
[color=#00CC00] _ _| _ _ _ _|_ _ (_|_\|<(_||_)(_|(_| |(/_ | [/color]
:~


#10

AskApache, WOW!

Suddenly, I just feel like sh…t :smiley:

Your comment about crestock’s aspx script just trigger something I never thought about. To me, whenever a form witj a GET method could be used, A POST method could “always” be used as well if preferred. I never realized that a script could actually ignore any of these two methods. In my searches on the web, I never read about such a problem.

Like Don Quichotte, I basically only fought windmills. Crestock is one of the search engine I need to crate. My initial scripts, concerned Dreamstime but I only retained crestock for this topic because I had not implemented any function or parsing. The script was quite naked compared to the one I use for Dreamstime. I thought it would be easier for all to read and react.

So, for Crestock or other agencies, I’ll stick to the GET method whenever it’s possible. But still, there is something that bugs me. How can the method be in cause when the url is correctly formed before the it is fetched by the script? Is it because GET does speak to scripts the same way POST does event if they say the same thing. Scripts can sometimes be POST Ignorant and GET friendly?


#11

This is what fascinates me… I love Communication Protocols, especially HTTP.

Here is what a POST looks like

[code]POST /online-tools/http-headers-tool/ HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.askapache.com/online-tools/http-headers-tool/
Cookie: ASKAPACHEID=27e2819f9b151f973526e73382dbb09f15813d5;
Content-Type: application/x-www-form-urlencoded
Content-Length: 631

h1targetturl=http://www.askapache.com/htaccess/apache-htaccess.html&h1ua=Mozilla/5.0+(Windows;+U;+
Windows+NT+5.1;+en-US;+rv:1.8.1.14)+Gecko/20080404+Firefox/2.0.0.14+(via+www.askapache.com
)&h1ref=http://www.askapache.com/online-tools/http-headers-tool/&h1lan=en-us,en;q=0.
9,de;q=0.8,ja;q=0.8,zh;q=0.7,zh-cn;q=0.6,nl;q=0.5,fr;q=0.5,it;q
=0.4,ko;q=0.3,es;q=0.2,ru;q=0.2,pt;q=0.1&h1char=ISO-8859-1,utf-8;q=0.7%
2C*;q=0.7&h1enc=gzip,deflate&token=b58a06fd141609b78bb27c9146bf20&submitwait=Initiate+HTTP+Header
+Capture[/code]
Now here is the exact same request, but this is a GET request.

GET /online-tools/http-headers-tool/?h1targetturl=http://www.askapache.com/htaccess/apache-htaccess .html&h1ua=Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.14)+Gecko/20080404+ Firefox/2.0.0.14+(via+www.askapache.com)&h1ref=http://www.askapache.com/online-tools/http -headers-tool/&h1lan=en-us,en;q=0.9,de;q=0.8,ja;q=0.8,zh;q=0.7,zh-cn;q=0. 6,nl;q=0.5,fr;q=0.5,it;q=0.4,ko;q=0.3,es;q=0.2,ru;q=0.2,pt;q=0. 1&h1char=ISO-8859-1,utf-8;q=0.7,*;q=0.7&h1enc=gzip,deflate&token=b58a06fd14551609b78bb27c91 46bf0&submitwait=Initiate+HTTP+Header+Capture HTTP/1.1 Host: www.askapache.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: http://www.askapache.com/online-tools/http-headers-tool/ Cookie: ASKAPACHEID=27e2819f9b151f973526e73382dbb09f15813d5;The GET request made it with only 1 TCP packet, the POST required 2. The main difference is this:

Content-Type: application/x-www-form-urlencoded Content-Length: 631A GET request should not have content or content-length, although hackers sometimes try and put data into a system by doing that.

Anything can view a GET because that is what was called, but a POST is only handled by stuff like interpreters and deeper in apache, so it is safer that way.

And you have a limit to how much data you can attach to a GET request before you run into all types of issues… a POST could go up as high as you want almost.

_____
[color=#00CC00] _ _| _ _ _ _|_ _ (_|_\|<(_||_)(_|(_| |(/_ | [/color]
:~


#12

Crytal Clear AskApache. Thanks for your time and explanations. I’ll close this case for now and will stick to the GET solution which works fine. At least, it gives me the opportunity to digg in the parsing issue. I learned quite a lot of new things with this post.

Friendly