|
|
I'm trying to get data from your site, using the C# code in this page: http://technet.rapaport.com/Info/RapLink/SampleCode/Full_Example.aspx I'm sending my username xxxx and password yyyy, but on the line: "WebResponse webResponse = webRequest.GetResponse();", I'm getting an error "The remote server returned an error: (401) Unauthorized.".
The requested URL is:
http://www.diamonds.net/rapnet/DownloadListings/download.aspx?CaratSizeFrom=0.5
&GradeReport=GIA&SellerLogin=123456&SortBy=owner&White=1&Fancy=1
&Programmatically=yes
Why do I get this error? Thanks
|
|
>> add a comment
|
Eliezer
08/30/2009 02:13
|
|
24
Replies:
|
Re: Authentication problem in RapLink feed
|
Hi,
Sorry for the confusion. The page you are trying to access works differently than the examples you find in TechNet. This page does not work in the new TechNet.
For this page you will need to use basic authentication, which would look something like this:
WebClient webclient = new WebClient();
webclient.Credentials = new NetworkCredential(username, password);
webclient.DownloadFile(FilePath); (URL)
One more thing: Don’t download files too often, usually once a day is good, but no more than once an hour. Secondly, you should build in some try – catch if you don’t get the file, or if you get an error. In that case wait for about 5 minutes and try again.
Regards,
Leo
|
|
>> add a comment
|
Leo Muller
8/30/2009 2:15:00 AM
|
|
|
Re: Authentication problem in RapLink feed
|
Hi,
Leo Muller wrote >> "This page does not work in the new TechNet".
Please provide us with information about page we should use in new TechNet.
What does
string URL = "GENERATED_URL_QUERY";
mean in each example?
Thanks a lot!
Best
|
|
>> add a comment
|
Yanek
11/5/2009 6:45:00 AM
|
|
|
Re: Authentication problem in RapLink feed
|
In rapnet.com, you generate the URL codes here:
http://www.rapnet.com/rapnet/DownloadListings/download.aspx
There is similar functionality on diamonds.net. What you have to do is to change this URL to a URL in technet, which will be: http://technet.rapaport.com/HTTP/RapLink/download.aspx. At the moment this is not clear at all from our website, and I will change to shortly.
So, if you get for example a code that is:
http://www.rapnet.com/rapnet/DownloadListings/download.aspx?ShapeIDs=1&WeightFrom=1&WeightTo=2&White=1&Fancy=1&Programmatically=yes
You should change it to:
http://technet.rapaport.com/HTTP/RapLink/download.aspx?ShapeIDs=1&WeightFrom=1&WeightTo=2&White=1&Fancy=1&Programmatically=yes
download.aspx
I you choose to pass the username and password directly to this URL, you will have to change this to HTTPS: https://technet.rapaport.com/HTTP/RapLink/download.aspx?ShapeIDs=1&WeightFrom=1&WeightTo=2&White=1&Fancy=1&Programmatically=yes
download.aspx
Hope this helped.
Regards,
Leo
|
|
>> add a comment
|
Leo Muller
11/5/2009 7:39:00 AM
|
|
|
Re: Authentication problem in RapLink feed
|
Leo
Thank you very much for the answer and for a new content on the site!
I've got the data from the new URL.
One more question about "Rapaport Technical Advisory - November 1, 2009".
Question is about "3. TERMINATION OF OLD RAPAPORT PRICE LIST DATA FORMATS".
Should I worry about price format if I just retrieve diamond listings from RapNet by the URL.
(i.e. URL = http://technet.rapaport.com/HTTP/RapLink/download.aspx?ShapeIDs=1&WeightFrom=1&WeightTo=2&White=1&Fancy=1&Programmatically=yes)
Regards,
Yanek
|
|
>> add a comment
|
Yanek
11/6/2009 2:46:00 AM
|
|
|
Re: Authentication problem in RapLink feed
|
Leo
Hello again
One more question about your previous message.
You wrote: >> "If you choose to pass the username and password directly to this URL, you will have to change this to HTTPS: https://technet.rapaport.com/HTTP/RapLink/download.aspx?ShapeIDs=1&WeightFrom=1&WeightTo=2&White=1&Fancy=1&Programmatically=yes"
Please provide URL example with username and password parameters.
Best regards,
Yanek
|
|
>> add a comment
|
Yanek
11/6/2009 8:39:00 AM
|
|
|
Re: Authentication problem in RapLink feed
|
If you download the data from our TechNet you don't have to worry about the termination of old formats, TechNet only has the new formats, and no changes are planned in these. Also, this has to do with the Rapaport Price Lists, and not with the RapLink, download service.
Take a look at this example:
http://technet.rapaport.com/Info/Prices/SampleCode/Full_Example.aspx
It passes the username and password without getting a user ticket first.
Regards,
Leo
|
|
>> add a comment
|
Leo Muller
11/8/2009 3:55:00 AM
|
|
|
Similar problem
|
I am having a similar problem, I am getting an error "HTTP Error 401.1 - Unauthorized: Access is denied due to invalid credentials. Internet Information Services (IIS)" I am passing the uername and password directly using https and still receiving the error, any idea on where I should start debugging?
|
|
>> add a comment
|
Jarod
11/10/2009 6:01:00 PM
|
|
|
HTTP Error 401
|
Can you post the code that you are trying?
|
|
>> add a comment
|
Leo Muller
11/11/2009 3:01:00 AM
|
|
|
HTTP Error 401
|
I am trying to send the code (coldfusion) but getting an error when trying to post.
|
|
>> add a comment
|
Jarod
11/11/2009 12:47:00 PM
|
|
|
Fixed
|
Fixed the problem, Had firebug running and noticed that the content-encoding was "gzip" and applied proper head tags on my end. Now authenticating and receiving content.
|
|
>> add a comment
|
Jarod
11/11/2009 5:55:00 PM
|
|
|
RE:HTTP Error 401
|
Good, do you want to send the code to me anyway, so I can add it to the code samples? Send it to tech@diamonds.net.
Thanks,
Leo
|
|
>> add a comment
|
Leo Muller
11/12/2009 12:46:00 AM
|
|
|
RE:HTTP Error 401
|
Ya, I will send you the completed code once I am finished with it. Should be later today.
|
|
>> add a comment
|
Jarod
11/12/2009 12:56:00 PM
|
|
|
Sample Code
|
I just emailed the code to you Leo feel free to use it as a sample. We are up and running.
|
|
>> add a comment
|
Jarod
11/13/2009 1:22:00 PM
|
|
|
|
Hi,
I try your PHP code on this page http://technet.rapaport.com/Info/RapLink/SampleCode/PHP_Example.aspx but I have an error message: Parse error: parse error, unexpected T_VARIABLE..... on line 3 // Line code => $request = curl_init($auth_url);
Here the PHP code:
//1 - Authenticate with TechNet. The authentication ticket will be stored in $auth_ticket. Note this MUST be HTTPS.
$auth_url = "https://technet.rapaport.com/HTTP/Authenticate.aspx";
$post_string = "username=YOURUSERNAME&password=" . urlencode("YOURPASSWORD") // I enter my username and my password in this line
//create HTTP POST request with curl:
$request = curl_init($auth_url); // initiate curl object
curl_setopt($request, CURLOPT_HEADER, 0); // set to 0 to eliminate header info from response
curl_setopt($request, CURLOPT_RETURNTRANSFER, 1); // Returns response data instead of TRUE(1)
//curl_setopt($request, CURLOPT_POSTFIELDS, $post_string); // use HTTP POST to send form data
curl_setopt($request, CURLOPT_SSL_VERIFYPEER, FALSE); // uncomment this line if you get no gateway response.
$auth_ticket = curl_exec($request); // execute curl post and store results in $auth_ticket
curl_close ($request);
//2 - prepare HTTP request for data. Copy the URL from the RapLink Feed page in RapNet.com:
// go to: http://www.rapnet.com/RapNet/DownloadListings/Download.aspx, choose your parameters, and click
// generate code. Make sure to specify the columns wanted. This can produce a very long URL.
$feed_url = "http://technet.rapaport.com/HTTP/RapLink/download.aspx?ShapeIDs=1,2,3,4,9&WeightFrom=0&WeightTo=5&LabIDs=1,4,10,5&White=1&Programmatically=yes";
$feed_url .= "&ticket=" & $auth_ticket; //add authentication ticket:
//prepare to save response as file.
$fp = fopen('import/rapnetfeed.csv', 'wb');
if ($fp == FALSE)
{
echo "File not opened";
exit;
}
//create HTTP GET request with curl
$request = curl_init($feed_url); // initiate curl object
curl_setopt($request, CURLOPT_FILE, $fp); //Ask cURL to write the contents to a file
curl_setopt($request, CURLOPT_HEADER, 0); // set to 0 to eliminate header info from response
curl_setopt($request, CURLOPT_TIMEOUT, 300); //set timeout to 5 mins
curl_exec($request); // execute curl post
// additional options may be required depending upon your server configuration
// you can find documentation on curl options at http://www.php.net/curl_setopt
curl_close ($request); // close curl object
fclose($fp); //close file;
Do you have a solution for this error?
|
|
>> add a comment
|
Bruno
1/3/2010 8:21:00 AM
|
|
|
|
Looks like there is a ";" missing at the end of the line before.
Leo
|
|
>> add a comment
|
Leo Muller
1/3/2010 8:53:00 AM
|
|
|
|
Thank you Leo for your answer and your help.
How open, write and save response as Excel file (.csv) with the $feed_url url (in PHP code please) ?
I try this code below but it doesn't work:
$feed_url = "http://technet.rapaport.com/HTTP/RapLink/download.aspx?ShapeIDs=1,2,3,4,9&WeightFrom=0&WeightTo=5&LabIDs=1,4,10,5&White=1&Programmatically=yes";
$feed_url .= "&ticket=" & $auth_ticket; //add authentication ticket:
//prepare to save response as file.
$fp = fopen('import/rapnetfeed.csv', 'wb');
if ($fp == FALSE)
{
echo "File not opened";
exit;
}
//create HTTP GET request with curl
$request = curl_init($feed_url); // initiate curl object
curl_setopt($request, CURLOPT_FILE, $fp); //Ask cURL to write the contents to a file
curl_setopt($request, CURLOPT_HEADER, 0); // set to 0 to eliminate header info from response
curl_setopt($request, CURLOPT_TIMEOUT, 300); //set timeout to 5 mins
curl_exec($request); // execute curl post
// additional options may be required depending upon your server configuration
// you can find documentation on curl options at http://www.php.net/curl_setopt
curl_close ($request); // close curl object
fclose($fp); //close file;
|
|
>> add a comment
|
Bruno
1/3/2010 10:22:00 AM
|
|
|
|
I found the mistake in the PHP code on this page http://technet.rapaport.com/Info/RapLink/SampleCode/PHP_Example.aspx:
Bad code: $feed_url .= "&ticket=" & $auth_ticket; //add authentication ticket:
Good code: $feed_url .= "&ticket=$auth_ticket"; //add authentication ticket: If you want to
Update the good PHP code inti your page http://technet.rapaport.com/Info/RapLink/SampleCode/PHP_Example.aspx for others developers
I have a new question: If I add UseCheckedCulommns= criterias into the technet.rapaport.com url the download doesn't work, for exemple: No problem with this url:
http://technet.rapaport.com/HTTP/RapLink/download.aspx?CaratSizeFrom=0.40
&CaratSizeTo=0.41&GradeReport=IGI,HRD&SortBy=owner&White=1
&Fancy=1&Programmatically=yes
But there is a problem with this url (with UseCheckedCulommns= criterias):
http://technet.rapaport.com/HTTP/RapLink/download.aspx?CaratSizeFrom=0.40
&CaratSizeTo=0.41&GradeReport=IGI,HRD&SortBy=owner&White=1
&Fancy=1&Programmatically=yes&UseCheckedCulommns=1&cCT=1
&cCERT=1&cCLAR=1&cCOLR=1&cCountry=1&cCITY=1&cCulet=1
&cCUT=1&cDPTH=1&cFancyColor=1&cFLR=1&cGIRDLE=1&cLOTNN=1 &cMEAS=1&cFancyColorOvertone=1&cPOL=1&cPX=1&cDPX=1&cRapSpec=1
&cOWNER=1&cSHP=1&cSTATE=1&cSTOCK_NO=1&cSYM=1&cTBL=1
&cSTONES=1&cCertificateImage=1&cCertID=1&cFancyColorIntensity=1
Do you confirm? If so, do you have a solution?
|
|
>> add a comment
|
Bruno
1/5/2010 4:23:00 PM
|
|
|
|
Thanks, I am updating the code sample, this should work as well, right?:
$feed_url .= "&ticket=".$auth_ticket;
|
|
>> add a comment
|
Leo Muller
1/6/2010 1:40:00 AM
|
|
|
|
Yes, there was a problem indeed. The problem was with the inclusion of the Fancy Colors, I fixed this and your code should work now.
Regards,
Leo
|
|
>> add a comment
|
Leo Muller
1/6/2010 2:30:00 AM
|
|
|
|
Thank you Leo, the download work now.
I have a new question:
When I download the data from your url http://technet.rapaport.com/HTTP/RapLink/download.aspx?CaratSizeFrom=0.40
&CaratSizeTo=0.41&GradeReport=IGI,HRD&SortBy=owner&White=1
&Fancy=1&Programmatically=yes&UseCheckedCulommns=1&cCT=1...... I save the data into a .csv file but into this file the Cut grade, Polish, Symetry fields are in a long format: Excellent, Very Good, None, etc... but the same data from the file .csv with this url http://www.rapnet.com/RapNet/DownloadListings/Download.aspx... for the Cut grade, Polish, Symetry fields are in a small format: EX, VG, G, N... this format is better because the database is too small, is it possible to change these criterias when I download the data from http://technet.rapaport.com/HTTP/RapLink/download.aspx ?
|
|
>> add a comment
|
Bruno
1/6/2010 5:44:00 AM
|
|
|
|
*** I changed and added news questions from my previous post ***
When I download the data from your url http://technet.rapaport.com/HTTP/RapLink/download.aspx?CaratSizeFrom=0.40
&CaratSizeTo=0.41&GradeReport=IGI,HRD&SortBy=owner&White=1
&Fancy=1&Programmatically=yes&UseCheckedCulommns=1&cCT=1...... I save the data into a .csv file but into this file the Cut grade, Polish, Symetry, Shape fields are in a long format: Excellent, Very Good, None, Round, Princess, etc... but the same data from the file .csv with this url http://www.rapnet.com/RapNet/DownloadListings/Download.aspx... for the Cut grade, Polish, Symetry fields are in a small format: EX, VG, G, N, BR, PR,... this format is better because the database is too small, is it possible to change these criterias when I download the data from http://technet.rapaport.com/HTTP/RapLink/download.aspx ?
If I used these criterias into the download url technet.rapaport.com: &UseCheckedCulommns=1&cCT=1&cCERT=1&cCLAR=1&cCOLR=1&cCountry=1&cCITY=1&cCulet=1&cCUT=1&cDPTH=1&cFancyColor=1&cFLR=1&cGIRDLE=1&cLOTNN=1&cMEAS=1&cFancyColorOvertone=1&cPOL=1&cPX=1&cDPX=1&cOWNER=1&cSHP=1&cSTOCK_NO=1&cSYM=1&cTBL=1&cCertificateImage=1&cCertID=1&cFancyColorIntensity=1
I found into the file .csv the fields sorted: Carat,Cert,Cert #,Clarity,Color,Country,City,Girdle,Culet,Cut Grade,Depth,Fancy Color,Fluor,Lot #,Measurements,Fancy Overtone,Polish,Price,%/Rap,Owner,Shape,Stock #,Sym,Table,Image,Fancy Intensity
1) there is a problem with the "Girdle" field, this field must between "Fluor" and "Lot" fields and not after the "City" field!
2) the "cCertID=1" is always empty into the .csv file, what is this field?
|
|
>> add a comment
|
Bruno
1/6/2010 8:07:00 AM
|
|
|
|
I also faced this problem with a long format of a lot of fields.
Please update information about them.
|
|
>> add a comment
|
Yanek
1/26/2010 8:43:00 AM
|
|
|
|
The best place to find the translations of the values is here:
http://technet.rapaport.com/Info/LotUpload/File_Upload_Instructions.aspx
it is the reverse of the upload. Regards, Leo
|
|
>> add a comment
|
Leo Muller
1/27/2010 1:17:00 AM
|
|
|
|
how to download data from rapaport weekly price data without user interaction using vb .net coding,
i also using below code mentioned , but not working ANY ONE HELP ME with vb .net2003 Code
Dim b() As Byte client.Credentials = New NetworkCredential(_strUsername, _strPassword, "http://www.rapnet.com/Rapnet/Prices/") b = client.DownloadData("http://www.rapnet.com/Rapnet/Prices/") intFileLength = b.GetLength(0) Return New StringReader(Encoding.ASCII.GetString(b)) Regards, Nayan Shah
|
|
>> add a comment
|
Nayan Shah
9/17/2012 3:01:00 AM
|
|