diff options
Diffstat (limited to 'Master/tlpkg/tlperl/lib/lwptut.pod')
| -rw-r--r-- | Master/tlpkg/tlperl/lib/lwptut.pod | 839 |
1 files changed, 839 insertions, 0 deletions
diff --git a/Master/tlpkg/tlperl/lib/lwptut.pod b/Master/tlpkg/tlperl/lib/lwptut.pod new file mode 100644 index 00000000000..34bd58e5e96 --- /dev/null +++ b/Master/tlpkg/tlperl/lib/lwptut.pod @@ -0,0 +1,839 @@ +=head1 NAME + +lwptut -- An LWP Tutorial + +=head1 DESCRIPTION + +LWP (short for "Library for WWW in Perl") is a very popular group of +Perl modules for accessing data on the Web. Like most Perl +module-distributions, each of LWP's component modules comes with +documentation that is a complete reference to its interface. However, +there are so many modules in LWP that it's hard to know where to start +looking for information on how to do even the simplest most common +things. + +Really introducing you to using LWP would require a whole book -- a book +that just happens to exist, called I<Perl & LWP>. But this article +should give you a taste of how you can go about some common tasks with +LWP. + + +=head2 Getting documents with LWP::Simple + +If you just want to get what's at a particular URL, the simplest way +to do it is LWP::Simple's functions. + +In a Perl program, you can call its C<get($url)> function. It will try +getting that URL's content. If it works, then it'll return the +content; but if there's some error, it'll return undef. + + my $url = 'http://freshair.npr.org/dayFA.cfm?todayDate=current'; + # Just an example: the URL for the most recent /Fresh Air/ show + + use LWP::Simple; + my $content = get $url; + die "Couldn't get $url" unless defined $content; + + # Then go do things with $content, like this: + + if($content =~ m/jazz/i) { + print "They're talking about jazz today on Fresh Air!\n"; + } + else { + print "Fresh Air is apparently jazzless today.\n"; + } + +The handiest variant on C<get> is C<getprint>, which is useful in Perl +one-liners. If it can get the page whose URL you provide, it sends it +to STDOUT; otherwise it complains to STDERR. + + % perl -MLWP::Simple -e "getprint 'http://cpan.org/RECENT'" + +That is the URL of a plain text file that lists new files in CPAN in +the past two weeks. You can easily make it part of a tidy little +shell command, like this one that mails you the list of new +C<Acme::> modules: + + % perl -MLWP::Simple -e "getprint 'http://cpan.org/RECENT'" \ + | grep "/by-module/Acme" | mail -s "New Acme modules! Joy!" $USER + +There are other useful functions in LWP::Simple, including one function +for running a HEAD request on a URL (useful for checking links, or +getting the last-revised time of a URL), and two functions for +saving/mirroring a URL to a local file. See L<the LWP::Simple +documentation|LWP::Simple> for the full details, or chapter 2 of I<Perl +& LWP> for more examples. + + + +=for comment + ########################################################################## + + + +=head2 The Basics of the LWP Class Model + +LWP::Simple's functions are handy for simple cases, but its functions +don't support cookies or authorization, don't support setting header +lines in the HTTP request, generally don't support reading header lines +in the HTTP response (notably the full HTTP error message, in case of an +error). To get at all those features, you'll have to use the full LWP +class model. + +While LWP consists of dozens of classes, the main two that you have to +understand are L<LWP::UserAgent> and L<HTTP::Response>. LWP::UserAgent +is a class for "virtual browsers" which you use for performing requests, +and L<HTTP::Response> is a class for the responses (or error messages) +that you get back from those requests. + +The basic idiom is C<< $response = $browser->get($url) >>, or more fully +illustrated: + + # Early in your program: + + use LWP 5.64; # Loads all important LWP classes, and makes + # sure your version is reasonably recent. + + my $browser = LWP::UserAgent->new; + + ... + + # Then later, whenever you need to make a get request: + my $url = 'http://freshair.npr.org/dayFA.cfm?todayDate=current'; + + my $response = $browser->get( $url ); + die "Can't get $url -- ", $response->status_line + unless $response->is_success; + + die "Hey, I was expecting HTML, not ", $response->content_type + unless $response->content_type eq 'text/html'; + # or whatever content-type you're equipped to deal with + + # Otherwise, process the content somehow: + + if($response->decoded_content =~ m/jazz/i) { + print "They're talking about jazz today on Fresh Air!\n"; + } + else { + print "Fresh Air is apparently jazzless today.\n"; + } + +There are two objects involved: C<$browser>, which holds an object of +class LWP::UserAgent, and then the C<$response> object, which is of +class HTTP::Response. You really need only one browser object per +program; but every time you make a request, you get back a new +HTTP::Response object, which will have some interesting attributes: + +=over + +=item * + +A status code indicating +success or failure +(which you can test with C<< $response->is_success >>). + +=item * + +An HTTP status +line that is hopefully informative if there's failure (which you can +see with C<< $response->status_line >>, +returning something like "404 Not Found"). + +=item * + +A MIME content-type like "text/html", "image/gif", +"application/xml", etc., which you can see with +C<< $response->content_type >> + +=item * + +The actual content of the response, in C<< $response->decoded_content >>. +If the response is HTML, that's where the HTML source will be; if +it's a GIF, then C<< $response->decoded_content >> will be the binary +GIF data. + +=item * + +And dozens of other convenient and more specific methods that are +documented in the docs for L<HTTP::Response>, and its superclasses +L<HTTP::Message> and L<HTTP::Headers>. + +=back + + + +=for comment + ########################################################################## + + + +=head2 Adding Other HTTP Request Headers + +The most commonly used syntax for requests is C<< $response = +$browser->get($url) >>, but in truth, you can add extra HTTP header +lines to the request by adding a list of key-value pairs after the URL, +like so: + + $response = $browser->get( $url, $key1, $value1, $key2, $value2, ... ); + +For example, here's how to send some more Netscape-like headers, in case +you're dealing with a site that would otherwise reject your request: + + + my @ns_headers = ( + 'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)', + 'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*', + 'Accept-Charset' => 'iso-8859-1,*,utf-8', + 'Accept-Language' => 'en-US', + ); + + ... + + $response = $browser->get($url, @ns_headers); + +If you weren't reusing that array, you could just go ahead and do this: + + $response = $browser->get($url, + 'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)', + 'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*', + 'Accept-Charset' => 'iso-8859-1,*,utf-8', + 'Accept-Language' => 'en-US', + ); + +If you were only ever changing the 'User-Agent' line, you could just change +the C<$browser> object's default line from "libwww-perl/5.65" (or the like) +to whatever you like, using the LWP::UserAgent C<agent> method: + + $browser->agent('Mozilla/4.76 [en] (Win98; U)'); + + + +=for comment + ########################################################################## + + + +=head2 Enabling Cookies + +A default LWP::UserAgent object acts like a browser with its cookies +support turned off. There are various ways of turning it on, by setting +its C<cookie_jar> attribute. A "cookie jar" is an object representing +a little database of all +the HTTP cookies that a browser can know about. It can correspond to a +file on disk (the way Netscape uses its F<cookies.txt> file), or it can +be just an in-memory object that starts out empty, and whose collection of +cookies will disappear once the program is finished running. + +To give a browser an in-memory empty cookie jar, you set its C<cookie_jar> +attribute like so: + + $browser->cookie_jar({}); + +To give it a copy that will be read from a file on disk, and will be saved +to it when the program is finished running, set the C<cookie_jar> attribute +like this: + + use HTTP::Cookies; + $browser->cookie_jar( HTTP::Cookies->new( + 'file' => '/some/where/cookies.lwp', + # where to read/write cookies + 'autosave' => 1, + # save it to disk when done + )); + +That file will be an LWP-specific format. If you want to be access the +cookies in your Netscape cookies file, you can use the +HTTP::Cookies::Netscape class: + + use HTTP::Cookies; + # yes, loads HTTP::Cookies::Netscape too + + $browser->cookie_jar( HTTP::Cookies::Netscape->new( + 'file' => 'c:/Program Files/Netscape/Users/DIR-NAME-HERE/cookies.txt', + # where to read cookies + )); + +You could add an C<< 'autosave' => 1 >> line as further above, but at +time of writing, it's uncertain whether Netscape might discard some of +the cookies you could be writing back to disk. + + + +=for comment + ########################################################################## + + + +=head2 Posting Form Data + +Many HTML forms send data to their server using an HTTP POST request, which +you can send with this syntax: + + $response = $browser->post( $url, + [ + formkey1 => value1, + formkey2 => value2, + ... + ], + ); + +Or if you need to send HTTP headers: + + $response = $browser->post( $url, + [ + formkey1 => value1, + formkey2 => value2, + ... + ], + headerkey1 => value1, + headerkey2 => value2, + ); + +For example, the following program makes a search request to AltaVista +(by sending some form data via an HTTP POST request), and extracts from +the HTML the report of the number of matches: + + use strict; + use warnings; + use LWP 5.64; + my $browser = LWP::UserAgent->new; + + my $word = 'tarragon'; + + my $url = 'http://www.altavista.com/sites/search/web'; + my $response = $browser->post( $url, + [ 'q' => $word, # the Altavista query string + 'pg' => 'q', 'avkw' => 'tgz', 'kl' => 'XX', + ] + ); + die "$url error: ", $response->status_line + unless $response->is_success; + die "Weird content type at $url -- ", $response->content_type + unless $response->content_is_html; + + if( $response->decoded_content =~ m{AltaVista found ([0-9,]+) results} ) { + # The substring will be like "AltaVista found 2,345 results" + print "$word: $1\n"; + } + else { + print "Couldn't find the match-string in the response\n"; + } + + + +=for comment + ########################################################################## + + + +=head2 Sending GET Form Data + +Some HTML forms convey their form data not by sending the data +in an HTTP POST request, but by making a normal GET request with +the data stuck on the end of the URL. For example, if you went to +C<imdb.com> and ran a search on "Blade Runner", the URL you'd see +in your browser window would be: + + http://us.imdb.com/Tsearch?title=Blade%20Runner&restrict=Movies+and+TV + +To run the same search with LWP, you'd use this idiom, which involves +the URI class: + + use URI; + my $url = URI->new( 'http://us.imdb.com/Tsearch' ); + # makes an object representing the URL + + $url->query_form( # And here the form data pairs: + 'title' => 'Blade Runner', + 'restrict' => 'Movies and TV', + ); + + my $response = $browser->get($url); + +See chapter 5 of I<Perl & LWP> for a longer discussion of HTML forms +and of form data, and chapters 6 through 9 for a longer discussion of +extracting data from HTML. + + + +=head2 Absolutizing URLs + +The URI class that we just mentioned above provides all sorts of methods +for accessing and modifying parts of URLs (such as asking sort of URL it +is with C<< $url->scheme >>, and asking what host it refers to with C<< +$url->host >>, and so on, as described in L<the docs for the URI +class|URI>. However, the methods of most immediate interest +are the C<query_form> method seen above, and now the C<new_abs> method +for taking a probably-relative URL string (like "../foo.html") and getting +back an absolute URL (like "http://www.perl.com/stuff/foo.html"), as +shown here: + + use URI; + $abs = URI->new_abs($maybe_relative, $base); + +For example, consider this program that matches URLs in the HTML +list of new modules in CPAN: + + use strict; + use warnings; + use LWP; + my $browser = LWP::UserAgent->new; + + my $url = 'http://www.cpan.org/RECENT.html'; + my $response = $browser->get($url); + die "Can't get $url -- ", $response->status_line + unless $response->is_success; + + my $html = $response->decoded_content; + while( $html =~ m/<A HREF=\"(.*?)\"/g ) { + print "$1\n"; + } + +When run, it emits output that starts out something like this: + + MIRRORING.FROM + RECENT + RECENT.html + authors/00whois.html + authors/01mailrc.txt.gz + authors/id/A/AA/AASSAD/CHECKSUMS + ... + +However, if you actually want to have those be absolute URLs, you +can use the URI module's C<new_abs> method, by changing the C<while> +loop to this: + + while( $html =~ m/<A HREF=\"(.*?)\"/g ) { + print URI->new_abs( $1, $response->base ) ,"\n"; + } + +(The C<< $response->base >> method from L<HTTP::Message|HTTP::Message> +is for returning what URL +should be used for resolving relative URLs -- it's usually just +the same as the URL that you requested.) + +That program then emits nicely absolute URLs: + + http://www.cpan.org/MIRRORING.FROM + http://www.cpan.org/RECENT + http://www.cpan.org/RECENT.html + http://www.cpan.org/authors/00whois.html + http://www.cpan.org/authors/01mailrc.txt.gz + http://www.cpan.org/authors/id/A/AA/AASSAD/CHECKSUMS + ... + +See chapter 4 of I<Perl & LWP> for a longer discussion of URI objects. + +Of course, using a regexp to match hrefs is a bit simplistic, and for +more robust programs, you'll probably want to use an HTML-parsing module +like L<HTML::LinkExtor> or L<HTML::TokeParser> or even maybe +L<HTML::TreeBuilder>. + + + + +=for comment + ########################################################################## + +=head2 Other Browser Attributes + +LWP::UserAgent objects have many attributes for controlling how they +work. Here are a few notable ones: + +=over + +=item * + +C<< $browser->timeout(15); >> + +This sets this browser object to give up on requests that don't answer +within 15 seconds. + + +=item * + +C<< $browser->protocols_allowed( [ 'http', 'gopher'] ); >> + +This sets this browser object to not speak any protocols other than HTTP +and gopher. If it tries accessing any other kind of URL (like an "ftp:" +or "mailto:" or "news:" URL), then it won't actually try connecting, but +instead will immediately return an error code 500, with a message like +"Access to 'ftp' URIs has been disabled". + + +=item * + +C<< use LWP::ConnCache; $browser->conn_cache(LWP::ConnCache->new()); >> + +This tells the browser object to try using the HTTP/1.1 "Keep-Alive" +feature, which speeds up requests by reusing the same socket connection +for multiple requests to the same server. + + +=item * + +C<< $browser->agent( 'SomeName/1.23 (more info here maybe)' ) >> + +This changes how the browser object will identify itself in +the default "User-Agent" line is its HTTP requests. By default, +it'll send "libwww-perl/I<versionnumber>", like +"libwww-perl/5.65". You can change that to something more descriptive +like this: + + $browser->agent( 'SomeName/3.14 (contact@robotplexus.int)' ); + +Or if need be, you can go in disguise, like this: + + $browser->agent( 'Mozilla/4.0 (compatible; MSIE 5.12; Mac_PowerPC)' ); + + +=item * + +C<< push @{ $ua->requests_redirectable }, 'POST'; >> + +This tells this browser to obey redirection responses to POST requests +(like most modern interactive browsers), even though the HTTP RFC says +that should not normally be done. + + +=back + + +For more options and information, see L<the full documentation for +LWP::UserAgent|LWP::UserAgent>. + + + +=for comment + ########################################################################## + + + +=head2 Writing Polite Robots + +If you want to make sure that your LWP-based program respects F<robots.txt> +files and doesn't make too many requests too fast, you can use the LWP::RobotUA +class instead of the LWP::UserAgent class. + +LWP::RobotUA class is just like LWP::UserAgent, and you can use it like so: + + use LWP::RobotUA; + my $browser = LWP::RobotUA->new('YourSuperBot/1.34', 'you@yoursite.com'); + # Your bot's name and your email address + + my $response = $browser->get($url); + +But HTTP::RobotUA adds these features: + + +=over + +=item * + +If the F<robots.txt> on C<$url>'s server forbids you from accessing +C<$url>, then the C<$browser> object (assuming it's of class LWP::RobotUA) +won't actually request it, but instead will give you back (in C<$response>) a 403 error +with a message "Forbidden by robots.txt". That is, if you have this line: + + die "$url -- ", $response->status_line, "\nAborted" + unless $response->is_success; + +then the program would die with an error message like this: + + http://whatever.site.int/pith/x.html -- 403 Forbidden by robots.txt + Aborted at whateverprogram.pl line 1234 + +=item * + +If this C<$browser> object sees that the last time it talked to +C<$url>'s server was too recently, then it will pause (via C<sleep>) to +avoid making too many requests too often. How long it will pause for, is +by default one minute -- but you can control it with the C<< +$browser->delay( I<minutes> ) >> attribute. + +For example, this code: + + $browser->delay( 7/60 ); + +...means that this browser will pause when it needs to avoid talking to +any given server more than once every 7 seconds. + +=back + +For more options and information, see L<the full documentation for +LWP::RobotUA|LWP::RobotUA>. + + + + + +=for comment + ########################################################################## + +=head2 Using Proxies + +In some cases, you will want to (or will have to) use proxies for +accessing certain sites and/or using certain protocols. This is most +commonly the case when your LWP program is running (or could be running) +on a machine that is behind a firewall. + +To make a browser object use proxies that are defined in the usual +environment variables (C<HTTP_PROXY>, etc.), just call the C<env_proxy> +on a user-agent object before you go making any requests on it. +Specifically: + + use LWP::UserAgent; + my $browser = LWP::UserAgent->new; + + # And before you go making any requests: + $browser->env_proxy; + +For more information on proxy parameters, see L<the LWP::UserAgent +documentation|LWP::UserAgent>, specifically the C<proxy>, C<env_proxy>, +and C<no_proxy> methods. + + + +=for comment + ########################################################################## + +=head2 HTTP Authentication + +Many web sites restrict access to documents by using "HTTP +Authentication". This isn't just any form of "enter your password" +restriction, but is a specific mechanism where the HTTP server sends the +browser an HTTP code that says "That document is part of a protected +'realm', and you can access it only if you re-request it and add some +special authorization headers to your request". + +For example, the Unicode.org admins stop email-harvesting bots from +harvesting the contents of their mailing list archives, by protecting +them with HTTP Authentication, and then publicly stating the username +and password (at C<http://www.unicode.org/mail-arch/>) -- namely +username "unicode-ml" and password "unicode". + +For example, consider this URL, which is part of the protected +area of the web site: + + http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html + +If you access that with a browser, you'll get a prompt +like +"Enter username and password for 'Unicode-MailList-Archives' at server +'www.unicode.org'". + +In LWP, if you just request that URL, like this: + + use LWP; + my $browser = LWP::UserAgent->new; + + my $url = + 'http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html'; + my $response = $browser->get($url); + + die "Error: ", $response->header('WWW-Authenticate') || 'Error accessing', + # ('WWW-Authenticate' is the realm-name) + "\n ", $response->status_line, "\n at $url\n Aborting" + unless $response->is_success; + +Then you'll get this error: + + Error: Basic realm="Unicode-MailList-Archives" + 401 Authorization Required + at http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html + Aborting at auth1.pl line 9. [or wherever] + +...because the C<$browser> doesn't know any the username and password +for that realm ("Unicode-MailList-Archives") at that host +("www.unicode.org"). The simplest way to let the browser know about this +is to use the C<credentials> method to let it know about a username and +password that it can try using for that realm at that host. The syntax is: + + $browser->credentials( + 'servername:portnumber', + 'realm-name', + 'username' => 'password' + ); + +In most cases, the port number is 80, the default TCP/IP port for HTTP; and +you usually call the C<credentials> method before you make any requests. +For example: + + $browser->credentials( + 'reports.mybazouki.com:80', + 'web_server_usage_reports', + 'plinky' => 'banjo123' + ); + +So if we add the following to the program above, right after the C<< +$browser = LWP::UserAgent->new; >> line... + + $browser->credentials( # add this to our $browser 's "key ring" + 'www.unicode.org:80', + 'Unicode-MailList-Archives', + 'unicode-ml' => 'unicode' + ); + +...then when we run it, the request succeeds, instead of causing the +C<die> to be called. + + + +=for comment + ########################################################################## + +=head2 Accessing HTTPS URLs + +When you access an HTTPS URL, it'll work for you just like an HTTP URL +would -- if your LWP installation has HTTPS support (via an appropriate +Secure Sockets Layer library). For example: + + use LWP; + my $url = 'https://www.paypal.com/'; # Yes, HTTPS! + my $browser = LWP::UserAgent->new; + my $response = $browser->get($url); + die "Error at $url\n ", $response->status_line, "\n Aborting" + unless $response->is_success; + print "Whee, it worked! I got that ", + $response->content_type, " document!\n"; + +If your LWP installation doesn't have HTTPS support set up, then the +response will be unsuccessful, and you'll get this error message: + + Error at https://www.paypal.com/ + 501 Protocol scheme 'https' is not supported + Aborting at paypal.pl line 7. [or whatever program and line] + +If your LWP installation I<does> have HTTPS support installed, then the +response should be successful, and you should be able to consult +C<$response> just like with any normal HTTP response. + +For information about installing HTTPS support for your LWP +installation, see the helpful F<README.SSL> file that comes in the +libwww-perl distribution. + + +=for comment + ########################################################################## + + + +=head2 Getting Large Documents + +When you're requesting a large (or at least potentially large) document, +a problem with the normal way of using the request methods (like C<< +$response = $browser->get($url) >>) is that the response object in +memory will have to hold the whole document -- I<in memory>. If the +response is a thirty megabyte file, this is likely to be quite an +imposition on this process's memory usage. + +A notable alternative is to have LWP save the content to a file on disk, +instead of saving it up in memory. This is the syntax to use: + + $response = $ua->get($url, + ':content_file' => $filespec, + ); + +For example, + + $response = $ua->get('http://search.cpan.org/', + ':content_file' => '/tmp/sco.html' + ); + +When you use this C<:content_file> option, the C<$response> will have +all the normal header lines, but C<< $response->content >> will be +empty. + +Note that this ":content_file" option isn't supported under older +versions of LWP, so you should consider adding C<use LWP 5.66;> to check +the LWP version, if you think your program might run on systems with +older versions. + +If you need to be compatible with older LWP versions, then use +this syntax, which does the same thing: + + use HTTP::Request::Common; + $response = $ua->request( GET($url), $filespec ); + + +=for comment + ########################################################################## + + +=head1 SEE ALSO + +Remember, this article is just the most rudimentary introduction to +LWP -- to learn more about LWP and LWP-related tasks, you really +must read from the following: + +=over + +=item * + +L<LWP::Simple> -- simple functions for getting/heading/mirroring URLs + +=item * + +L<LWP> -- overview of the libwww-perl modules + +=item * + +L<LWP::UserAgent> -- the class for objects that represent "virtual browsers" + +=item * + +L<HTTP::Response> -- the class for objects that represent the response to +a LWP response, as in C<< $response = $browser->get(...) >> + +=item * + +L<HTTP::Message> and L<HTTP::Headers> -- classes that provide more methods +to HTTP::Response. + +=item * + +L<URI> -- class for objects that represent absolute or relative URLs + +=item * + +L<URI::Escape> -- functions for URL-escaping and URL-unescaping strings +(like turning "this & that" to and from "this%20%26%20that"). + +=item * + +L<HTML::Entities> -- functions for HTML-escaping and HTML-unescaping strings +(like turning "C. & E. BrontE<euml>" to and from "C. & E. Brontë") + +=item * + +L<HTML::TokeParser> and L<HTML::TreeBuilder> -- classes for parsing HTML + +=item * + +L<HTML::LinkExtor> -- class for finding links in HTML documents + +=item * + +The book I<Perl & LWP> by Sean M. Burke. O'Reilly & Associates, +2002. ISBN: 0-596-00178-9, L<http://www.oreilly.com/catalog/perllwp/>. The +whole book is also available free online: +L<http://lwp.interglacial.com>. + +=back + + +=head1 COPYRIGHT + +Copyright 2002, Sean M. Burke. You can redistribute this document and/or +modify it, but only under the same terms as Perl itself. + +=head1 AUTHOR + +Sean M. Burke C<sburke@cpan.org> + +=for comment + ########################################################################## + +=cut + +# End of Pod |