diff options
Diffstat (limited to 'Master/tlpkg/tlperl/lib/pods/perlunitut.pod')
-rw-r--r-- | Master/tlpkg/tlperl/lib/pods/perlunitut.pod | 208 |
1 files changed, 0 insertions, 208 deletions
diff --git a/Master/tlpkg/tlperl/lib/pods/perlunitut.pod b/Master/tlpkg/tlperl/lib/pods/perlunitut.pod deleted file mode 100644 index 9e5af04ec79..00000000000 --- a/Master/tlpkg/tlperl/lib/pods/perlunitut.pod +++ /dev/null @@ -1,208 +0,0 @@ -=head1 NAME - -perlunitut - Perl Unicode Tutorial - -=head1 DESCRIPTION - -The days of just flinging strings around are over. It's well established that -modern programs need to be capable of communicating funny accented letters, and -things like euro symbols. This means that programmers need new habits. It's -easy to program Unicode capable software, but it does require discipline to do -it right. - -There's a lot to know about character sets, and text encodings. It's probably -best to spend a full day learning all this, but the basics can be learned in -minutes. - -These are not the very basics, though. It is assumed that you already -know the difference between bytes and characters, and realise (and accept!) -that there are many different character sets and encodings, and that your -program has to be explicit about them. Recommended reading is "The Absolute -Minimum Every Software Developer Absolutely, Positively Must Know About Unicode -and Character Sets (No Excuses!)" by Joel Spolsky, at -L<http://joelonsoftware.com/articles/Unicode.html>. - -This tutorial speaks in rather absolute terms, and provides only a limited view -of the wealth of character string related features that Perl has to offer. For -most projects, this information will probably suffice. - -=head2 Definitions - -It's important to set a few things straight first. This is the most important -part of this tutorial. This view may conflict with other information that you -may have found on the web, but that's mostly because many sources are wrong. - -You may have to re-read this entire section a few times... - -=head3 Unicode - -B<Unicode> is a character set with room for lots of characters. The ordinal -value of a character is called a B<code point>. (But in practice, the -distinction between code point and character is blurred, so the terms often -are used interchangeably.) - -There are many, many code points, but computers work with bytes, and a byte has -room for only 256 values. Unicode has many more characters than that, -so you need a method to make these accessible. - -Unicode is encoded using several competing encodings, of which UTF-8 is the -most used. In a Unicode encoding, multiple subsequent bytes can be used to -store a single code point, or simply: character. - -=head3 UTF-8 - -B<UTF-8> is a Unicode encoding. Many people think that Unicode and UTF-8 are -the same thing, but they're not. There are more Unicode encodings, but much of -the world has standardized on UTF-8. - -UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take -only one byte per character. All other characters are encoded as two or more -(up to six) bytes using a complex scheme. Fortunately, Perl handles this for -us, so we don't have to worry about this. - -=head3 Text strings (character strings) - -B<Text strings>, or B<character strings> are made of characters. Bytes are -irrelevant here, and so are encodings. Each character is just that: the -character. - -On a text string, you would do things like: - - $text =~ s/foo/bar/; - if ($string =~ /^\d+$/) { ... } - $text = ucfirst $text; - my $character_count = length $text; - -The value of a character (C<ord>, C<chr>) is the corresponding Unicode code -point. - -=head3 Binary strings (byte strings) - -B<Binary strings>, or B<byte strings> are made of bytes. Here, you don't have -characters, just bytes. All communication with the outside world (anything -outside of your current Perl process) is done in binary. - -On a binary string, you would do things like: - - my (@length_content) = unpack "(V/a)*", $binary; - $binary =~ s/\x00\x0F/\xFF\xF0/; # for the brave :) - print {$fh} $binary; - my $byte_count = length $binary; - -=head3 Encoding - -B<Encoding> (as a verb) is the conversion from I<text> to I<binary>. To encode, -you have to supply the target encoding, for example C<iso-8859-1> or C<UTF-8>. -Some encodings, like the C<iso-8859> ("latin") range, do not support the full -Unicode standard; characters that can't be represented are lost in the -conversion. - -=head3 Decoding - -B<Decoding> is the conversion from I<binary> to I<text>. To decode, you have to -know what encoding was used during the encoding phase. And most of all, it must -be something decodable. It doesn't make much sense to decode a PNG image into a -text string. - -=head3 Internal format - -Perl has an B<internal format>, an encoding that it uses to encode text strings -so it can store them in memory. All text strings are in this internal format. -In fact, text strings are never in any other format! - -You shouldn't worry about what this format is, because conversion is -automatically done when you decode or encode. - -=head2 Your new toolkit - -Add to your standard heading the following line: - - use Encode qw(encode decode); - -Or, if you're lazy, just: - - use Encode; - -=head2 I/O flow (the actual 5 minute tutorial) - -The typical input/output flow of a program is: - - 1. Receive and decode - 2. Process - 3. Encode and output - -If your input is binary, and is supposed to remain binary, you shouldn't decode -it to a text string, of course. But in all other cases, you should decode it. - -Decoding can't happen reliably if you don't know how the data was encoded. If -you get to choose, it's a good idea to standardize on UTF-8. - - my $foo = decode('UTF-8', get 'http://example.com/'); - my $bar = decode('ISO-8859-1', readline STDIN); - my $xyzzy = decode('Windows-1251', $cgi->param('foo')); - -Processing happens as you knew before. The only difference is that you're now -using characters instead of bytes. That's very useful if you use things like -C<substr>, or C<length>. - -It's important to realize that there are no bytes in a text string. Of course, -Perl has its internal encoding to store the string in memory, but ignore that. -If you have to do anything with the number of bytes, it's probably best to move -that part to step 3, just after you've encoded the string. Then you know -exactly how many bytes it will be in the destination string. - -The syntax for encoding text strings to binary strings is as simple as decoding: - - $body = encode('UTF-8', $body); - -If you needed to know the length of the string in bytes, now's the perfect time -for that. Because C<$body> is now a byte string, C<length> will report the -number of bytes, instead of the number of characters. The number of -characters is no longer known, because characters only exist in text strings. - - my $byte_count = length $body; - -And if the protocol you're using supports a way of letting the recipient know -which character encoding you used, please help the receiving end by using that -feature! For example, E-mail and HTTP support MIME headers, so you can use the -C<Content-Type> header. They can also have C<Content-Length> to indicate the -number of I<bytes>, which is always a good idea to supply if the number is -known. - - "Content-Type: text/plain; charset=UTF-8", - "Content-Length: $byte_count" - -=head1 SUMMARY - -Decode everything you receive, encode everything you send out. (If it's text -data.) - -=head1 Q and A (or FAQ) - -After reading this document, you ought to read L<perlunifaq> too. - -=head1 ACKNOWLEDGEMENTS - -Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the -Amsterdam Perl Mongers meetings got me interested and determined to find out -how to use character encodings in Perl in ways that don't break easily. - -Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild" (Dutch -Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial. - -Thanks to the people who asked about this kind of stuff in several Perl IRC -channels, and have constantly reminded me that a simpler explanation was -needed. - -Thanks to the people who reviewed this document for me, before it went public. -They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan -Gray. - -=head1 AUTHOR - -Juerd Waalboer <#####@juerd.nl> - -=head1 SEE ALSO - -L<perlunifaq>, L<perlunicode>, L<perluniintro>, L<Encode> - |