summaryrefslogtreecommitdiff
path: root/Master/tlpkg/tlperl/lib/pods/perlunicode.pod
diff options
context:
space:
mode:
Diffstat (limited to 'Master/tlpkg/tlperl/lib/pods/perlunicode.pod')
-rw-r--r--Master/tlpkg/tlperl/lib/pods/perlunicode.pod1826
1 files changed, 0 insertions, 1826 deletions
diff --git a/Master/tlpkg/tlperl/lib/pods/perlunicode.pod b/Master/tlpkg/tlperl/lib/pods/perlunicode.pod
deleted file mode 100644
index 77daca34a7d..00000000000
--- a/Master/tlpkg/tlperl/lib/pods/perlunicode.pod
+++ /dev/null
@@ -1,1826 +0,0 @@
-=head1 NAME
-
-perlunicode - Unicode support in Perl
-
-=head1 DESCRIPTION
-
-=head2 Important Caveats
-
-Unicode support is an extensive requirement. While Perl does not
-implement the Unicode standard or the accompanying technical reports
-from cover to cover, Perl does support many Unicode features.
-
-People who want to learn to use Unicode in Perl, should probably read
-the L<Perl Unicode tutorial, perlunitut|perlunitut> and
-L<perluniintro>, before reading
-this reference document.
-
-Also, the use of Unicode may present security issues that aren't obvious.
-Read L<Unicode Security Considerations|http://www.unicode.org/reports/tr36>.
-
-=over 4
-
-=item Safest if you "use feature 'unicode_strings'"
-
-In order to preserve backward compatibility, Perl does not turn
-on full internal Unicode support unless the pragma
-C<use feature 'unicode_strings'> is specified. (This is automatically
-selected if you use C<use 5.012> or higher.) Failure to do this can
-trigger unexpected surprises. See L</The "Unicode Bug"> below.
-
-This pragma doesn't affect I/O, and there are still several places
-where Unicode isn't fully supported, such as in filenames.
-
-=item Input and Output Layers
-
-Perl knows when a filehandle uses Perl's internal Unicode encodings
-(UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with
-the ":encoding(utf8)" layer. Other encodings can be converted to Perl's
-encoding on input or from Perl's encoding on output by use of the
-":encoding(...)" layer. See L<open>.
-
-To indicate that Perl source itself is in UTF-8, use C<use utf8;>.
-
-=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
-
-As a compatibility measure, the C<use utf8> pragma must be explicitly
-included to enable recognition of UTF-8 in the Perl scripts themselves
-(in string or regular expression literals, or in identifier names) on
-ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based
-machines. B<These are the only times when an explicit C<use utf8>
-is needed.> See L<utf8>.
-
-=item BOM-marked scripts and UTF-16 scripts autodetected
-
-If a Perl script begins marked with the Unicode BOM (UTF-16LE, UTF16-BE,
-or UTF-8), or if the script looks like non-BOM-marked UTF-16 of either
-endianness, Perl will correctly read in the script as Unicode.
-(BOMless UTF-8 cannot be effectively recognized or differentiated from
-ISO 8859-1 or other eight-bit encodings.)
-
-=item C<use encoding> needed to upgrade non-Latin-1 byte strings
-
-By default, there is a fundamental asymmetry in Perl's Unicode model:
-implicit upgrading from byte strings to Unicode strings assumes that
-they were encoded in I<ISO 8859-1 (Latin-1)>, but Unicode strings are
-downgraded with UTF-8 encoding. This happens because the first 256
-codepoints in Unicode happens to agree with Latin-1.
-
-See L</"Byte and Character Semantics"> for more details.
-
-=back
-
-=head2 Byte and Character Semantics
-
-Beginning with version 5.6, Perl uses logically-wide characters to
-represent strings internally.
-
-Starting in Perl 5.14, Perl-level operations work with
-characters rather than bytes within the scope of a
-C<L<use feature 'unicode_strings'|feature>> (or equivalently
-C<use 5.012> or higher). (This is not true if bytes have been
-explicitly requested by C<L<use bytes|bytes>>, nor necessarily true
-for interactions with the platform's operating system.)
-
-For earlier Perls, and when C<unicode_strings> is not in effect, Perl
-provides a fairly safe environment that can handle both types of
-semantics in programs. For operations where Perl can unambiguously
-decide that the input data are characters, Perl switches to character
-semantics. For operations where this determination cannot be made
-without additional information from the user, Perl decides in favor of
-compatibility and chooses to use byte semantics.
-
-When C<use locale> (but not C<use locale ':not_characters'>) is in
-effect, Perl uses the semantics associated with the current locale.
-(C<use locale> overrides C<use feature 'unicode_strings'> in the same scope;
-while C<use locale ':not_characters'> effectively also selects
-C<use feature 'unicode_strings'> in its scope; see L<perllocale>.)
-Otherwise, Perl uses the platform's native
-byte semantics for characters whose code points are less than 256, and
-Unicode semantics for those greater than 255. On EBCDIC platforms, this
-is almost seamless, as the EBCDIC code pages that Perl handles are
-equivalent to Unicode's first 256 code points. (The exception is that
-EBCDIC regular expression case-insensitive matching rules are not as
-as robust as Unicode's.) But on ASCII platforms, Perl uses US-ASCII
-(or Basic Latin in Unicode terminology) byte semantics, meaning that characters
-whose ordinal numbers are in the range 128 - 255 are undefined except for their
-ordinal numbers. This means that none have case (upper and lower), nor are any
-a member of character classes, like C<[:alpha:]> or C<\w>. (But all do belong
-to the C<\W> class or the Perl regular expression extension C<[:^alpha:]>.)
-
-This behavior preserves compatibility with earlier versions of Perl,
-which allowed byte semantics in Perl operations only if
-none of the program's inputs were marked as being a source of Unicode
-character data. Such data may come from filehandles, from calls to
-external programs, from information provided by the system (such as %ENV),
-or from literals and constants in the source text.
-
-The C<utf8> pragma is primarily a compatibility device that enables
-recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
-Note that this pragma is only required while Perl defaults to byte
-semantics; when character semantics become the default, this pragma
-may become a no-op. See L<utf8>.
-
-If strings operating under byte semantics and strings with Unicode
-character data are concatenated, the new string will have
-character semantics. This can cause surprises: See L</BUGS>, below.
-You can choose to be warned when this happens. See L<encoding::warnings>.
-
-Under character semantics, many operations that formerly operated on
-bytes now operate on characters. A character in Perl is
-logically just a number ranging from 0 to 2**31 or so. Larger
-characters may encode into longer sequences of bytes internally, but
-this internal detail is mostly hidden for Perl code.
-See L<perluniintro> for more.
-
-=head2 Effects of Character Semantics
-
-Character semantics have the following effects:
-
-=over 4
-
-=item *
-
-Strings--including hash keys--and regular expression patterns may
-contain characters that have an ordinal value larger than 255.
-
-If you use a Unicode editor to edit your program, Unicode characters may
-occur directly within the literal strings in UTF-8 encoding, or UTF-16.
-(The former requires a BOM or C<use utf8>, the latter requires a BOM.)
-
-Unicode characters can also be added to a string by using the C<\N{U+...}>
-notation. The Unicode code for the desired character, in hexadecimal,
-should be placed in the braces, after the C<U>. For instance, a smiley face is
-C<\N{U+263A}>.
-
-Alternatively, you can use the C<\x{...}> notation for characters 0x100 and
-above. For characters below 0x100 you may get byte semantics instead of
-character semantics; see L</The "Unicode Bug">. On EBCDIC machines there is
-the additional problem that the value for such characters gives the EBCDIC
-character rather than the Unicode one, thus it is more portable to use
-C<\N{U+...}> instead.
-
-Additionally, you can use the C<\N{...}> notation and put the official
-Unicode character name within the braces, such as
-C<\N{WHITE SMILING FACE}>. This automatically loads the L<charnames>
-module with the C<:full> and C<:short> options. If you prefer different
-options for this module, you can instead, before the C<\N{...}>,
-explicitly load it with your desired options; for example,
-
- use charnames ':loose';
-
-=item *
-
-If an appropriate L<encoding> is specified, identifiers within the
-Perl script may contain Unicode alphanumeric characters, including
-ideographs. Perl does not currently attempt to canonicalize variable
-names.
-
-=item *
-
-Regular expressions match characters instead of bytes. "." matches
-a character instead of a byte.
-
-=item *
-
-Bracketed character classes in regular expressions match characters instead of
-bytes and match against the character properties specified in the
-Unicode properties database. C<\w> can be used to match a Japanese
-ideograph, for instance.
-
-=item *
-
-Named Unicode properties, scripts, and block ranges may be used (like bracketed
-character classes) by using the C<\p{}> "matches property" construct and
-the C<\P{}> negation, "doesn't match property".
-See L</"Unicode Character Properties"> for more details.
-
-You can define your own character properties and use them
-in the regular expression with the C<\p{}> or C<\P{}> construct.
-See L</"User-Defined Character Properties"> for more details.
-
-=item *
-
-The special pattern C<\X> matches a logical character, an "extended grapheme
-cluster" in Standardese. In Unicode what appears to the user to be a single
-character, for example an accented C<G>, may in fact be composed of a sequence
-of characters, in this case a C<G> followed by an accent character. C<\X>
-will match the entire sequence.
-
-=item *
-
-The C<tr///> operator translates characters instead of bytes. Note
-that the C<tr///CU> functionality has been removed. For similar
-functionality see pack('U0', ...) and pack('C0', ...).
-
-=item *
-
-Case translation operators use the Unicode case translation tables
-when character input is provided. Note that C<uc()>, or C<\U> in
-interpolated strings, translates to uppercase, while C<ucfirst>,
-or C<\u> in interpolated strings, translates to titlecase in languages
-that make the distinction (which is equivalent to uppercase in languages
-without the distinction).
-
-=item *
-
-Most operators that deal with positions or lengths in a string will
-automatically switch to using character positions, including
-C<chop()>, C<chomp()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
-C<sprintf()>, C<write()>, and C<length()>. An operator that
-specifically does not switch is C<vec()>. Operators that really don't
-care include operators that treat strings as a bucket of bits such as
-C<sort()>, and operators dealing with filenames.
-
-=item *
-
-The C<pack()>/C<unpack()> letter C<C> does I<not> change, since it is often
-used for byte-oriented formats. Again, think C<char> in the C language.
-
-There is a new C<U> specifier that converts between Unicode characters
-and code points. There is also a C<W> specifier that is the equivalent of
-C<chr>/C<ord> and properly handles character values even if they are above 255.
-
-=item *
-
-The C<chr()> and C<ord()> functions work on characters, similar to
-C<pack("W")> and C<unpack("W")>, I<not> C<pack("C")> and
-C<unpack("C")>. C<pack("C")> and C<unpack("C")> are methods for
-emulating byte-oriented C<chr()> and C<ord()> on Unicode strings.
-While these methods reveal the internal encoding of Unicode strings,
-that is not something one normally needs to care about at all.
-
-=item *
-
-The bit string operators, C<& | ^ ~>, can operate on character data.
-However, for backward compatibility, such as when using bit string
-operations when characters are all less than 256 in ordinal value, one
-should not use C<~> (the bit complement) with characters of both
-values less than 256 and values greater than 256. Most importantly,
-DeMorgan's laws (C<~($x|$y) eq ~$x&~$y> and C<~($x&$y) eq ~$x|~$y>)
-will not hold. The reason for this mathematical I<faux pas> is that
-the complement cannot return B<both> the 8-bit (byte-wide) bit
-complement B<and> the full character-wide bit complement.
-
-=item *
-
-There is a CPAN module, L<Unicode::Casing>, which allows you to define
-your own mappings to be used in C<lc()>, C<lcfirst()>, C<uc()>,
-C<ucfirst()>, and C<fc> (or their double-quoted string inlined
-versions such as C<\U>).
-(Prior to Perl 5.16, this functionality was partially provided
-in the Perl core, but suffered from a number of insurmountable
-drawbacks, so the CPAN module was written instead.)
-
-=back
-
-=over 4
-
-=item *
-
-And finally, C<scalar reverse()> reverses by character rather than by byte.
-
-=back
-
-=head2 Unicode Character Properties
-
-(The only time that Perl considers a sequence of individual code
-points as a single logical character is in the C<\X> construct, already
-mentioned above. Therefore "character" in this discussion means a single
-Unicode code point.)
-
-Very nearly all Unicode character properties are accessible through
-regular expressions by using the C<\p{}> "matches property" construct
-and the C<\P{}> "doesn't match property" for its negation.
-
-For instance, C<\p{Uppercase}> matches any single character with the Unicode
-"Uppercase" property, while C<\p{L}> matches any character with a
-General_Category of "L" (letter) property. Brackets are not
-required for single letter property names, so C<\p{L}> is equivalent to C<\pL>.
-
-More formally, C<\p{Uppercase}> matches any single character whose Unicode
-Uppercase property value is True, and C<\P{Uppercase}> matches any character
-whose Uppercase property value is False, and they could have been written as
-C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively.
-
-This formality is needed when properties are not binary; that is, if they can
-take on more values than just True and False. For example, the Bidi_Class (see
-L</"Bidirectional Character Types"> below), can take on several different
-values, such as Left, Right, Whitespace, and others. To match these, one needs
-to specify both the property name (Bidi_Class), AND the value being
-matched against
-(Left, Right, etc.). This is done, as in the examples above, by having the
-two components separated by an equal sign (or interchangeably, a colon), like
-C<\p{Bidi_Class: Left}>.
-
-All Unicode-defined character properties may be written in these compound forms
-of C<\p{property=value}> or C<\p{property:value}>, but Perl provides some
-additional properties that are written only in the single form, as well as
-single-form short-cuts for all binary properties and certain others described
-below, in which you may omit the property name and the equals or colon
-separator.
-
-Most Unicode character properties have at least two synonyms (or aliases if you
-prefer): a short one that is easier to type and a longer one that is more
-descriptive and hence easier to understand. Thus the "L" and "Letter" properties
-above are equivalent and can be used interchangeably. Likewise,
-"Upper" is a synonym for "Uppercase", and we could have written
-C<\p{Uppercase}> equivalently as C<\p{Upper}>. Also, there are typically
-various synonyms for the values the property can be. For binary properties,
-"True" has 3 synonyms: "T", "Yes", and "Y"; and "False has correspondingly "F",
-"No", and "N". But be careful. A short form of a value for one property may
-not mean the same thing as the same short form for another. Thus, for the
-General_Category property, "L" means "Letter", but for the Bidi_Class property,
-"L" means "Left". A complete list of properties and synonyms is in
-L<perluniprops>.
-
-Upper/lower case differences in property names and values are irrelevant;
-thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>.
-Similarly, you can add or subtract underscores anywhere in the middle of a
-word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space
-is irrelevant adjacent to non-word characters, such as the braces and the equals
-or colon separators, so C<\p{ Upper }> and C<\p{ Upper_case : Y }> are
-equivalent to these as well. In fact, white space and even
-hyphens can usually be added or deleted anywhere. So even C<\p{ Up-per case = Yes}> is
-equivalent. All this is called "loose-matching" by Unicode. The few places
-where stricter matching is used is in the middle of numbers, and in the Perl
-extension properties that begin or end with an underscore. Stricter matching
-cares about white space (except adjacent to non-word characters),
-hyphens, and non-interior underscores.
-
-You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret
-(^) between the first brace and the property name: C<\p{^Tamil}> is
-equal to C<\P{Tamil}>.
-
-Almost all properties are immune to case-insensitive matching. That is,
-adding a C</i> regular expression modifier does not change what they
-match. There are two sets that are affected.
-The first set is
-C<Uppercase_Letter>,
-C<Lowercase_Letter>,
-and C<Titlecase_Letter>,
-all of which match C<Cased_Letter> under C</i> matching.
-And the second set is
-C<Uppercase>,
-C<Lowercase>,
-and C<Titlecase>,
-all of which match C<Cased> under C</i> matching.
-This set also includes its subsets C<PosixUpper> and C<PosixLower> both
-of which under C</i> matching match C<PosixAlpha>.
-(The difference between these sets is that some things, such as Roman
-numerals, come in both upper and lower case so they are C<Cased>, but aren't considered
-letters, so they aren't C<Cased_Letter>s.)
-
-The result is undefined if you try to match a non-Unicode code point
-(that is, one above 0x10FFFF) against a Unicode property. Currently, a
-warning is raised, and the match will fail. In some cases, this is
-counterintuitive, as both these fail:
-
- chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails.
- chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Fails!
-
-=head3 B<General_Category>
-
-Every Unicode character is assigned a general category, which is the "most
-usual categorization of a character" (from
-L<http://www.unicode.org/reports/tr44>).
-
-The compound way of writing these is like C<\p{General_Category=Number}>
-(short, C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up
-through the equal or colon separator is omitted. So you can instead just write
-C<\pN>.
-
-Here are the short and long forms of the General Category properties:
-
- Short Long
-
- L Letter
- LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}])
- Lu Uppercase_Letter
- Ll Lowercase_Letter
- Lt Titlecase_Letter
- Lm Modifier_Letter
- Lo Other_Letter
-
- M Mark
- Mn Nonspacing_Mark
- Mc Spacing_Mark
- Me Enclosing_Mark
-
- N Number
- Nd Decimal_Number (also Digit)
- Nl Letter_Number
- No Other_Number
-
- P Punctuation (also Punct)
- Pc Connector_Punctuation
- Pd Dash_Punctuation
- Ps Open_Punctuation
- Pe Close_Punctuation
- Pi Initial_Punctuation
- (may behave like Ps or Pe depending on usage)
- Pf Final_Punctuation
- (may behave like Ps or Pe depending on usage)
- Po Other_Punctuation
-
- S Symbol
- Sm Math_Symbol
- Sc Currency_Symbol
- Sk Modifier_Symbol
- So Other_Symbol
-
- Z Separator
- Zs Space_Separator
- Zl Line_Separator
- Zp Paragraph_Separator
-
- C Other
- Cc Control (also Cntrl)
- Cf Format
- Cs Surrogate
- Co Private_Use
- Cn Unassigned
-
-Single-letter properties match all characters in any of the
-two-letter sub-properties starting with the same letter.
-C<LC> and C<L&> are special: both are aliases for the set consisting of everything matched by C<Ll>, C<Lu>, and C<Lt>.
-
-=head3 B<Bidirectional Character Types>
-
-Because scripts differ in their directionality (Hebrew and Arabic are
-written right to left, for example) Unicode supplies these properties in
-the Bidi_Class class:
-
- Property Meaning
-
- L Left-to-Right
- LRE Left-to-Right Embedding
- LRO Left-to-Right Override
- R Right-to-Left
- AL Arabic Letter
- RLE Right-to-Left Embedding
- RLO Right-to-Left Override
- PDF Pop Directional Format
- EN European Number
- ES European Separator
- ET European Terminator
- AN Arabic Number
- CS Common Separator
- NSM Non-Spacing Mark
- BN Boundary Neutral
- B Paragraph Separator
- S Segment Separator
- WS Whitespace
- ON Other Neutrals
-
-This property is always written in the compound form.
-For example, C<\p{Bidi_Class:R}> matches characters that are normally
-written right to left.
-
-=head3 B<Scripts>
-
-The world's languages are written in many different scripts. This sentence
-(unless you're reading it in translation) is written in Latin, while Russian is
-written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in
-Hiragana or Katakana. There are many more.
-
-The Unicode Script and Script_Extensions properties give what script a
-given character is in. Either property can be specified with the
-compound form like
-C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>), or
-C<\p{Script_Extensions=Javanese}> (short: C<\p{scx=java}>).
-In addition, Perl furnishes shortcuts for all
-C<Script> property names. You can omit everything up through the equals
-(or colon), and simply write C<\p{Latin}> or C<\P{Cyrillic}>.
-(This is not true for C<Script_Extensions>, which is required to be
-written in the compound form.)
-
-The difference between these two properties involves characters that are
-used in multiple scripts. For example the digits '0' through '9' are
-used in many parts of the world. These are placed in a script named
-C<Common>. Other characters are used in just a few scripts. For
-example, the "KATAKANA-HIRAGANA DOUBLE HYPHEN" is used in both Japanese
-scripts, Katakana and Hiragana, but nowhere else. The C<Script>
-property places all characters that are used in multiple scripts in the
-C<Common> script, while the C<Script_Extensions> property places those
-that are used in only a few scripts into each of those scripts; while
-still using C<Common> for those used in many scripts. Thus both these
-match:
-
- "0" =~ /\p{sc=Common}/ # Matches
- "0" =~ /\p{scx=Common}/ # Matches
-
-and only the first of these match:
-
- "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Common} # Matches
- "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Common} # No match
-
-And only the last two of these match:
-
- "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Hiragana} # No match
- "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Katakana} # No match
- "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Hiragana} # Matches
- "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Katakana} # Matches
-
-C<Script_Extensions> is thus an improved C<Script>, in which there are
-fewer characters in the C<Common> script, and correspondingly more in
-other scripts. It is new in Unicode version 6.0, and its data are likely
-to change significantly in later releases, as things get sorted out.
-
-(Actually, besides C<Common>, the C<Inherited> script, contains
-characters that are used in multiple scripts. These are modifier
-characters which modify other characters, and inherit the script value
-of the controlling character. Some of these are used in many scripts,
-and so go into C<Inherited> in both C<Script> and C<Script_Extensions>.
-Others are used in just a few scripts, so are in C<Inherited> in
-C<Script>, but not in C<Script_Extensions>.)
-
-It is worth stressing that there are several different sets of digits in
-Unicode that are equivalent to 0-9 and are matchable by C<\d> in a
-regular expression. If they are used in a single language only, they
-are in that language's C<Script> and C<Script_Extension>. If they are
-used in more than one script, they will be in C<sc=Common>, but only
-if they are used in many scripts should they be in C<scx=Common>.
-
-A complete list of scripts and their shortcuts is in L<perluniprops>.
-
-=head3 B<Use of "Is" Prefix>
-
-For backward compatibility (with Perl 5.6), all properties mentioned
-so far may have C<Is> or C<Is_> prepended to their name, so C<\P{Is_Lu}>, for
-example, is equal to C<\P{Lu}>, and C<\p{IsScript:Arabic}> is equal to
-C<\p{Arabic}>.
-
-=head3 B<Blocks>
-
-In addition to B<scripts>, Unicode also defines B<blocks> of
-characters. The difference between scripts and blocks is that the
-concept of scripts is closer to natural languages, while the concept
-of blocks is more of an artificial grouping based on groups of Unicode
-characters with consecutive ordinal values. For example, the "Basic Latin"
-block is all characters whose ordinals are between 0 and 127, inclusive; in
-other words, the ASCII characters. The "Latin" script contains some letters
-from this as well as several other blocks, like "Latin-1 Supplement",
-"Latin Extended-A", etc., but it does not contain all the characters from
-those blocks. It does not, for example, contain the digits 0-9, because
-those digits are shared across many scripts, and hence are in the
-C<Common> script.
-
-For more about scripts versus blocks, see UAX#24 "Unicode Script Property":
-L<http://www.unicode.org/reports/tr24>
-
-The C<Script> or C<Script_Extensions> properties are likely to be the
-ones you want to use when processing
-natural language; the Block property may occasionally be useful in working
-with the nuts and bolts of Unicode.
-
-Block names are matched in the compound form, like C<\p{Block: Arrows}> or
-C<\p{Blk=Hebrew}>. Unlike most other properties, only a few block names have a
-Unicode-defined short name. But Perl does provide a (slight) shortcut: You
-can say, for example C<\p{In_Arrows}> or C<\p{In_Hebrew}>. For backwards
-compatibility, the C<In> prefix may be omitted if there is no naming conflict
-with a script or any other property, and you can even use an C<Is> prefix
-instead in those cases. But it is not a good idea to do this, for a couple
-reasons:
-
-=over 4
-
-=item 1
-
-It is confusing. There are many naming conflicts, and you may forget some.
-For example, C<\p{Hebrew}> means the I<script> Hebrew, and NOT the I<block>
-Hebrew. But would you remember that 6 months from now?
-
-=item 2
-
-It is unstable. A new version of Unicode may pre-empt the current meaning by
-creating a property with the same name. There was a time in very early Unicode
-releases when C<\p{Hebrew}> would have matched the I<block> Hebrew; now it
-doesn't.
-
-=back
-
-Some people prefer to always use C<\p{Block: foo}> and C<\p{Script: bar}>
-instead of the shortcuts, whether for clarity, because they can't remember the
-difference between 'In' and 'Is' anyway, or they aren't confident that those who
-eventually will read their code will know that difference.
-
-A complete list of blocks and their shortcuts is in L<perluniprops>.
-
-=head3 B<Other Properties>
-
-There are many more properties than the very basic ones described here.
-A complete list is in L<perluniprops>.
-
-Unicode defines all its properties in the compound form, so all single-form
-properties are Perl extensions. Most of these are just synonyms for the
-Unicode ones, but some are genuine extensions, including several that are in
-the compound form. And quite a few of these are actually recommended by Unicode
-(in L<http://www.unicode.org/reports/tr18>).
-
-This section gives some details on all extensions that aren't just
-synonyms for compound-form Unicode properties
-(for those properties, you'll have to refer to the
-L<Unicode Standard|http://www.unicode.org/reports/tr44>.
-
-=over
-
-=item B<C<\p{All}>>
-
-This matches any of the 1_114_112 Unicode code points. It is a synonym for
-C<\p{Any}>.
-
-=item B<C<\p{Alnum}>>
-
-This matches any C<\p{Alphabetic}> or C<\p{Decimal_Number}> character.
-
-=item B<C<\p{Any}>>
-
-This matches any of the 1_114_112 Unicode code points. It is a synonym for
-C<\p{All}>.
-
-=item B<C<\p{ASCII}>>
-
-This matches any of the 128 characters in the US-ASCII character set,
-which is a subset of Unicode.
-
-=item B<C<\p{Assigned}>>
-
-This matches any assigned code point; that is, any code point whose general
-category is not Unassigned (or equivalently, not Cn).
-
-=item B<C<\p{Blank}>>
-
-This is the same as C<\h> and C<\p{HorizSpace}>: A character that changes the
-spacing horizontally.
-
-=item B<C<\p{Decomposition_Type: Non_Canonical}>> (Short: C<\p{Dt=NonCanon}>)
-
-Matches a character that has a non-canonical decomposition.
-
-To understand the use of this rarely used property=value combination, it is
-necessary to know some basics about decomposition.
-Consider a character, say H. It could appear with various marks around it,
-such as an acute accent, or a circumflex, or various hooks, circles, arrows,
-I<etc.>, above, below, to one side or the other, etc. There are many
-possibilities among the world's languages. The number of combinations is
-astronomical, and if there were a character for each combination, it would
-soon exhaust Unicode's more than a million possible characters. So Unicode
-took a different approach: there is a character for the base H, and a
-character for each of the possible marks, and these can be variously combined
-to get a final logical character. So a logical character--what appears to be a
-single character--can be a sequence of more than one individual characters.
-This is called an "extended grapheme cluster"; Perl furnishes the C<\X>
-regular expression construct to match such sequences.
-
-But Unicode's intent is to unify the existing character set standards and
-practices, and several pre-existing standards have single characters that
-mean the same thing as some of these combinations. An example is ISO-8859-1,
-which has quite a few of these in the Latin-1 range, an example being "LATIN
-CAPITAL LETTER E WITH ACUTE". Because this character was in this pre-existing
-standard, Unicode added it to its repertoire. But this character is considered
-by Unicode to be equivalent to the sequence consisting of the character
-"LATIN CAPITAL LETTER E" followed by the character "COMBINING ACUTE ACCENT".
-
-"LATIN CAPITAL LETTER E WITH ACUTE" is called a "pre-composed" character, and
-its equivalence with the sequence is called canonical equivalence. All
-pre-composed characters are said to have a decomposition (into the equivalent
-sequence), and the decomposition type is also called canonical.
-
-However, many more characters have a different type of decomposition, a
-"compatible" or "non-canonical" decomposition. The sequences that form these
-decompositions are not considered canonically equivalent to the pre-composed
-character. An example, again in the Latin-1 range, is the "SUPERSCRIPT ONE".
-It is somewhat like a regular digit 1, but not exactly; its decomposition
-into the digit 1 is called a "compatible" decomposition, specifically a
-"super" decomposition. There are several such compatibility
-decompositions (see L<http://www.unicode.org/reports/tr44>), including one
-called "compat", which means some miscellaneous type of decomposition
-that doesn't fit into the decomposition categories that Unicode has chosen.
-
-Note that most Unicode characters don't have a decomposition, so their
-decomposition type is "None".
-
-For your convenience, Perl has added the C<Non_Canonical> decomposition
-type to mean any of the several compatibility decompositions.
-
-=item B<C<\p{Graph}>>
-
-Matches any character that is graphic. Theoretically, this means a character
-that on a printer would cause ink to be used.
-
-=item B<C<\p{HorizSpace}>>
-
-This is the same as C<\h> and C<\p{Blank}>: a character that changes the
-spacing horizontally.
-
-=item B<C<\p{In=*}>>
-
-This is a synonym for C<\p{Present_In=*}>
-
-=item B<C<\p{PerlSpace}>>
-
-This is the same as C<\s>, restricted to ASCII, namely C<S<[ \f\n\r\t]>>.
-
-Mnemonic: Perl's (original) space
-
-=item B<C<\p{PerlWord}>>
-
-This is the same as C<\w>, restricted to ASCII, namely C<[A-Za-z0-9_]>
-
-Mnemonic: Perl's (original) word.
-
-=item B<C<\p{Posix...}>>
-
-There are several of these, which are equivalents using the C<\p>
-notation for Posix classes and are described in
-L<perlrecharclass/POSIX Character Classes>.
-
-=item B<C<\p{Present_In: *}>> (Short: C<\p{In=*}>)
-
-This property is used when you need to know in what Unicode version(s) a
-character is.
-
-The "*" above stands for some two digit Unicode version number, such as
-C<1.1> or C<4.0>; or the "*" can also be C<Unassigned>. This property will
-match the code points whose final disposition has been settled as of the
-Unicode release given by the version number; C<\p{Present_In: Unassigned}>
-will match those code points whose meaning has yet to be assigned.
-
-For example, C<U+0041> "LATIN CAPITAL LETTER A" was present in the very first
-Unicode release available, which is C<1.1>, so this property is true for all
-valid "*" versions. On the other hand, C<U+1EFF> was not assigned until version
-5.1 when it became "LATIN SMALL LETTER Y WITH LOOP", so the only "*" that
-would match it are 5.1, 5.2, and later.
-
-Unicode furnishes the C<Age> property from which this is derived. The problem
-with Age is that a strict interpretation of it (which Perl takes) has it
-matching the precise release a code point's meaning is introduced in. Thus
-C<U+0041> would match only 1.1; and C<U+1EFF> only 5.1. This is not usually what
-you want.
-
-Some non-Perl implementations of the Age property may change its meaning to be
-the same as the Perl Present_In property; just be aware of that.
-
-Another confusion with both these properties is that the definition is not
-that the code point has been I<assigned>, but that the meaning of the code point
-has been I<determined>. This is because 66 code points will always be
-unassigned, and so the Age for them is the Unicode version in which the decision
-to make them so was made. For example, C<U+FDD0> is to be permanently
-unassigned to a character, and the decision to do that was made in version 3.1,
-so C<\p{Age=3.1}> matches this character, as also does C<\p{Present_In: 3.1}> and up.
-
-=item B<C<\p{Print}>>
-
-This matches any character that is graphical or blank, except controls.
-
-=item B<C<\p{SpacePerl}>>
-
-This is the same as C<\s>, including beyond ASCII.
-
-Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab
-which both the Posix standard and Unicode consider white space.)
-
-=item B<C<\p{Title}>> and B<C<\p{Titlecase}>>
-
-Under case-sensitive matching, these both match the same code points as
-C<\p{General Category=Titlecase_Letter}> (C<\p{gc=lt}>). The difference
-is that under C</i> caseless matching, these match the same as
-C<\p{Cased}>, whereas C<\p{gc=lt}> matches C<\p{Cased_Letter>).
-
-=item B<C<\p{VertSpace}>>
-
-This is the same as C<\v>: A character that changes the spacing vertically.
-
-=item B<C<\p{Word}>>
-
-This is the same as C<\w>, including over 100_000 characters beyond ASCII.
-
-=item B<C<\p{XPosix...}>>
-
-There are several of these, which are the standard Posix classes
-extended to the full Unicode range. They are described in
-L<perlrecharclass/POSIX Character Classes>.
-
-=back
-
-=head2 User-Defined Character Properties
-
-You can define your own binary character properties by defining subroutines
-whose names begin with "In" or "Is". The subroutines can be defined in any
-package. The user-defined properties can be used in the regular expression
-C<\p> and C<\P> constructs; if you are using a user-defined property from a
-package other than the one you are in, you must specify its package in the
-C<\p> or C<\P> construct.
-
- # assuming property Is_Foreign defined in Lang::
- package main; # property package name required
- if ($txt =~ /\p{Lang::IsForeign}+/) { ... }
-
- package Lang; # property package name not required
- if ($txt =~ /\p{IsForeign}+/) { ... }
-
-
-Note that the effect is compile-time and immutable once defined.
-However, the subroutines are passed a single parameter, which is 0 if
-case-sensitive matching is in effect and non-zero if caseless matching
-is in effect. The subroutine may return different values depending on
-the value of the flag, and one set of values will immutably be in effect
-for all case-sensitive matches, and the other set for all case-insensitive
-matches.
-
-Note that if the regular expression is tainted, then Perl will die rather
-than calling the subroutine, where the name of the subroutine is
-determined by the tainted data.
-
-The subroutines must return a specially-formatted string, with one
-or more newline-separated lines. Each line must be one of the following:
-
-=over 4
-
-=item *
-
-A single hexadecimal number denoting a Unicode code point to include.
-
-=item *
-
-Two hexadecimal numbers separated by horizontal whitespace (space or
-tabular characters) denoting a range of Unicode code points to include.
-
-=item *
-
-Something to include, prefixed by "+": a built-in character
-property (prefixed by "utf8::") or a fully qualified (including package
-name) user-defined character property,
-to represent all the characters in that property; two hexadecimal code
-points for a range; or a single hexadecimal code point.
-
-=item *
-
-Something to exclude, prefixed by "-": an existing character
-property (prefixed by "utf8::") or a fully qualified (including package
-name) user-defined character property,
-to represent all the characters in that property; two hexadecimal code
-points for a range; or a single hexadecimal code point.
-
-=item *
-
-Something to negate, prefixed "!": an existing character
-property (prefixed by "utf8::") or a fully qualified (including package
-name) user-defined character property,
-to represent all the characters in that property; two hexadecimal code
-points for a range; or a single hexadecimal code point.
-
-=item *
-
-Something to intersect with, prefixed by "&": an existing character
-property (prefixed by "utf8::") or a fully qualified (including package
-name) user-defined character property,
-for all the characters except the characters in the property; two
-hexadecimal code points for a range; or a single hexadecimal code point.
-
-=back
-
-For example, to define a property that covers both the Japanese
-syllabaries (hiragana and katakana), you can define
-
- sub InKana {
- return <<END;
- 3040\t309F
- 30A0\t30FF
- END
- }
-
-Imagine that the here-doc end marker is at the beginning of the line.
-Now you can use C<\p{InKana}> and C<\P{InKana}>.
-
-You could also have used the existing block property names:
-
- sub InKana {
- return <<'END';
- +utf8::InHiragana
- +utf8::InKatakana
- END
- }
-
-Suppose you wanted to match only the allocated characters,
-not the raw block ranges: in other words, you want to remove
-the non-characters:
-
- sub InKana {
- return <<'END';
- +utf8::InHiragana
- +utf8::InKatakana
- -utf8::IsCn
- END
- }
-
-The negation is useful for defining (surprise!) negated classes.
-
- sub InNotKana {
- return <<'END';
- !utf8::InHiragana
- -utf8::InKatakana
- +utf8::IsCn
- END
- }
-
-This will match all non-Unicode code points, since every one of them is
-not in Kana. You can use intersection to exclude these, if desired, as
-this modified example shows:
-
- sub InNotKana {
- return <<'END';
- !utf8::InHiragana
- -utf8::InKatakana
- +utf8::IsCn
- &utf8::Any
- END
- }
-
-C<&utf8::Any> must be the last line in the definition.
-
-Intersection is used generally for getting the common characters matched
-by two (or more) classes. It's important to remember not to use "&" for
-the first set; that would be intersecting with nothing, resulting in an
-empty set.
-
-(Note that official Unicode properties differ from these in that they
-automatically exclude non-Unicode code points and a warning is raised if
-a match is attempted on one of those.)
-
-=head2 User-Defined Case Mappings (for serious hackers only)
-
-B<This feature has been removed as of Perl 5.16.>
-The CPAN module L<Unicode::Casing> provides better functionality without
-the drawbacks that this feature had. If you are using a Perl earlier
-than 5.16, this feature was most fully documented in the 5.14 version of
-this pod:
-L<http://perldoc.perl.org/5.14.0/perlunicode.html#User-Defined-Case-Mappings-%28for-serious-hackers-only%29>
-
-=head2 Character Encodings for Input and Output
-
-See L<Encode>.
-
-=head2 Unicode Regular Expression Support Level
-
-The following list of Unicode supported features for regular expressions describes
-all features currently directly supported by core Perl. The references to "Level N"
-and the section numbers refer to the Unicode Technical Standard #18,
-"Unicode Regular Expressions", version 13, from August 2008.
-
-=over 4
-
-=item *
-
-Level 1 - Basic Unicode Support
-
- RL1.1 Hex Notation - done [1]
- RL1.2 Properties - done [2][3]
- RL1.2a Compatibility Properties - done [4]
- RL1.3 Subtraction and Intersection - MISSING [5]
- RL1.4 Simple Word Boundaries - done [6]
- RL1.5 Simple Loose Matches - done [7]
- RL1.6 Line Boundaries - MISSING [8][9]
- RL1.7 Supplementary Code Points - done [10]
-
- [1] \x{...}
- [2] \p{...} \P{...}
- [3] supports not only minimal list, but all Unicode character
- properties (see Unicode Character Properties above)
- [4] \d \D \s \S \w \W \X [:prop:] [:^prop:]
- [5] can use regular expression look-ahead [a] or
- user-defined character properties [b] to emulate set
- operations
- [6] \b \B
- [7] note that Perl does Full case-folding in matching (but with
- bugs), not Simple: for example U+1F88 is equivalent to
- U+1F00 U+03B9, instead of just U+1F80. This difference
- matters mainly for certain Greek capital letters with certain
- modifiers: the Full case-folding decomposes the letter,
- while the Simple case-folding would map it to a single
- character.
- [8] should do ^ and $ also on U+000B (\v in C), FF (\f), CR
- (\r), CRLF (\r\n), NEL (U+0085), LS (U+2028), and PS
- (U+2029); should also affect <>, $., and script line
- numbers; should not split lines within CRLF [c] (i.e. there
- is no empty line between \r and \n)
- [9] Linebreaking conformant with UAX#14 "Unicode Line Breaking
- Algorithm" is available through the Unicode::LineBreaking
- module.
- [10] UTF-8/UTF-EBDDIC used in Perl allows not only U+10000 to
- U+10FFFF but also beyond U+10FFFF
-
-[a] You can mimic class subtraction using lookahead.
-For example, what UTS#18 might write as
-
- [{Greek}-[{UNASSIGNED}]]
-
-in Perl can be written as:
-
- (?!\p{Unassigned})\p{InGreekAndCoptic}
- (?=\p{Assigned})\p{InGreekAndCoptic}
-
-But in this particular example, you probably really want
-
- \p{GreekAndCoptic}
-
-which will match assigned characters known to be part of the Greek script.
-
-Also see the L<Unicode::Regex::Set> module; it does implement the full
-UTS#18 grouping, intersection, union, and removal (subtraction) syntax.
-
-[b] '+' for union, '-' for removal (set-difference), '&' for intersection
-(see L</"User-Defined Character Properties">)
-
-[c] Try the C<:crlf> layer (see L<PerlIO>).
-
-=item *
-
-Level 2 - Extended Unicode Support
-
- RL2.1 Canonical Equivalents - MISSING [10][11]
- RL2.2 Default Grapheme Clusters - MISSING [12]
- RL2.3 Default Word Boundaries - MISSING [14]
- RL2.4 Default Loose Matches - MISSING [15]
- RL2.5 Name Properties - DONE
- RL2.6 Wildcard Properties - MISSING
-
- [10] see UAX#15 "Unicode Normalization Forms"
- [11] have Unicode::Normalize but not integrated to regexes
- [12] have \X but we don't have a "Grapheme Cluster Mode"
- [14] see UAX#29, Word Boundaries
- [15] This is covered in Chapter 3.13 (in Unicode 6.0)
-
-=item *
-
-Level 3 - Tailored Support
-
- RL3.1 Tailored Punctuation - MISSING
- RL3.2 Tailored Grapheme Clusters - MISSING [17][18]
- RL3.3 Tailored Word Boundaries - MISSING
- RL3.4 Tailored Loose Matches - MISSING
- RL3.5 Tailored Ranges - MISSING
- RL3.6 Context Matching - MISSING [19]
- RL3.7 Incremental Matches - MISSING
- ( RL3.8 Unicode Set Sharing )
- RL3.9 Possible Match Sets - MISSING
- RL3.10 Folded Matching - MISSING [20]
- RL3.11 Submatchers - MISSING
-
- [17] see UAX#10 "Unicode Collation Algorithms"
- [18] have Unicode::Collate but not integrated to regexes
- [19] have (?<=x) and (?=x), but look-aheads or look-behinds
- should see outside of the target substring
- [20] need insensitive matching for linguistic features other
- than case; for example, hiragana to katakana, wide and
- narrow, simplified Han to traditional Han (see UTR#30
- "Character Foldings")
-
-=back
-
-=head2 Unicode Encodings
-
-Unicode characters are assigned to I<code points>, which are abstract
-numbers. To use these numbers, various encodings are needed.
-
-=over 4
-
-=item *
-
-UTF-8
-
-UTF-8 is a variable-length (1 to 4 bytes), byte-order independent
-encoding. For ASCII (and we really do mean 7-bit ASCII, not another
-8-bit encoding), UTF-8 is transparent.
-
-The following table is from Unicode 3.2.
-
- Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
-
- U+0000..U+007F 00..7F
- U+0080..U+07FF * C2..DF 80..BF
- U+0800..U+0FFF E0 * A0..BF 80..BF
- U+1000..U+CFFF E1..EC 80..BF 80..BF
- U+D000..U+D7FF ED 80..9F 80..BF
- U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++
- U+E000..U+FFFF EE..EF 80..BF 80..BF
- U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF
- U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
- U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
-
-Note the gaps marked by "*" before several of the byte entries above. These are
-caused by legal UTF-8 avoiding non-shortest encodings: it is technically
-possible to UTF-8-encode a single code point in different ways, but that is
-explicitly forbidden, and the shortest possible encoding should always be used
-(and that is what Perl does).
-
-Another way to look at it is via bits:
-
- Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
-
- 0aaaaaaa 0aaaaaaa
- 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
- ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
- 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
-
-As you can see, the continuation bytes all begin with "10", and the
-leading bits of the start byte tell how many bytes there are in the
-encoded character.
-
-The original UTF-8 specification allowed up to 6 bytes, to allow
-encoding of numbers up to 0x7FFF_FFFF. Perl continues to allow those,
-and has extended that up to 13 bytes to encode code points up to what
-can fit in a 64-bit word. However, Perl will warn if you output any of
-these as being non-portable; and under strict UTF-8 input protocols,
-they are forbidden.
-
-The Unicode non-character code points are also disallowed in UTF-8 in
-"open interchange". See L</Non-character code points>.
-
-=item *
-
-UTF-EBCDIC
-
-Like UTF-8 but EBCDIC-safe, in the way that UTF-8 is ASCII-safe.
-
-=item *
-
-UTF-16, UTF-16BE, UTF-16LE, Surrogates, and BOMs (Byte Order Marks)
-
-The followings items are mostly for reference and general Unicode
-knowledge, Perl doesn't use these constructs internally.
-
-Like UTF-8, UTF-16 is a variable-width encoding, but where
-UTF-8 uses 8-bit code units, UTF-16 uses 16-bit code units.
-All code points occupy either 2 or 4 bytes in UTF-16: code points
-C<U+0000..U+FFFF> are stored in a single 16-bit unit, and code
-points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is
-using I<surrogates>, the first 16-bit unit being the I<high
-surrogate>, and the second being the I<low surrogate>.
-
-Surrogates are code points set aside to encode the C<U+10000..U+10FFFF>
-range of Unicode code points in pairs of 16-bit units. The I<high
-surrogates> are the range C<U+D800..U+DBFF> and the I<low surrogates>
-are the range C<U+DC00..U+DFFF>. The surrogate encoding is
-
- $hi = ($uni - 0x10000) / 0x400 + 0xD800;
- $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
-
-and the decoding is
-
- $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
-
-Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16
-itself can be used for in-memory computations, but if storage or
-transfer is required either UTF-16BE (big-endian) or UTF-16LE
-(little-endian) encodings must be chosen.
-
-This introduces another problem: what if you just know that your data
-is UTF-16, but you don't know which endianness? Byte Order Marks, or
-BOMs, are a solution to this. A special character has been reserved
-in Unicode to function as a byte order marker: the character with the
-code point C<U+FEFF> is the BOM.
-
-The trick is that if you read a BOM, you will know the byte order,
-since if it was written on a big-endian platform, you will read the
-bytes C<0xFE 0xFF>, but if it was written on a little-endian platform,
-you will read the bytes C<0xFF 0xFE>. (And if the originating platform
-was writing in UTF-8, you will read the bytes C<0xEF 0xBB 0xBF>.)
-
-The way this trick works is that the character with the code point
-C<U+FFFE> is not supposed to be in input streams, so the
-sequence of bytes C<0xFF 0xFE> is unambiguously "BOM, represented in
-little-endian format" and cannot be C<U+FFFE>, represented in big-endian
-format".
-
-Surrogates have no meaning in Unicode outside their use in pairs to
-represent other code points. However, Perl allows them to be
-represented individually internally, for example by saying
-C<chr(0xD801)>, so that all code points, not just those valid for open
-interchange, are
-representable. Unicode does define semantics for them, such as their
-General Category is "Cs". But because their use is somewhat dangerous,
-Perl will warn (using the warning category "surrogate", which is a
-sub-category of "utf8") if an attempt is made
-to do things like take the lower case of one, or match
-case-insensitively, or to output them. (But don't try this on Perls
-before 5.14.)
-
-=item *
-
-UTF-32, UTF-32BE, UTF-32LE
-
-The UTF-32 family is pretty much like the UTF-16 family, expect that
-the units are 32-bit, and therefore the surrogate scheme is not
-needed. UTF-32 is a fixed-width encoding. The BOM signatures are
-C<0x00 0x00 0xFE 0xFF> for BE and C<0xFF 0xFE 0x00 0x00> for LE.
-
-=item *
-
-UCS-2, UCS-4
-
-Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
-encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>,
-because it does not use surrogates. UCS-4 is a 32-bit encoding,
-functionally identical to UTF-32 (the difference being that
-UCS-4 forbids neither surrogates nor code points larger than 0x10_FFFF).
-
-=item *
-
-UTF-7
-
-A seven-bit safe (non-eight-bit) encoding, which is useful if the
-transport or storage is not eight-bit safe. Defined by RFC 2152.
-
-=back
-
-=head2 Non-character code points
-
-66 code points are set aside in Unicode as "non-character code points".
-These all have the Unassigned (Cn) General Category, and they never will
-be assigned. These are never supposed to be in legal Unicode input
-streams, so that code can use them as sentinels that can be mixed in
-with character data, and they always will be distinguishable from that data.
-To keep them out of Perl input streams, strict UTF-8 should be
-specified, such as by using the layer C<:encoding('UTF-8')>. The
-non-character code points are the 32 between U+FDD0 and U+FDEF, and the
-34 code points U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, ... U+10FFFE, U+10FFFF.
-Some people are under the mistaken impression that these are "illegal",
-but that is not true. An application or cooperating set of applications
-can legally use them at will internally; but these code points are
-"illegal for open interchange". Therefore, Perl will not accept these
-from input streams unless lax rules are being used, and will warn
-(using the warning category "nonchar", which is a sub-category of "utf8") if
-an attempt is made to output them.
-
-=head2 Beyond Unicode code points
-
-The maximum Unicode code point is U+10FFFF. But Perl accepts code
-points up to the maximum permissible unsigned number available on the
-platform. However, Perl will not accept these from input streams unless
-lax rules are being used, and will warn (using the warning category
-"non_unicode", which is a sub-category of "utf8") if an attempt is made to
-operate on or output them. For example, C<uc(0x11_0000)> will generate
-this warning, returning the input parameter as its result, as the upper
-case of every non-Unicode code point is the code point itself.
-
-=head2 Security Implications of Unicode
-
-Read L<Unicode Security Considerations|http://www.unicode.org/reports/tr36>.
-Also, note the following:
-
-=over 4
-
-=item *
-
-Malformed UTF-8
-
-Unfortunately, the original specification of UTF-8 leaves some room for
-interpretation of how many bytes of encoded output one should generate
-from one input Unicode character. Strictly speaking, the shortest
-possible sequence of UTF-8 bytes should be generated,
-because otherwise there is potential for an input buffer overflow at
-the receiving end of a UTF-8 connection. Perl always generates the
-shortest length UTF-8, and with warnings on, Perl will warn about
-non-shortest length UTF-8 along with other malformations, such as the
-surrogates, which are not Unicode code points valid for interchange.
-
-=item *
-
-Regular expression pattern matching may surprise you if you're not
-accustomed to Unicode. Starting in Perl 5.14, several pattern
-modifiers are available to control this, called the character set
-modifiers. Details are given in L<perlre/Character set modifiers>.
-
-=back
-
-As discussed elsewhere, Perl has one foot (two hooves?) planted in
-each of two worlds: the old world of bytes and the new world of
-characters, upgrading from bytes to characters when necessary.
-If your legacy code does not explicitly use Unicode, no automatic
-switch-over to characters should happen. Characters shouldn't get
-downgraded to bytes, either. It is possible to accidentally mix bytes
-and characters, however (see L<perluniintro>), in which case C<\w> in
-regular expressions might start behaving differently (unless the C</a>
-modifier is in effect). Review your code. Use warnings and the C<strict> pragma.
-
-=head2 Unicode in Perl on EBCDIC
-
-The way Unicode is handled on EBCDIC platforms is still
-experimental. On such platforms, references to UTF-8 encoding in this
-document and elsewhere should be read as meaning the UTF-EBCDIC
-specified in Unicode Technical Report 16, unless ASCII vs. EBCDIC issues
-are specifically discussed. There is no C<utfebcdic> pragma or
-":utfebcdic" layer; rather, "utf8" and ":utf8" are reused to mean
-the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
-for more discussion of the issues.
-
-=head2 Locales
-
-See L<perllocale/Unicode and UTF-8>
-
-=head2 When Unicode Does Not Happen
-
-While Perl does have extensive ways to input and output in Unicode,
-and a few other "entry points" like the @ARGV array (which can sometimes be
-interpreted as UTF-8), there are still many places where Unicode
-(in some encoding or another) could be given as arguments or received as
-results, or both, but it is not.
-
-The following are such interfaces. Also, see L</The "Unicode Bug">.
-For all of these interfaces Perl
-currently (as of 5.8.3) simply assumes byte strings both as arguments
-and results, or UTF-8 strings if the (problematic) C<encoding> pragma has been used.
-
-One reason that Perl does not attempt to resolve the role of Unicode in
-these situations is that the answers are highly dependent on the operating
-system and the file system(s). For example, whether filenames can be
-in Unicode and in exactly what kind of encoding, is not exactly a
-portable concept. Similarly for C<qx> and C<system>: how well will the
-"command-line interface" (and which of them?) handle Unicode?
-
-=over 4
-
-=item *
-
-chdir, chmod, chown, chroot, exec, link, lstat, mkdir,
-rename, rmdir, stat, symlink, truncate, unlink, utime, -X
-
-=item *
-
-%ENV
-
-=item *
-
-glob (aka the <*>)
-
-=item *
-
-open, opendir, sysopen
-
-=item *
-
-qx (aka the backtick operator), system
-
-=item *
-
-readdir, readlink
-
-=back
-
-=head2 The "Unicode Bug"
-
-The term, "Unicode bug" has been applied to an inconsistency
-on ASCII platforms with the
-Unicode code points in the Latin-1 Supplement block, that
-is, between 128 and 255. Without a locale specified, unlike all other
-characters or code points, these characters have very different semantics in
-byte semantics versus character semantics, unless
-C<use feature 'unicode_strings'> is specified, directly or indirectly.
-(It is indirectly specified by a C<use v5.12> or higher.)
-
-In character semantics these upper-Latin1 characters are interpreted as
-Unicode code points, which means
-they have the same semantics as Latin-1 (ISO-8859-1).
-
-In byte semantics (without C<unicode_strings>), they are considered to
-be unassigned characters, meaning that the only semantics they have is
-their ordinal numbers, and that they are
-not members of various character classes. None are considered to match C<\w>
-for example, but all match C<\W>.
-
-Perl 5.12.0 added C<unicode_strings> to force character semantics on
-these code points in some circumstances, which fixed portions of the
-bug; Perl 5.14.0 fixed almost all of it; and Perl 5.16.0 fixed the
-remainder (so far as we know, anyway). The lesson here is to enable
-C<unicode_strings> to avoid the headaches described below.
-
-The old, problematic behavior affects these areas:
-
-=over 4
-
-=item *
-
-Changing the case of a scalar, that is, using C<uc()>, C<ucfirst()>, C<lc()>,
-and C<lcfirst()>, or C<\L>, C<\U>, C<\u> and C<\l> in double-quotish
-contexts, such as regular expression substitutions.
-Under C<unicode_strings> starting in Perl 5.12.0, character semantics are
-generally used. See L<perlfunc/lc> for details on how this works
-in combination with various other pragmas.
-
-=item *
-
-Using caseless (C</i>) regular expression matching.
-Starting in Perl 5.14.0, regular expressions compiled within
-the scope of C<unicode_strings> use character semantics
-even when executed or compiled into larger
-regular expressions outside the scope.
-
-=item *
-
-Matching any of several properties in regular expressions, namely C<\b>,
-C<\B>, C<\s>, C<\S>, C<\w>, C<\W>, and all the Posix character classes
-I<except> C<[[:ascii:]]>.
-Starting in Perl 5.14.0, regular expressions compiled within
-the scope of C<unicode_strings> use character semantics
-even when executed or compiled into larger
-regular expressions outside the scope.
-
-=item *
-
-In C<quotemeta> or its inline equivalent C<\Q>, no code points above 127
-are quoted in UTF-8 encoded strings, but in byte encoded strings, code
-points between 128-255 are always quoted.
-Starting in Perl 5.16.0, consistent quoting rules are used within the
-scope of C<unicode_strings>, as described in L<perlfunc/quotemeta>.
-
-=back
-
-This behavior can lead to unexpected results in which a string's semantics
-suddenly change if a code point above 255 is appended to or removed from it,
-which changes the string's semantics from byte to character or vice versa. As
-an example, consider the following program and its output:
-
- $ perl -le'
- no feature 'unicode_strings';
- $s1 = "\xC2";
- $s2 = "\x{2660}";
- for ($s1, $s2, $s1.$s2) {
- print /\w/ || 0;
- }
- '
- 0
- 0
- 1
-
-If there's no C<\w> in C<s1> or in C<s2>, why does their concatenation have one?
-
-This anomaly stems from Perl's attempt to not disturb older programs that
-didn't use Unicode, and hence had no semantics for characters outside of the
-ASCII range (except in a locale), along with Perl's desire to add Unicode
-support seamlessly. The result wasn't seamless: these characters were
-orphaned.
-
-For Perls earlier than those described above, or when a string is passed
-to a function outside the subpragma's scope, a workaround is to always
-call C<utf8::upgrade($string)>,
-or to use the standard module L<Encode>. Also, a scalar that has any characters
-whose ordinal is above 0x100, or which were specified using either of the
-C<\N{...}> notations, will automatically have character semantics.
-
-=head2 Forcing Unicode in Perl (Or Unforcing Unicode in Perl)
-
-Sometimes (see L</"When Unicode Does Not Happen"> or L</The "Unicode Bug">)
-there are situations where you simply need to force a byte
-string into UTF-8, or vice versa. The low-level calls
-utf8::upgrade($bytestring) and utf8::downgrade($utf8string[, FAIL_OK]) are
-the answers.
-
-Note that utf8::downgrade() can fail if the string contains characters
-that don't fit into a byte.
-
-Calling either function on a string that already is in the desired state is a
-no-op.
-
-=head2 Using Unicode in XS
-
-If you want to handle Perl Unicode in XS extensions, you may find the
-following C APIs useful. See also L<perlguts/"Unicode Support"> for an
-explanation about Unicode at the XS level, and L<perlapi> for the API
-details.
-
-=over 4
-
-=item *
-
-C<DO_UTF8(sv)> returns true if the C<UTF8> flag is on and the bytes
-pragma is not in effect. C<SvUTF8(sv)> returns true if the C<UTF8>
-flag is on; the bytes pragma is ignored. The C<UTF8> flag being on
-does B<not> mean that there are any characters of code points greater
-than 255 (or 127) in the scalar or that there are even any characters
-in the scalar. What the C<UTF8> flag means is that the sequence of
-octets in the representation of the scalar is the sequence of UTF-8
-encoded code points of the characters of a string. The C<UTF8> flag
-being off means that each octet in this representation encodes a
-single character with code point 0..255 within the string. Perl's
-Unicode model is not to use UTF-8 until it is absolutely necessary.
-
-=item *
-
-C<uvchr_to_utf8(buf, chr)> writes a Unicode character code point into
-a buffer encoding the code point as UTF-8, and returns a pointer
-pointing after the UTF-8 bytes. It works appropriately on EBCDIC machines.
-
-=item *
-
-C<utf8_to_uvchr_buf(buf, bufend, lenp)> reads UTF-8 encoded bytes from a
-buffer and
-returns the Unicode character code point and, optionally, the length of
-the UTF-8 byte sequence. It works appropriately on EBCDIC machines.
-
-=item *
-
-C<utf8_length(start, end)> returns the length of the UTF-8 encoded buffer
-in characters. C<sv_len_utf8(sv)> returns the length of the UTF-8 encoded
-scalar.
-
-=item *
-
-C<sv_utf8_upgrade(sv)> converts the string of the scalar to its UTF-8
-encoded form. C<sv_utf8_downgrade(sv)> does the opposite, if
-possible. C<sv_utf8_encode(sv)> is like sv_utf8_upgrade except that
-it does not set the C<UTF8> flag. C<sv_utf8_decode()> does the
-opposite of C<sv_utf8_encode()>. Note that none of these are to be
-used as general-purpose encoding or decoding interfaces: C<use Encode>
-for that. C<sv_utf8_upgrade()> is affected by the encoding pragma
-but C<sv_utf8_downgrade()> is not (since the encoding pragma is
-designed to be a one-way street).
-
-=item *
-
-C<is_utf8_string(buf, len)> returns true if C<len> bytes of the buffer
-are valid UTF-8.
-
-=item *
-
-C<is_utf8_char(s)> returns true if the pointer points to a valid UTF-8
-character. However, this function should not be used because of
-security concerns. Instead, use C<is_utf8_string()>.
-
-=item *
-
-C<UTF8SKIP(buf)> will return the number of bytes in the UTF-8 encoded
-character in the buffer. C<UNISKIP(chr)> will return the number of bytes
-required to UTF-8-encode the Unicode character code point. C<UTF8SKIP()>
-is useful for example for iterating over the characters of a UTF-8
-encoded buffer; C<UNISKIP()> is useful, for example, in computing
-the size required for a UTF-8 encoded buffer.
-
-=item *
-
-C<utf8_distance(a, b)> will tell the distance in characters between the
-two pointers pointing to the same UTF-8 encoded buffer.
-
-=item *
-
-C<utf8_hop(s, off)> will return a pointer to a UTF-8 encoded buffer
-that is C<off> (positive or negative) Unicode characters displaced
-from the UTF-8 buffer C<s>. Be careful not to overstep the buffer:
-C<utf8_hop()> will merrily run off the end or the beginning of the
-buffer if told to do so.
-
-=item *
-
-C<pv_uni_display(dsv, spv, len, pvlim, flags)> and
-C<sv_uni_display(dsv, ssv, pvlim, flags)> are useful for debugging the
-output of Unicode strings and scalars. By default they are useful
-only for debugging--they display B<all> characters as hexadecimal code
-points--but with the flags C<UNI_DISPLAY_ISPRINT>,
-C<UNI_DISPLAY_BACKSLASH>, and C<UNI_DISPLAY_QQ> you can make the
-output more readable.
-
-=item *
-
-C<foldEQ_utf8(s1, pe1, l1, u1, s2, pe2, l2, u2)> can be used to
-compare two strings case-insensitively in Unicode. For case-sensitive
-comparisons you can just use C<memEQ()> and C<memNE()> as usual, except
-if one string is in utf8 and the other isn't.
-
-=back
-
-For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
-in the Perl source code distribution.
-
-=head2 Hacking Perl to work on earlier Unicode versions (for very serious hackers only)
-
-Perl by default comes with the latest supported Unicode version built in, but
-you can change to use any earlier one.
-
-Download the files in the desired version of Unicode from the Unicode web
-site L<http://www.unicode.org>). These should replace the existing files in
-F<lib/unicore> in the Perl source tree. Follow the instructions in
-F<README.perl> in that directory to change some of their names, and then build
-perl (see L<INSTALL>).
-
-=head1 BUGS
-
-=head2 Interaction with Locales
-
-See L<perllocale/Unicode and UTF-8>
-
-=head2 Problems with characters in the Latin-1 Supplement range
-
-See L</The "Unicode Bug">
-
-=head2 Interaction with Extensions
-
-When Perl exchanges data with an extension, the extension should be
-able to understand the UTF8 flag and act accordingly. If the
-extension doesn't recognize that flag, it's likely that the extension
-will return incorrectly-flagged data.
-
-So if you're working with Unicode data, consult the documentation of
-every module you're using if there are any issues with Unicode data
-exchange. If the documentation does not talk about Unicode at all,
-suspect the worst and probably look at the source to learn how the
-module is implemented. Modules written completely in Perl shouldn't
-cause problems. Modules that directly or indirectly access code written
-in other programming languages are at risk.
-
-For affected functions, the simple strategy to avoid data corruption is
-to always make the encoding of the exchanged data explicit. Choose an
-encoding that you know the extension can handle. Convert arguments passed
-to the extensions to that encoding and convert results back from that
-encoding. Write wrapper functions that do the conversions for you, so
-you can later change the functions when the extension catches up.
-
-To provide an example, let's say the popular Foo::Bar::escape_html
-function doesn't deal with Unicode data yet. The wrapper function
-would convert the argument to raw UTF-8 and convert the result back to
-Perl's internal representation like so:
-
- sub my_escape_html ($) {
- my($what) = shift;
- return unless defined $what;
- Encode::decode_utf8(Foo::Bar::escape_html(
- Encode::encode_utf8($what)));
- }
-
-Sometimes, when the extension does not convert data but just stores
-and retrieves them, you will be able to use the otherwise
-dangerous Encode::_utf8_on() function. Let's say the popular
-C<Foo::Bar> extension, written in C, provides a C<param> method that
-lets you store and retrieve data according to these prototypes:
-
- $self->param($name, $value); # set a scalar
- $value = $self->param($name); # retrieve a scalar
-
-If it does not yet provide support for any encoding, one could write a
-derived class with such a C<param> method:
-
- sub param {
- my($self,$name,$value) = @_;
- utf8::upgrade($name); # make sure it is UTF-8 encoded
- if (defined $value) {
- utf8::upgrade($value); # make sure it is UTF-8 encoded
- return $self->SUPER::param($name,$value);
- } else {
- my $ret = $self->SUPER::param($name);
- Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
- return $ret;
- }
- }
-
-Some extensions provide filters on data entry/exit points, such as
-DB_File::filter_store_key and family. Look out for such filters in
-the documentation of your extensions, they can make the transition to
-Unicode data much easier.
-
-=head2 Speed
-
-Some functions are slower when working on UTF-8 encoded strings than
-on byte encoded strings. All functions that need to hop over
-characters such as length(), substr() or index(), or matching regular
-expressions can work B<much> faster when the underlying data are
-byte-encoded.
-
-In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1
-a caching scheme was introduced which will hopefully make the slowness
-somewhat less spectacular, at least for some operations. In general,
-operations with UTF-8 encoded strings are still slower. As an example,
-the Unicode properties (character classes) like C<\p{Nd}> are known to
-be quite a bit slower (5-20 times) than their simpler counterparts
-like C<\d> (then again, there are hundreds of Unicode characters matching C<Nd>
-compared with the 10 ASCII characters matching C<d>).
-
-=head2 Problems on EBCDIC platforms
-
-There are several known problems with Perl on EBCDIC platforms. If you
-want to use Perl there, send email to perlbug@perl.org.
-
-In earlier versions, when byte and character data were concatenated,
-the new string was sometimes created by
-decoding the byte strings as I<ISO 8859-1 (Latin-1)>, even if the
-old Unicode string used EBCDIC.
-
-If you find any of these, please report them as bugs.
-
-=head2 Porting code from perl-5.6.X
-
-Perl 5.8 has a different Unicode model from 5.6. In 5.6 the programmer
-was required to use the C<utf8> pragma to declare that a given scope
-expected to deal with Unicode data and had to make sure that only
-Unicode data were reaching that scope. If you have code that is
-working with 5.6, you will need some of the following adjustments to
-your code. The examples are written such that the code will continue
-to work under 5.6, so you should be safe to try them out.
-
-=over 3
-
-=item *
-
-A filehandle that should read or write UTF-8
-
- if ($] > 5.007) {
- binmode $fh, ":encoding(utf8)";
- }
-
-=item *
-
-A scalar that is going to be passed to some extension
-
-Be it Compress::Zlib, Apache::Request or any extension that has no
-mention of Unicode in the manpage, you need to make sure that the
-UTF8 flag is stripped off. Note that at the time of this writing
-(October 2002) the mentioned modules are not UTF-8-aware. Please
-check the documentation to verify if this is still true.
-
- if ($] > 5.007) {
- require Encode;
- $val = Encode::encode_utf8($val); # make octets
- }
-
-=item *
-
-A scalar we got back from an extension
-
-If you believe the scalar comes back as UTF-8, you will most likely
-want the UTF8 flag restored:
-
- if ($] > 5.007) {
- require Encode;
- $val = Encode::decode_utf8($val);
- }
-
-=item *
-
-Same thing, if you are really sure it is UTF-8
-
- if ($] > 5.007) {
- require Encode;
- Encode::_utf8_on($val);
- }
-
-=item *
-
-A wrapper for fetchrow_array and fetchrow_hashref
-
-When the database contains only UTF-8, a wrapper function or method is
-a convenient way to replace all your fetchrow_array and
-fetchrow_hashref calls. A wrapper function will also make it easier to
-adapt to future enhancements in your database driver. Note that at the
-time of this writing (October 2002), the DBI has no standardized way
-to deal with UTF-8 data. Please check the documentation to verify if
-that is still true.
-
- sub fetchrow {
- # $what is one of fetchrow_{array,hashref}
- my($self, $sth, $what) = @_;
- if ($] < 5.007) {
- return $sth->$what;
- } else {
- require Encode;
- if (wantarray) {
- my @arr = $sth->$what;
- for (@arr) {
- defined && /[^\000-\177]/ && Encode::_utf8_on($_);
- }
- return @arr;
- } else {
- my $ret = $sth->$what;
- if (ref $ret) {
- for my $k (keys %$ret) {
- defined
- && /[^\000-\177]/
- && Encode::_utf8_on($_) for $ret->{$k};
- }
- return $ret;
- } else {
- defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret;
- return $ret;
- }
- }
- }
- }
-
-
-=item *
-
-A large scalar that you know can only contain ASCII
-
-Scalars that contain only ASCII and are marked as UTF-8 are sometimes
-a drag to your program. If you recognize such a situation, just remove
-the UTF8 flag:
-
- utf8::downgrade($val) if $] > 5.007;
-
-=back
-
-=head1 SEE ALSO
-
-L<perlunitut>, L<perluniintro>, L<perluniprops>, L<Encode>, L<open>, L<utf8>, L<bytes>,
-L<perlretut>, L<perlvar/"${^UNICODE}">
-L<http://www.unicode.org/reports/tr44>).
-
-=cut