summaryrefslogtreecommitdiff
path: root/Master/tlpkg/tlperl/lib/pods/perlunicode.pod
diff options
context:
space:
mode:
Diffstat (limited to 'Master/tlpkg/tlperl/lib/pods/perlunicode.pod')
-rw-r--r--Master/tlpkg/tlperl/lib/pods/perlunicode.pod605
1 files changed, 253 insertions, 352 deletions
diff --git a/Master/tlpkg/tlperl/lib/pods/perlunicode.pod b/Master/tlpkg/tlperl/lib/pods/perlunicode.pod
index d77c40cbbde..77daca34a7d 100644
--- a/Master/tlpkg/tlperl/lib/pods/perlunicode.pod
+++ b/Master/tlpkg/tlperl/lib/pods/perlunicode.pod
@@ -90,10 +90,12 @@ semantics. For operations where this determination cannot be made
without additional information from the user, Perl decides in favor of
compatibility and chooses to use byte semantics.
-When C<use locale> is in effect (which overrides
-C<use feature 'unicode_strings'> in the same scope), Perl uses the
-semantics associated
-with the current locale. Otherwise, Perl uses the platform's native
+When C<use locale> (but not C<use locale ':not_characters'>) is in
+effect, Perl uses the semantics associated with the current locale.
+(C<use locale> overrides C<use feature 'unicode_strings'> in the same scope;
+while C<use locale ':not_characters'> effectively also selects
+C<use feature 'unicode_strings'> in its scope; see L<perllocale>.)
+Otherwise, Perl uses the platform's native
byte semantics for characters whose code points are less than 256, and
Unicode semantics for those greater than 255. On EBCDIC platforms, this
is almost seamless, as the EBCDIC code pages that Perl handles are
@@ -155,15 +157,17 @@ Alternatively, you can use the C<\x{...}> notation for characters 0x100 and
above. For characters below 0x100 you may get byte semantics instead of
character semantics; see L</The "Unicode Bug">. On EBCDIC machines there is
the additional problem that the value for such characters gives the EBCDIC
-character rather than the Unicode one.
+character rather than the Unicode one, thus it is more portable to use
+C<\N{U+...}> instead.
-Additionally, if you
+Additionally, you can use the C<\N{...}> notation and put the official
+Unicode character name within the braces, such as
+C<\N{WHITE SMILING FACE}>. This automatically loads the L<charnames>
+module with the C<:full> and C<:short> options. If you prefer different
+options for this module, you can instead, before the C<\N{...}>,
+explicitly load it with your desired options; for example,
- use charnames ':full';
-
-you can use the C<\N{...}> notation and put the official Unicode
-character name within the braces, such as C<\N{WHITE SMILING FACE}>.
-See L<charnames>.
+ use charnames ':loose';
=item *
@@ -260,11 +264,13 @@ complement B<and> the full character-wide bit complement.
=item *
-You can define your own mappings to be used in C<lc()>,
-C<lcfirst()>, C<uc()>, and C<ucfirst()> (or their double-quoted string inlined
-versions such as C<\U>). See
-L<User-Defined Case-Mappings|/"User-Defined Case Mappings (for serious hackers only)">
-for more details.
+There is a CPAN module, L<Unicode::Casing>, which allows you to define
+your own mappings to be used in C<lc()>, C<lcfirst()>, C<uc()>,
+C<ucfirst()>, and C<fc> (or their double-quoted string inlined
+versions such as C<\U>).
+(Prior to Perl 5.16, this functionality was partially provided
+in the Perl core, but suffered from a number of insurmountable
+drawbacks, so the CPAN module was written instead.)
=back
@@ -301,7 +307,8 @@ This formality is needed when properties are not binary; that is, if they can
take on more values than just True and False. For example, the Bidi_Class (see
L</"Bidirectional Character Types"> below), can take on several different
values, such as Left, Right, Whitespace, and others. To match these, one needs
-to specify the property name (Bidi_Class), AND the value being matched against
+to specify both the property name (Bidi_Class), AND the value being
+matched against
(Left, Right, etc.). This is done, as in the examples above, by having the
two components separated by an equal sign (or interchangeably, a colon), like
C<\p{Bidi_Class: Left}>.
@@ -364,6 +371,14 @@ of which under C</i> matching match C<PosixAlpha>.
numerals, come in both upper and lower case so they are C<Cased>, but aren't considered
letters, so they aren't C<Cased_Letter>s.)
+The result is undefined if you try to match a non-Unicode code point
+(that is, one above 0x10FFFF) against a Unicode property. Currently, a
+warning is raised, and the match will fail. In some cases, this is
+counterintuitive, as both these fail:
+
+ chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails.
+ chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Fails!
+
=head3 B<General_Category>
Every Unicode character is assigned a general category, which is the "most
@@ -469,11 +484,63 @@ The world's languages are written in many different scripts. This sentence
written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in
Hiragana or Katakana. There are many more.
-The Unicode Script property gives what script a given character is in,
-and the property can be specified with the compound form like
-C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>). Perl furnishes shortcuts for all
-script names. You can omit everything up through the equals (or colon), and
-simply write C<\p{Latin}> or C<\P{Cyrillic}>.
+The Unicode Script and Script_Extensions properties give what script a
+given character is in. Either property can be specified with the
+compound form like
+C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>), or
+C<\p{Script_Extensions=Javanese}> (short: C<\p{scx=java}>).
+In addition, Perl furnishes shortcuts for all
+C<Script> property names. You can omit everything up through the equals
+(or colon), and simply write C<\p{Latin}> or C<\P{Cyrillic}>.
+(This is not true for C<Script_Extensions>, which is required to be
+written in the compound form.)
+
+The difference between these two properties involves characters that are
+used in multiple scripts. For example the digits '0' through '9' are
+used in many parts of the world. These are placed in a script named
+C<Common>. Other characters are used in just a few scripts. For
+example, the "KATAKANA-HIRAGANA DOUBLE HYPHEN" is used in both Japanese
+scripts, Katakana and Hiragana, but nowhere else. The C<Script>
+property places all characters that are used in multiple scripts in the
+C<Common> script, while the C<Script_Extensions> property places those
+that are used in only a few scripts into each of those scripts; while
+still using C<Common> for those used in many scripts. Thus both these
+match:
+
+ "0" =~ /\p{sc=Common}/ # Matches
+ "0" =~ /\p{scx=Common}/ # Matches
+
+and only the first of these match:
+
+ "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Common} # Matches
+ "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Common} # No match
+
+And only the last two of these match:
+
+ "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Hiragana} # No match
+ "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Katakana} # No match
+ "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Hiragana} # Matches
+ "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Katakana} # Matches
+
+C<Script_Extensions> is thus an improved C<Script>, in which there are
+fewer characters in the C<Common> script, and correspondingly more in
+other scripts. It is new in Unicode version 6.0, and its data are likely
+to change significantly in later releases, as things get sorted out.
+
+(Actually, besides C<Common>, the C<Inherited> script, contains
+characters that are used in multiple scripts. These are modifier
+characters which modify other characters, and inherit the script value
+of the controlling character. Some of these are used in many scripts,
+and so go into C<Inherited> in both C<Script> and C<Script_Extensions>.
+Others are used in just a few scripts, so are in C<Inherited> in
+C<Script>, but not in C<Script_Extensions>.)
+
+It is worth stressing that there are several different sets of digits in
+Unicode that are equivalent to 0-9 and are matchable by C<\d> in a
+regular expression. If they are used in a single language only, they
+are in that language's C<Script> and C<Script_Extension>. If they are
+used in more than one script, they will be in C<sc=Common>, but only
+if they are used in many scripts should they be in C<scx=Common>.
A complete list of scripts and their shortcuts is in L<perluniprops>.
@@ -496,20 +563,14 @@ other words, the ASCII characters. The "Latin" script contains some letters
from this as well as several other blocks, like "Latin-1 Supplement",
"Latin Extended-A", etc., but it does not contain all the characters from
those blocks. It does not, for example, contain the digits 0-9, because
-those digits are shared across many scripts. The digits 0-9 and similar groups,
-like punctuation, are in the script called C<Common>. There is also a
-script called C<Inherited> for characters that modify other characters,
-and inherit the script value of the controlling character. (Note that
-there are several different sets of digits in Unicode that are
-equivalent to 0-9 and are matchable by C<\d> in a regular expression.
-If they are used in a single language only, they are in that language's
-script. Only sets are used across several languages are in the
-C<Common> script.)
+those digits are shared across many scripts, and hence are in the
+C<Common> script.
For more about scripts versus blocks, see UAX#24 "Unicode Script Property":
L<http://www.unicode.org/reports/tr24>
-The Script property is likely to be the one you want to use when processing
+The C<Script> or C<Script_Extensions> properties are likely to be the
+ones you want to use when processing
natural language; the Block property may occasionally be useful in working
with the nuts and bolts of Unicode.
@@ -557,8 +618,9 @@ Unicode ones, but some are genuine extensions, including several that are in
the compound form. And quite a few of these are actually recommended by Unicode
(in L<http://www.unicode.org/reports/tr18>).
-This section gives some details on all extensions that aren't synonyms for
-compound-form Unicode properties (for those, you'll have to refer to the
+This section gives some details on all extensions that aren't just
+synonyms for compound-form Unicode properties
+(for those properties, you'll have to refer to the
L<Unicode Standard|http://www.unicode.org/reports/tr44>.
=over
@@ -719,6 +781,13 @@ This is the same as C<\s>, including beyond ASCII.
Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab
which both the Posix standard and Unicode consider white space.)
+=item B<C<\p{Title}>> and B<C<\p{Titlecase}>>
+
+Under case-sensitive matching, these both match the same code points as
+C<\p{General Category=Titlecase_Letter}> (C<\p{gc=lt}>). The difference
+is that under C</i> caseless matching, these match the same as
+C<\p{Cased}>, whereas C<\p{gc=lt}> matches C<\p{Cased_Letter>).
+
=item B<C<\p{VertSpace}>>
This is the same as C<\v>: A character that changes the spacing vertically.
@@ -781,28 +850,32 @@ tabular characters) denoting a range of Unicode code points to include.
=item *
Something to include, prefixed by "+": a built-in character
-property (prefixed by "utf8::") or a user-defined character property,
+property (prefixed by "utf8::") or a fully qualified (including package
+name) user-defined character property,
to represent all the characters in that property; two hexadecimal code
points for a range; or a single hexadecimal code point.
=item *
Something to exclude, prefixed by "-": an existing character
-property (prefixed by "utf8::") or a user-defined character property,
+property (prefixed by "utf8::") or a fully qualified (including package
+name) user-defined character property,
to represent all the characters in that property; two hexadecimal code
points for a range; or a single hexadecimal code point.
=item *
Something to negate, prefixed "!": an existing character
-property (prefixed by "utf8::") or a user-defined character property,
+property (prefixed by "utf8::") or a fully qualified (including package
+name) user-defined character property,
to represent all the characters in that property; two hexadecimal code
points for a range; or a single hexadecimal code point.
=item *
Something to intersect with, prefixed by "&": an existing character
-property (prefixed by "utf8::") or a user-defined character property,
+property (prefixed by "utf8::") or a fully qualified (including package
+name) user-defined character property,
for all the characters except the characters in the property; two
hexadecimal code points for a range; or a single hexadecimal code point.
@@ -852,204 +925,38 @@ The negation is useful for defining (surprise!) negated classes.
END
}
-Intersection is useful for getting the common characters matched by
-two (or more) classes.
+This will match all non-Unicode code points, since every one of them is
+not in Kana. You can use intersection to exclude these, if desired, as
+this modified example shows:
- sub InFooAndBar {
+ sub InNotKana {
return <<'END';
- +main::Foo
- &main::Bar
- END
- }
-
-It's important to remember not to use "&" for the first set; that
-would be intersecting with nothing, resulting in an empty set.
-
-=head2 User-Defined Case Mappings (for serious hackers only)
-
-B<This featured is deprecated and is scheduled to be removed in Perl
-5.16.>
-The CPAN module L<Unicode::Casing> provides better functionality
-without the drawbacks described below.
-
-You can define your own mappings to be used in C<lc()>,
-C<lcfirst()>, C<uc()>, and C<ucfirst()> (or their string-inlined versions,
-C<\L>, C<\l>, C<\U>, and C<\u>). The mappings are currently only valid
-on strings encoded in UTF-8, but see below for a partial workaround for
-this restriction.
-
-The principle is similar to that of user-defined character
-properties: define subroutines that do the mappings.
-C<ToLower> is used for C<lc()>, C<\L>, C<lcfirst()>, and C<\l>; C<ToTitle> for
-C<ucfirst()> and C<\u>; and C<ToUpper> for C<uc()> and C<\U>.
-
-C<ToUpper()> should look something like this:
-
- sub ToUpper {
- return <<END;
- 0061\t007A\t0041
- 0101\t\t0100
+ !utf8::InHiragana
+ -utf8::InKatakana
+ +utf8::IsCn
+ &utf8::Any
END
}
-This sample C<ToUpper()> has the effect of mapping "a-z" to "A-Z", 0x101
-to 0x100, and all other characters map to themselves. The first
-returned line means to map the code point at 0x61 ("a") to 0x41 ("A"),
-the code point at 0x62 ("b") to 0x42 ("B"), ..., 0x7A ("z") to 0x5A
-("Z"). The second line maps just the code point 0x101 to 0x100. Since
-there are no other mappings defined, all other code points map to
-themselves.
-
-This mechanism is not well behaved as far as affecting other packages
-and scopes. All non-threaded programs have exactly one uppercasing
-behavior, one lowercasing behavior, and one titlecasing behavior in
-effect for utf8-encoded strings for the duration of the program. Each
-of these behaviors is irrevocably determined the first time the
-corresponding function is called to change a utf8-encoded string's case.
-If a corresponding C<To-> function has been defined in the package that
-makes that first call, the mapping defined by that function will be the
-mapping used for the duration of the program's execution across all
-packages and scopes. If no corresponding C<To-> function has been
-defined in that package, the standard official mapping will be used for
-all packages and scopes, and any corresponding C<To-> function anywhere
-will be ignored. Threaded programs have similar behavior. If the
-program's casing behavior has been decided at the time of a thread's
-creation, the thread will inherit that behavior. But, if the behavior
-hasn't been decided, the thread gets to decide for itself, and its
-decision does not affect other threads nor its creator.
-
-As shown by the example above, you have to furnish a complete mapping;
-you can't just override a couple of characters and leave the rest
-unchanged. You can find all the official mappings in the directory
-C<$Config{privlib}>F</unicore/To/>. The mapping data is returned as the
-here-document. The C<utf8::ToSpecI<Foo>> hashes in those files are special
-exception mappings derived from
-C<$Config{privlib}>F</unicore/SpecialCasing.txt>. (The "Digit" and
-"Fold" mappings that one can see in the directory are not directly
-user-accessible, one can use either the L<Unicode::UCD> module, or just match
-case-insensitively, which is what uses the "Fold" mapping. Neither are user
-overridable.)
-
-If you have many mappings to change, you can take the official mapping data,
-change by hand the affected code points, and place the whole thing into your
-subroutine. But this will only be valid on Perls that use the same Unicode
-version. Another option would be to have your subroutine read the official
-mapping files and overwrite the affected code points.
-
-If you have only a few mappings to change, starting in 5.14 you can use the
-following trick, here illustrated for Turkish.
-
- use Config;
- use charnames ":full";
-
- sub ToUpper {
- my $official = do "$Config{privlib}/unicore/To/Upper.pl";
- $utf8::ToSpecUpper{'i'} =
- "\N{LATIN CAPITAL LETTER I WITH DOT ABOVE}";
- return $official;
- }
+C<&utf8::Any> must be the last line in the definition.
-This takes the official mappings and overrides just one, for "LATIN SMALL
-LETTER I". The keys to the hash must be the bytes that form the UTF-8
-(on EBCDIC platforms, UTF-EBCDIC) of the character, as illustrated by
-the inverse function.
+Intersection is used generally for getting the common characters matched
+by two (or more) classes. It's important to remember not to use "&" for
+the first set; that would be intersecting with nothing, resulting in an
+empty set.
- sub ToLower {
- my $official = do $lower;
- $utf8::ToSpecLower{"\xc4\xb0"} = "i";
- return $official;
- }
+(Note that official Unicode properties differ from these in that they
+automatically exclude non-Unicode code points and a warning is raised if
+a match is attempted on one of those.)
-This example is for an ASCII platform, and C<\xc4\xb0> is the string of
-bytes that together form the UTF-8 that represents C<\N{LATIN CAPITAL
-LETTER I WITH DOT ABOVE}>, C<U+0130>. You can avoid having to figure out
-these bytes, and at the same time make it work on all platforms by
-instead writing:
-
- sub ToLower {
- my $official = do $lower;
- my $sequence = "\N{LATIN CAPITAL LETTER I WITH DOT ABOVE}";
- utf8::encode($sequence);
- $utf8::ToSpecLower{$sequence} = "i";
- return $official;
- }
+=head2 User-Defined Case Mappings (for serious hackers only)
-This works because C<utf8::encode()> takes the single character and
-converts it to the sequence of bytes that constitute it. Note that we took
-advantage of the fact that C<"i"> is the same in UTF-8 or UTF_EBCIDIC as not;
-otherwise we would have had to write
-
- $utf8::ToSpecLower{$sequence} = "\N{LATIN SMALL LETTER I}";
-
-in the ToLower example, and in the ToUpper example, use
-
- my $sequence = "\N{LATIN SMALL LETTER I}";
- utf8::encode($sequence);
-
-A big caveat to the above trick and to this whole mechanism in general,
-is that they work only on strings encoded in UTF-8. You can partially
-get around this by using C<use subs>. (But better to just convert to
-use L<Unicode::Casing>.) For example:
-(The trick illustrated here does work in earlier releases, but only if all the
-characters you want to override have ordinal values of 256 or higher, or
-if you use the other tricks given just below.)
-
-The mappings are in effect only for the package they are defined in, and only
-on scalars that have been marked as having Unicode characters, for example by
-using C<utf8::upgrade()>. Although probably not advisable, you can
-cause the mappings to be used globally by importing into C<CORE::GLOBAL>
-(see L<CORE>).
-
-You can partially get around the restriction that the source strings
-must be in utf8 by using C<use subs> (or by importing into C<CORE::GLOBAL>) by:
-
- use subs qw(uc ucfirst lc lcfirst);
-
- sub uc($) {
- my $string = shift;
- utf8::upgrade($string);
- return CORE::uc($string);
- }
-
- sub lc($) {
- my $string = shift;
- utf8::upgrade($string);
-
- # Unless an I is before a dot_above, it turns into a dotless i.
- # (The character class with the combining classes matches non-above
- # marks following the I. Any number of these may be between the 'I' and
- # the dot_above, and the dot_above will still apply to the 'I'.
- use charnames ":full";
- $string =~
- s/I
- (?! [^\p{ccc=0}\p{ccc=Above}]* \N{COMBINING DOT ABOVE} )
- /\N{LATIN SMALL LETTER DOTLESS I}/gx;
-
- # But when the I is followed by a dot_above, remove the
- # dot_above so the end result will be i.
- $string =~ s/I
- ([^\p{ccc=0}\p{ccc=Above}]* )
- \N{COMBINING DOT ABOVE}
- /i$1/gx;
- return CORE::lc($string);
- }
-
-These examples (also for Turkish) make sure the input is in UTF-8, and then
-call the corresponding official function, which will use the C<ToUpper()> and
-C<ToLower()> functions you have defined.
-(For Turkish, there are other required functions: C<ucfirst>, C<lcfirst>,
-and C<ToTitle>. These are very similar to the ones given above.)
-
-The reason this is only a partial fix is that it doesn't affect the C<\l>,
-C<\L>, C<\u>, and C<\U> case-change operations in regular expressions,
-which still require the source to be encoded in utf8 (see L</The "Unicode
-Bug">). (Again, use L<Unicode::Casing> instead.)
-
-The C<lc()> example shows how you can add context-dependent casing. Note
-that context-dependent casing suffers from the problem that the string
-passed to the casing function may not have sufficient context to make
-the proper choice. Also, it will not be called for C<\l>, C<\L>, C<\u>,
-and C<\U>.
+B<This feature has been removed as of Perl 5.16.>
+The CPAN module L<Unicode::Casing> provides better functionality without
+the drawbacks that this feature had. If you are using a Perl earlier
+than 5.16, this feature was most fully documented in the 5.14 version of
+this pod:
+L<http://perldoc.perl.org/5.14.0/perlunicode.html#User-Defined-Case-Mappings-%28for-serious-hackers-only%29>
=head2 Character Encodings for Input and Output
@@ -1068,41 +975,41 @@ and the section numbers refer to the Unicode Technical Standard #18,
Level 1 - Basic Unicode Support
- RL1.1 Hex Notation - done [1]
- RL1.2 Properties - done [2][3]
- RL1.2a Compatibility Properties - done [4]
- RL1.3 Subtraction and Intersection - MISSING [5]
- RL1.4 Simple Word Boundaries - done [6]
- RL1.5 Simple Loose Matches - done [7]
- RL1.6 Line Boundaries - MISSING [8][9]
- RL1.7 Supplementary Code Points - done [10]
-
- [1] \x{...}
- [2] \p{...} \P{...}
- [3] supports not only minimal list, but all Unicode character
- properties (see L</Unicode Character Properties>)
- [4] \d \D \s \S \w \W \X [:prop:] [:^prop:]
- [5] can use regular expression look-ahead [a] or
- user-defined character properties [b] to emulate set
- operations
- [6] \b \B
- [7] note that Perl does Full case-folding in matching (but with
- bugs), not Simple: for example U+1F88 is equivalent to
- U+1F00 U+03B9, not with 1F80. This difference matters
- mainly for certain Greek capital letters with certain
- modifiers: the Full case-folding decomposes the letter,
- while the Simple case-folding would map it to a single
- character.
- [8] should do ^ and $ also on U+000B (\v in C), FF (\f), CR
- (\r), CRLF (\r\n), NEL (U+0085), LS (U+2028), and PS
- (U+2029); should also affect <>, $., and script line
- numbers; should not split lines within CRLF [c] (i.e. there
- is no empty line between \r and \n)
- [9] Linebreaking conformant with UAX#14 "Unicode Line Breaking
- Algorithm" is available through the Unicode::LineBreaking
- module.
- [10] UTF-8/UTF-EBDDIC used in Perl allows not only U+10000 to
- U+10FFFF but also beyond U+10FFFF
+ RL1.1 Hex Notation - done [1]
+ RL1.2 Properties - done [2][3]
+ RL1.2a Compatibility Properties - done [4]
+ RL1.3 Subtraction and Intersection - MISSING [5]
+ RL1.4 Simple Word Boundaries - done [6]
+ RL1.5 Simple Loose Matches - done [7]
+ RL1.6 Line Boundaries - MISSING [8][9]
+ RL1.7 Supplementary Code Points - done [10]
+
+ [1] \x{...}
+ [2] \p{...} \P{...}
+ [3] supports not only minimal list, but all Unicode character
+ properties (see Unicode Character Properties above)
+ [4] \d \D \s \S \w \W \X [:prop:] [:^prop:]
+ [5] can use regular expression look-ahead [a] or
+ user-defined character properties [b] to emulate set
+ operations
+ [6] \b \B
+ [7] note that Perl does Full case-folding in matching (but with
+ bugs), not Simple: for example U+1F88 is equivalent to
+ U+1F00 U+03B9, instead of just U+1F80. This difference
+ matters mainly for certain Greek capital letters with certain
+ modifiers: the Full case-folding decomposes the letter,
+ while the Simple case-folding would map it to a single
+ character.
+ [8] should do ^ and $ also on U+000B (\v in C), FF (\f), CR
+ (\r), CRLF (\r\n), NEL (U+0085), LS (U+2028), and PS
+ (U+2029); should also affect <>, $., and script line
+ numbers; should not split lines within CRLF [c] (i.e. there
+ is no empty line between \r and \n)
+ [9] Linebreaking conformant with UAX#14 "Unicode Line Breaking
+ Algorithm" is available through the Unicode::LineBreaking
+ module.
+ [10] UTF-8/UTF-EBDDIC used in Perl allows not only U+10000 to
+ U+10FFFF but also beyond U+10FFFF
[a] You can mimic class subtraction using lookahead.
For example, what UTS#18 might write as
@@ -1120,7 +1027,7 @@ But in this particular example, you probably really want
which will match assigned characters known to be part of the Greek script.
-Also see the Unicode::Regex::Set module, it does implement the full
+Also see the L<Unicode::Regex::Set> module; it does implement the full
UTS#18 grouping, intersection, union, and removal (subtraction) syntax.
[b] '+' for union, '-' for removal (set-difference), '&' for intersection
@@ -1132,46 +1039,43 @@ UTS#18 grouping, intersection, union, and removal (subtraction) syntax.
Level 2 - Extended Unicode Support
- RL2.1 Canonical Equivalents - MISSING [10][11]
- RL2.2 Default Grapheme Clusters - MISSING [12]
- RL2.3 Default Word Boundaries - MISSING [14]
- RL2.4 Default Loose Matches - MISSING [15]
- RL2.5 Name Properties - MISSING [16]
- RL2.6 Wildcard Properties - MISSING
+ RL2.1 Canonical Equivalents - MISSING [10][11]
+ RL2.2 Default Grapheme Clusters - MISSING [12]
+ RL2.3 Default Word Boundaries - MISSING [14]
+ RL2.4 Default Loose Matches - MISSING [15]
+ RL2.5 Name Properties - DONE
+ RL2.6 Wildcard Properties - MISSING
- [10] see UAX#15 "Unicode Normalization Forms"
- [11] have Unicode::Normalize but not integrated to regexes
- [12] have \X but we don't have a "Grapheme Cluster Mode"
- [14] see UAX#29, Word Boundaries
- [15] see UAX#21 "Case Mappings"
- [16] missing loose match [e]
-
-[e] C<\N{...}> allows namespaces (see L<charnames>).
+ [10] see UAX#15 "Unicode Normalization Forms"
+ [11] have Unicode::Normalize but not integrated to regexes
+ [12] have \X but we don't have a "Grapheme Cluster Mode"
+ [14] see UAX#29, Word Boundaries
+ [15] This is covered in Chapter 3.13 (in Unicode 6.0)
=item *
Level 3 - Tailored Support
- RL3.1 Tailored Punctuation - MISSING
- RL3.2 Tailored Grapheme Clusters - MISSING [17][18]
- RL3.3 Tailored Word Boundaries - MISSING
- RL3.4 Tailored Loose Matches - MISSING
- RL3.5 Tailored Ranges - MISSING
- RL3.6 Context Matching - MISSING [19]
- RL3.7 Incremental Matches - MISSING
+ RL3.1 Tailored Punctuation - MISSING
+ RL3.2 Tailored Grapheme Clusters - MISSING [17][18]
+ RL3.3 Tailored Word Boundaries - MISSING
+ RL3.4 Tailored Loose Matches - MISSING
+ RL3.5 Tailored Ranges - MISSING
+ RL3.6 Context Matching - MISSING [19]
+ RL3.7 Incremental Matches - MISSING
( RL3.8 Unicode Set Sharing )
- RL3.9 Possible Match Sets - MISSING
- RL3.10 Folded Matching - MISSING [20]
- RL3.11 Submatchers - MISSING
-
- [17] see UAX#10 "Unicode Collation Algorithms"
- [18] have Unicode::Collate but not integrated to regexes
- [19] have (?<=x) and (?=x), but look-aheads or look-behinds
- should see outside of the target substring
- [20] need insensitive matching for linguistic features other
- than case; for example, hiragana to katakana, wide and
- narrow, simplified Han to traditional Han (see UTR#30
- "Character Foldings")
+ RL3.9 Possible Match Sets - MISSING
+ RL3.10 Folded Matching - MISSING [20]
+ RL3.11 Submatchers - MISSING
+
+ [17] see UAX#10 "Unicode Collation Algorithms"
+ [18] have Unicode::Collate but not integrated to regexes
+ [19] have (?<=x) and (?=x), but look-aheads or look-behinds
+ should see outside of the target substring
+ [20] need insensitive matching for linguistic features other
+ than case; for example, hiragana to katakana, wide and
+ narrow, simplified Han to traditional Han (see UTR#30
+ "Character Foldings")
=back
@@ -1192,14 +1096,14 @@ encoding. For ASCII (and we really do mean 7-bit ASCII, not another
The following table is from Unicode 3.2.
- Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
+ Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
U+0000..U+007F 00..7F
U+0080..U+07FF * C2..DF 80..BF
U+0800..U+0FFF E0 * A0..BF 80..BF
U+1000..U+CFFF E1..EC 80..BF 80..BF
U+D000..U+D7FF ED 80..9F 80..BF
- U+D800..U+DFFF +++++++ utf16 surrogates, not legal utf8 +++++++
+ U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++
U+E000..U+FFFF EE..EF 80..BF 80..BF
U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF
U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
@@ -1213,12 +1117,12 @@ explicitly forbidden, and the shortest possible encoding should always be used
Another way to look at it is via bits:
- Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
+ Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
- 0aaaaaaa 0aaaaaaa
- 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
- ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
- 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
+ 0aaaaaaa 0aaaaaaa
+ 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
+ ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
+ 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
As you can see, the continuation bytes all begin with "10", and the
leading bits of the start byte tell how many bytes there are in the
@@ -1467,55 +1371,69 @@ readdir, readlink
=head2 The "Unicode Bug"
-The term, the "Unicode bug" has been applied to an inconsistency
+The term, "Unicode bug" has been applied to an inconsistency
on ASCII platforms with the
Unicode code points in the Latin-1 Supplement block, that
is, between 128 and 255. Without a locale specified, unlike all other
characters or code points, these characters have very different semantics in
byte semantics versus character semantics, unless
-C<use feature 'unicode_strings'> is specified.
-(The lesson here is to specify C<unicode_strings> to avoid the
-headaches.)
+C<use feature 'unicode_strings'> is specified, directly or indirectly.
+(It is indirectly specified by a C<use v5.12> or higher.)
-In character semantics they are interpreted as Unicode code points, which means
+In character semantics these upper-Latin1 characters are interpreted as
+Unicode code points, which means
they have the same semantics as Latin-1 (ISO-8859-1).
-In byte semantics, they are considered to be unassigned characters, meaning
-that the only semantics they have is their ordinal numbers, and that they are
+In byte semantics (without C<unicode_strings>), they are considered to
+be unassigned characters, meaning that the only semantics they have is
+their ordinal numbers, and that they are
not members of various character classes. None are considered to match C<\w>
for example, but all match C<\W>.
-The behavior is known to have effects on these areas:
+Perl 5.12.0 added C<unicode_strings> to force character semantics on
+these code points in some circumstances, which fixed portions of the
+bug; Perl 5.14.0 fixed almost all of it; and Perl 5.16.0 fixed the
+remainder (so far as we know, anyway). The lesson here is to enable
+C<unicode_strings> to avoid the headaches described below.
+
+The old, problematic behavior affects these areas:
=over 4
=item *
Changing the case of a scalar, that is, using C<uc()>, C<ucfirst()>, C<lc()>,
-and C<lcfirst()>, or C<\L>, C<\U>, C<\u> and C<\l> in regular expression
-substitutions.
+and C<lcfirst()>, or C<\L>, C<\U>, C<\u> and C<\l> in double-quotish
+contexts, such as regular expression substitutions.
+Under C<unicode_strings> starting in Perl 5.12.0, character semantics are
+generally used. See L<perlfunc/lc> for details on how this works
+in combination with various other pragmas.
=item *
-Using caseless (C</i>) regular expression matching
+Using caseless (C</i>) regular expression matching.
+Starting in Perl 5.14.0, regular expressions compiled within
+the scope of C<unicode_strings> use character semantics
+even when executed or compiled into larger
+regular expressions outside the scope.
=item *
Matching any of several properties in regular expressions, namely C<\b>,
C<\B>, C<\s>, C<\S>, C<\w>, C<\W>, and all the Posix character classes
I<except> C<[[:ascii:]]>.
+Starting in Perl 5.14.0, regular expressions compiled within
+the scope of C<unicode_strings> use character semantics
+even when executed or compiled into larger
+regular expressions outside the scope.
=item *
-In C<quotemeta> or its inline equivalent C<\Q>, no characters
-code points above 127 are quoted in UTF-8 encoded strings, but in
-byte encoded strings, code points between 128-255 are always quoted.
-
-=item *
-
-User-defined case change mappings. You can create a C<ToUpper()> function, for
-example, which overrides Perl's built-in case mappings. The scalar must be
-encoded in utf8 for your function to actually be invoked.
+In C<quotemeta> or its inline equivalent C<\Q>, no code points above 127
+are quoted in UTF-8 encoded strings, but in byte encoded strings, code
+points between 128-255 are always quoted.
+Starting in Perl 5.16.0, consistent quoting rules are used within the
+scope of C<unicode_strings>, as described in L<perlfunc/quotemeta>.
=back
@@ -1544,21 +1462,9 @@ ASCII range (except in a locale), along with Perl's desire to add Unicode
support seamlessly. The result wasn't seamless: these characters were
orphaned.
-Starting in Perl 5.14, C<use feature 'unicode_strings'> can be used to
-cause Perl to use Unicode semantics on all string operations within the
-scope of the feature subpragma. Regular expressions compiled in its
-scope retain that behavior even when executed or compiled into larger
-regular expressions outside the scope. (The pragma does not, however,
-affect the C<quotemeta> behavior. Nor does it affect the deprecated
-user-defined case changing operations--these still require a UTF-8
-encoded string to operate.)
-
-In Perl 5.12, the subpragma affected casing changes, but not regular
-expressions. See L<perlfunc/lc> for details on how this pragma works in
-combination with various others for casing.
-
-For earlier Perls, or when a string is passed to a function outside the
-subpragma's scope, a workaround is to always call C<utf8::upgrade($string)>,
+For Perls earlier than those described above, or when a string is passed
+to a function outside the subpragma's scope, a workaround is to always
+call C<utf8::upgrade($string)>,
or to use the standard module L<Encode>. Also, a scalar that has any characters
whose ordinal is above 0x100, or which were specified using either of the
C<\N{...}> notations, will automatically have character semantics.
@@ -1608,7 +1514,8 @@ pointing after the UTF-8 bytes. It works appropriately on EBCDIC machines.
=item *
-C<utf8_to_uvchr(buf, lenp)> reads UTF-8 encoded bytes from a buffer and
+C<utf8_to_uvchr_buf(buf, bufend, lenp)> reads UTF-8 encoded bytes from a
+buffer and
returns the Unicode character code point and, optionally, the length of
the UTF-8 byte sequence. It works appropriately on EBCDIC machines.
@@ -1632,13 +1539,14 @@ designed to be a one-way street).
=item *
-C<is_utf8_char(s)> returns true if the pointer points to a valid UTF-8
-character.
+C<is_utf8_string(buf, len)> returns true if C<len> bytes of the buffer
+are valid UTF-8.
=item *
-C<is_utf8_string(buf, len)> returns true if C<len> bytes of the buffer
-are valid UTF-8.
+C<is_utf8_char(s)> returns true if the pointer points to a valid UTF-8
+character. However, this function should not be used because of
+security concerns. Instead, use C<is_utf8_string()>.
=item *
@@ -1693,14 +1601,7 @@ Download the files in the desired version of Unicode from the Unicode web
site L<http://www.unicode.org>). These should replace the existing files in
F<lib/unicore> in the Perl source tree. Follow the instructions in
F<README.perl> in that directory to change some of their names, and then build
-perl (see F<INSTALL>).
-
-It is even possible to copy the built files to a different directory, and then
-change F<utf8_heavy.pl> in the directory C<$Config{privlib}> to point to the
-new directory, or maybe make a copy of that directory before making the change,
-and using C<@INC> or the C<-I> run-time flag to switch between versions at will
-(but because of caching, not in the middle of a process), but all this is
-beyond the scope of these instructions.
+perl (see L<INSTALL>).
=head1 BUGS
@@ -1815,7 +1716,7 @@ working with 5.6, you will need some of the following adjustments to
your code. The examples are written such that the code will continue
to work under 5.6, so you should be safe to try them out.
-=over 4
+=over 3
=item *