diff options
Diffstat (limited to 'Master/tlpkg/tlperl/lib/Locale/Maketext/TPJ13.pod')
-rw-r--r-- | Master/tlpkg/tlperl/lib/Locale/Maketext/TPJ13.pod | 775 |
1 files changed, 0 insertions, 775 deletions
diff --git a/Master/tlpkg/tlperl/lib/Locale/Maketext/TPJ13.pod b/Master/tlpkg/tlperl/lib/Locale/Maketext/TPJ13.pod deleted file mode 100644 index b9586b2cbf4..00000000000 --- a/Master/tlpkg/tlperl/lib/Locale/Maketext/TPJ13.pod +++ /dev/null @@ -1,775 +0,0 @@ -# This document contains text in Perl "POD" format. -# Use a POD viewer like perldoc or perlman to render it. - -=head1 NAME - -Locale::Maketext::TPJ13 -- article about software localization - -=head1 SYNOPSIS - - # This an article, not a module. - -=head1 DESCRIPTION - -The following article by Sean M. Burke and Jordan Lachler -first appeared in I<The Perl Journal> #13 -and is copyright 1999 The Perl Journal. It appears -courtesy of Jon Orwant and The Perl Journal. This document may be -distributed under the same terms as Perl itself. - -=head1 Localization and Perl: gettext breaks, Maketext fixes - -by Sean M. Burke and Jordan Lachler - -This article points out cases where gettext (a common system for -localizing software interfaces -- i.e., making them work in the user's -language of choice) fails because of basic differences between human -languages. This article then describes Maketext, a new system capable -of correctly treating these differences. - -=head2 A Localization Horror Story: It Could Happen To You - -=over - -"There are a number of languages spoken by human beings in this -world." - --- Harald Tveit Alvestrand, in RFC 1766, "Tags for the -Identification of Languages" - -=back - -Imagine that your task for the day is to localize a piece of software --- and luckily for you, the only output the program emits is two -messages, like this: - - I scanned 12 directories. - - Your query matched 10 files in 4 directories. - -So how hard could that be? You look at the code that -produces the first item, and it reads: - - printf("I scanned %g directories.", - $directory_count); - -You think about that, and realize that it doesn't even work right for -English, as it can produce this output: - - I scanned 1 directories. - -So you rewrite it to read: - - printf("I scanned %g %s.", - $directory_count, - $directory_count == 1 ? - "directory" : "directories", - ); - -...which does the Right Thing. (In case you don't recall, "%g" is for -locale-specific number interpolation, and "%s" is for string -interpolation.) - -But you still have to localize it for all the languages you're -producing this software for, so you pull Locale::gettext off of CPAN -so you can access the C<gettext> C functions you've heard are standard -for localization tasks. - -And you write: - - printf(gettext("I scanned %g %s."), - $dir_scan_count, - $dir_scan_count == 1 ? - gettext("directory") : gettext("directories"), - ); - -But you then read in the gettext manual (Drepper, Miller, and Pinard 1995) -that this is not a good idea, since how a single word like "directory" -or "directories" is translated may depend on context -- and this is -true, since in a case language like German or Russian, you'd may need -these words with a different case ending in the first instance (where the -word is the object of a verb) than in the second instance, which you haven't even -gotten to yet (where the word is the object of a preposition, "in %g -directories") -- assuming these keep the same syntax when translated -into those languages. - -So, on the advice of the gettext manual, you rewrite: - - printf( $dir_scan_count == 1 ? - gettext("I scanned %g directory.") : - gettext("I scanned %g directories."), - $dir_scan_count ); - -So, you email your various translators (the boss decides that the -languages du jour are Chinese, Arabic, Russian, and Italian, so you -have one translator for each), asking for translations for "I scanned -%g directory." and "I scanned %g directories.". When they reply, -you'll put that in the lexicons for gettext to use when it localizes -your software, so that when the user is running under the "zh" -(Chinese) locale, gettext("I scanned %g directory.") will return the -appropriate Chinese text, with a "%g" in there where printf can then -interpolate $dir_scan. - -Your Chinese translator emails right back -- he says both of these -phrases translate to the same thing in Chinese, because, in linguistic -jargon, Chinese "doesn't have number as a grammatical category" -- -whereas English does. That is, English has grammatical rules that -refer to "number", i.e., whether something is grammatically singular -or plural; and one of these rules is the one that forces nouns to take -a plural suffix (generally "s") when in a plural context, as they are when -they follow a number other than "one" (including, oddly enough, "zero"). -Chinese has no such rules, and so has just the one phrase where English -has two. But, no problem, you can have this one Chinese phrase appear -as the translation for the two English phrases in the "zh" gettext -lexicon for your program. - -Emboldened by this, you dive into the second phrase that your software -needs to output: "Your query matched 10 files in 4 directories.". You notice -that if you want to treat phrases as indivisible, as the gettext -manual wisely advises, you need four cases now, instead of two, to -cover the permutations of singular and plural on the two items, -$dir_count and $file_count. So you try this: - - printf( $file_count == 1 ? - ( $directory_count == 1 ? - gettext("Your query matched %g file in %g directory.") : - gettext("Your query matched %g file in %g directories.") ) : - ( $directory_count == 1 ? - gettext("Your query matched %g files in %g directory.") : - gettext("Your query matched %g files in %g directories.") ), - $file_count, $directory_count, - ); - -(The case of "1 file in 2 [or more] directories" could, I suppose, -occur in the case of symlinking or something of the sort.) - -It occurs to you that this is not the prettiest code you've ever -written, but this seems the way to go. You mail off to the -translators asking for translations for these four cases. The -Chinese guy replies with the one phrase that these all translate to in -Chinese, and that phrase has two "%g"s in it, as it should -- but -there's a problem. He translates it word-for-word back: "In %g -directories contains %g files match your query." The %g -slots are in an order reverse to what they are in English. You wonder -how you'll get gettext to handle that. - -But you put it aside for the moment, and optimistically hope that the -other translators won't have this problem, and that their languages -will be better behaved -- i.e., that they will be just like English. - -But the Arabic translator is the next to write back. First off, your -code for "I scanned %g directory." or "I scanned %g directories." -assumes there's only singular or plural. But, to use linguistic -jargon again, Arabic has grammatical number, like English (but unlike -Chinese), but it's a three-term category: singular, dual, and plural. -In other words, the way you say "directory" depends on whether there's -one directory, or I<two> of them, or I<more than two> of them. Your -test of C<($directory == 1)> no longer does the job. And it means -that where English's grammatical category of number necessitates -only the two permutations of the first sentence based on "directory -[singular]" and "directories [plural]", Arabic has three -- and, -worse, in the second sentence ("Your query matched %g file in %g -directory."), where English has four, Arabic has nine. You sense -an unwelcome, exponential trend taking shape. - -Your Italian translator emails you back and says that "I searched 0 -directories" (a possible English output of your program) is stilted, -and if you think that's fine English, that's your problem, but that -I<just will not do> in the language of Dante. He insists that where -$directory_count is 0, your program should produce the Italian text -for "I I<didn't> scan I<any> directories.". And ditto for "I didn't -match any files in any directories", although he says the last part -about "in any directories" should probably just be left off. - -You wonder how you'll get gettext to handle this; to accommodate the -ways Arabic, Chinese, and Italian deal with numbers in just these few -very simple phrases, you need to write code that will ask gettext for -different queries depending on whether the numerical values in -question are 1, 2, more than 2, or in some cases 0, and you still haven't -figured out the problem with the different word order in Chinese. - -Then your Russian translator calls on the phone, to I<personally> tell -you the bad news about how really unpleasant your life is about to -become: - -Russian, like German or Latin, is an inflectional language; that is, nouns -and adjectives have to take endings that depend on their case -(i.e., nominative, accusative, genitive, etc...) -- which is roughly a matter of -what role they have in syntax of the sentence -- -as well as on the grammatical gender (i.e., masculine, feminine, neuter) -and number (i.e., singular or plural) of the noun, as well as on the -declension class of the noun. But unlike with most other inflected languages, -putting a number-phrase (like "ten" or "forty-three", or their Arabic -numeral equivalents) in front of noun in Russian can change the case and -number that noun is, and therefore the endings you have to put on it. - -He elaborates: In "I scanned %g directories", you'd I<expect> -"directories" to be in the accusative case (since it is the direct -object in the sentence) and the plural number, -except where $directory_count is 1, then you'd expect the singular, of -course. Just like Latin or German. I<But!> Where $directory_count % -10 is 1 ("%" for modulo, remember), assuming $directory count is an -integer, and except where $directory_count % 100 is 11, "directories" -is forced to become grammatically singular, which means it gets the -ending for the accusative singular... You begin to visualize the code -it'd take to test for the problem so far, I<and still work for Chinese -and Arabic and Italian>, and how many gettext items that'd take, but -he keeps going... But where $directory_count % 10 is 2, 3, or 4 -(except where $directory_count % 100 is 12, 13, or 14), the word for -"directories" is forced to be genitive singular -- which means another -ending... The room begins to spin around you, slowly at first... But -with I<all other> integer values, since "directory" is an inanimate -noun, when preceded by a number and in the nominative or accusative -cases (as it is here, just your luck!), it does stay plural, but it is -forced into the genitive case -- yet another ending... And -you never hear him get to the part about how you're going to run into -similar (but maybe subtly different) problems with other Slavic -languages like Polish, because the floor comes up to meet you, and you -fade into unconsciousness. - - -The above cautionary tale relates how an attempt at localization can -lead from programmer consternation, to program obfuscation, to a need -for sedation. But careful evaluation shows that your choice of tools -merely needed further consideration. - -=head2 The Linguistic View - -=over - -"It is more complicated than you think." - --- The Eighth Networking Truth, from RFC 1925 - -=back - -The field of Linguistics has expended a great deal of effort over the -past century trying to find grammatical patterns which hold across -languages; it's been a constant process -of people making generalizations that should apply to all languages, -only to find out that, all too often, these generalizations fail -- -sometimes failing for just a few languages, sometimes whole classes of -languages, and sometimes nearly every language in the world except -English. Broad statistical trends are evident in what the "average -language" is like as far as what its rules can look like, must look -like, and cannot look like. But the "average language" is just as -unreal a concept as the "average person" -- it runs up against the -fact no language (or person) is, in fact, average. The wisdom of past -experience leads us to believe that any given language can do whatever -it wants, in any order, with appeal to any kind of grammatical -categories wants -- case, number, tense, real or metaphoric -characteristics of the things that words refer to, arbitrary or -predictable classifications of words based on what endings or prefixes -they can take, degree or means of certainty about the truth of -statements expressed, and so on, ad infinitum. - -Mercifully, most localization tasks are a matter of finding ways to -translate whole phrases, generally sentences, where the context is -relatively set, and where the only variation in content is I<usually> -in a number being expressed -- as in the example sentences above. -Translating specific, fully-formed sentences is, in practice, fairly -foolproof -- which is good, because that's what's in the phrasebooks -that so many tourists rely on. Now, a given phrase (whether in a -phrasebook or in a gettext lexicon) in one language I<might> have a -greater or lesser applicability than that phrase's translation into -another language -- for example, strictly speaking, in Arabic, the -"your" in "Your query matched..." would take a different form -depending on whether the user is male or female; so the Arabic -translation "your[feminine] query" is applicable in fewer cases than -the corresponding English phrase, which doesn't distinguish the user's -gender. (In practice, it's not feasible to have a program know the -user's gender, so the masculine "you" in Arabic is usually used, by -default.) - -But in general, such surprises are rare when entire sentences are -being translated, especially when the functional context is restricted -to that of a computer interacting with a user either to convey a fact -or to prompt for a piece of information. So, for purposes of -localization, translation by phrase (generally by sentence) is both the -simplest and the least problematic. - -=head2 Breaking gettext - -=over - -"It Has To Work." - --- First Networking Truth, RFC 1925 - -=back - -Consider that sentences in a tourist phrasebook are of two types: ones -like "How do I get to the marketplace?" that don't have any blanks to -fill in, and ones like "How much do these ___ cost?", where there's -one or more blanks to fill in (and these are usually linked to a -list of words that you can put in that blank: "fish", "potatoes", -"tomatoes", etc.) The ones with no blanks are no problem, but the -fill-in-the-blank ones may not be really straightforward. If it's a -Swahili phrasebook, for example, the authors probably didn't bother to -tell you the complicated ways that the verb "cost" changes its -inflectional prefix depending on the noun you're putting in the blank. -The trader in the marketplace will still understand what you're saying if -you say "how much do these potatoes cost?" with the wrong -inflectional prefix on "cost". After all, I<you> can't speak proper Swahili, -I<you're> just a tourist. But while tourists can be stupid, computers -are supposed to be smart; the computer should be able to fill in the -blank, and still have the results be grammatical. - -In other words, a phrasebook entry takes some values as parameters -(the things that you fill in the blank or blanks), and provides a value -based on these parameters, where the way you get that final value from -the given values can, properly speaking, involve an arbitrarily -complex series of operations. (In the case of Chinese, it'd be not at -all complex, at least in cases like the examples at the beginning of -this article; whereas in the case of Russian it'd be a rather complex -series of operations. And in some languages, the -complexity could be spread around differently: while the act of -putting a number-expression in front of a noun phrase might not be -complex by itself, it may change how you have to, for example, inflect -a verb elsewhere in the sentence. This is what in syntax is called -"long-distance dependencies".) - -This talk of parameters and arbitrary complexity is just another way -to say that an entry in a phrasebook is what in a programming language -would be called a "function". Just so you don't miss it, this is the -crux of this article: I<A phrase is a function; a phrasebook is a -bunch of functions.> - -The reason that using gettext runs into walls (as in the above -second-person horror story) is that you're trying to use a string (or -worse, a choice among a bunch of strings) to do what you really need a -function for -- which is futile. Preforming (s)printf interpolation -on the strings which you get back from gettext does allow you to do I<some> -common things passably well... sometimes... sort of; but, to paraphrase -what some people say about C<csh> script programming, "it fools you -into thinking you can use it for real things, but you can't, and you -don't discover this until you've already spent too much time trying, -and by then it's too late." - -=head2 Replacing gettext - -So, what needs to replace gettext is a system that supports lexicons -of functions instead of lexicons of strings. An entry in a lexicon -from such a system should I<not> look like this: - - "J'ai trouv\xE9 %g fichiers dans %g r\xE9pertoires" - -[\xE9 is e-acute in Latin-1. Some pod renderers would -scream if I used the actual character here. -- SB] - -but instead like this, bearing in mind that this is just a first stab: - - sub I_found_X1_files_in_X2_directories { - my( $files, $dirs ) = @_[0,1]; - $files = sprintf("%g %s", $files, - $files == 1 ? 'fichier' : 'fichiers'); - $dirs = sprintf("%g %s", $dirs, - $dirs == 1 ? "r\xE9pertoire" : "r\xE9pertoires"); - return "J'ai trouv\xE9 $files dans $dirs."; - } - -Now, there's no particularly obvious way to store anything but strings -in a gettext lexicon; so it looks like we just have to start over and -make something better, from scratch. I call my shot at a -gettext-replacement system "Maketext", or, in CPAN terms, -Locale::Maketext. - -When designing Maketext, I chose to plan its main features in terms of -"buzzword compliance". And here are the buzzwords: - -=head2 Buzzwords: Abstraction and Encapsulation - -The complexity of the language you're trying to output a phrase in is -entirely abstracted inside (and encapsulated within) the Maketext module -for that interface. When you call: - - print $lang->maketext("You have [quant,_1,piece] of new mail.", - scalar(@messages)); - -you don't know (and in fact can't easily find out) whether this will -involve lots of figuring, as in Russian (if $lang is a handle to the -Russian module), or relatively little, as in Chinese. That kind of -abstraction and encapsulation may encourage other pleasant buzzwords -like modularization and stratification, depending on what design -decisions you make. - -=head2 Buzzword: Isomorphism - -"Isomorphism" means "having the same structure or form"; in discussions -of program design, the word takes on the special, specific meaning that -your implementation of a solution to a problem I<has the same -structure> as, say, an informal verbal description of the solution, or -maybe of the problem itself. Isomorphism is, all things considered, -a good thing -- it's what problem-solving (and solution-implementing) -should look like. - -What's wrong the with gettext-using code like this... - - printf( $file_count == 1 ? - ( $directory_count == 1 ? - "Your query matched %g file in %g directory." : - "Your query matched %g file in %g directories." ) : - ( $directory_count == 1 ? - "Your query matched %g files in %g directory." : - "Your query matched %g files in %g directories." ), - $file_count, $directory_count, - ); - -is first off that it's not well abstracted -- these ways of testing -for grammatical number (as in the expressions like C<foo == 1 ? -singular_form : plural_form>) should be abstracted to each language -module, since how you get grammatical number is language-specific. - -But second off, it's not isomorphic -- the "solution" (i.e., the -phrasebook entries) for Chinese maps from these four English phrases to -the one Chinese phrase that fits for all of them. In other words, the -informal solution would be "The way to say what you want in Chinese is -with the one phrase 'For your question, in Y directories you would -find X files'" -- and so the implemented solution should be, -isomorphically, just a straightforward way to spit out that one -phrase, with numerals properly interpolated. It shouldn't have to map -from the complexity of other languages to the simplicity of this one. - -=head2 Buzzword: Inheritance - -There's a great deal of reuse possible for sharing of phrases between -modules for related dialects, or for sharing of auxiliary functions -between related languages. (By "auxiliary functions", I mean -functions that don't produce phrase-text, but which, say, return an -answer to "does this number require a plural noun after it?". Such -auxiliary functions would be used in the internal logic of functions -that actually do produce phrase-text.) - -In the case of sharing phrases, consider that you have an interface -already localized for American English (probably by having been -written with that as the native locale, but that's incidental). -Localizing it for UK English should, in practical terms, be just a -matter of running it past a British person with the instructions to -indicate what few phrases would benefit from a change in spelling or -possibly minor rewording. In that case, you should be able to put in -the UK English localization module I<only> those phrases that are -UK-specific, and for all the rest, I<inherit> from the American -English module. (And I expect this same situation would apply with -Brazilian and Continental Portugese, possibly with some I<very> -closely related languages like Czech and Slovak, and possibly with the -slightly different "versions" of written Mandarin Chinese, as I hear exist in -Taiwan and mainland China.) - -As to sharing of auxiliary functions, consider the problem of Russian -numbers from the beginning of this article; obviously, you'd want to -write only once the hairy code that, given a numeric value, would -return some specification of which case and number a given quantified -noun should use. But suppose that you discover, while localizing an -interface for, say, Ukranian (a Slavic language related to Russian, -spoken by several million people, many of whom would be relieved to -find that your Web site's or software's interface is available in -their language), that the rules in Ukranian are the same as in Russian -for quantification, and probably for many other grammatical functions. -While there may well be no phrases in common between Russian and -Ukranian, you could still choose to have the Ukranian module inherit -from the Russian module, just for the sake of inheriting all the -various grammatical methods. Or, probably better organizationally, -you could move those functions to a module called C<_E_Slavic> or -something, which Russian and Ukrainian could inherit useful functions -from, but which would (presumably) provide no lexicon. - -=head2 Buzzword: Concision - -Okay, concision isn't a buzzword. But it should be, so I decree that -as a new buzzword, "concision" means that simple common things should -be expressible in very few lines (or maybe even just a few characters) -of code -- call it a special case of "making simple things easy and -hard things possible", and see also the role it played in the -MIDI::Simple language, discussed elsewhere in this issue [TPJ#13]. - -Consider our first stab at an entry in our "phrasebook of functions": - - sub I_found_X1_files_in_X2_directories { - my( $files, $dirs ) = @_[0,1]; - $files = sprintf("%g %s", $files, - $files == 1 ? 'fichier' : 'fichiers'); - $dirs = sprintf("%g %s", $dirs, - $dirs == 1 ? "r\xE9pertoire" : "r\xE9pertoires"); - return "J'ai trouv\xE9 $files dans $dirs."; - } - -You may sense that a lexicon (to use a non-committal catch-all term for a -collection of things you know how to say, regardless of whether they're -phrases or words) consisting of functions I<expressed> as above would -make for rather long-winded and repetitive code -- even if you wisely -rewrote this to have quantification (as we call adding a number -expression to a noun phrase) be a function called like: - - sub I_found_X1_files_in_X2_directories { - my( $files, $dirs ) = @_[0,1]; - $files = quant($files, "fichier"); - $dirs = quant($dirs, "r\xE9pertoire"); - return "J'ai trouv\xE9 $files dans $dirs."; - } - -And you may also sense that you do not want to bother your translators -with having to write Perl code -- you'd much rather that they spend -their I<very costly time> on just translation. And this is to say -nothing of the near impossibility of finding a commercial translator -who would know even simple Perl. - -In a first-hack implementation of Maketext, each language-module's -lexicon looked like this: - - %Lexicon = ( - "I found %g files in %g directories" - => sub { - my( $files, $dirs ) = @_[0,1]; - $files = quant($files, "fichier"); - $dirs = quant($dirs, "r\xE9pertoire"); - return "J'ai trouv\xE9 $files dans $dirs."; - }, - ... and so on with other phrase => sub mappings ... - ); - -but I immediately went looking for some more concise way to basically -denote the same phrase-function -- a way that would also serve to -concisely denote I<most> phrase-functions in the lexicon for I<most> -languages. After much time and even some actual thought, I decided on -this system: - -* Where a value in a %Lexicon hash is a contentful string instead of -an anonymous sub (or, conceivably, a coderef), it would be interpreted -as a sort of shorthand expression of what the sub does. When accessed -for the first time in a session, it is parsed, turned into Perl code, -and then eval'd into an anonymous sub; then that sub replaces the -original string in that lexicon. (That way, the work of parsing and -evaling the shorthand form for a given phrase is done no more than -once per session.) - -* Calls to C<maketext> (as Maketext's main function is called) happen -thru a "language session handle", notionally very much like an IO -handle, in that you open one at the start of the session, and use it -for "sending signals" to an object in order to have it return the text -you want. - -So, this: - - $lang->maketext("You have [quant,_1,piece] of new mail.", - scalar(@messages)); - -basically means this: look in the lexicon for $lang (which may inherit -from any number of other lexicons), and find the function that we -happen to associate with the string "You have [quant,_1,piece] of new -mail" (which is, and should be, a functioning "shorthand" for this -function in the native locale -- English in this case). If you find -such a function, call it with $lang as its first parameter (as if it -were a method), and then a copy of scalar(@messages) as its second, -and then return that value. If that function was found, but was in -string shorthand instead of being a fully specified function, parse it -and make it into a function before calling it the first time. - -* The shorthand uses code in brackets to indicate method calls that -should be performed. A full explanation is not in order here, but a -few examples will suffice: - - "You have [quant,_1,piece] of new mail." - -The above code is shorthand for, and will be interpreted as, -this: - - sub { - my $handle = $_[0]; - my(@params) = @_; - return join '', - "You have ", - $handle->quant($params[1], 'piece'), - "of new mail."; - } - -where "quant" is the name of a method you're using to quantify the -noun "piece" with the number $params[0]. - -A string with no brackety calls, like this: - - "Your search expression was malformed." - -is somewhat of a degenerate case, and just gets turned into: - - sub { return "Your search expression was malformed." } - -However, not everything you can write in Perl code can be written in -the above shorthand system -- not by a long shot. For example, consider -the Italian translator from the beginning of this article, who wanted -the Italian for "I didn't find any files" as a special case, instead -of "I found 0 files". That couldn't be specified (at least not easily -or simply) in our shorthand system, and it would have to be written -out in full, like this: - - sub { # pretend the English strings are in Italian - my($handle, $files, $dirs) = @_[0,1,2]; - return "I didn't find any files" unless $files; - return join '', - "I found ", - $handle->quant($files, 'file'), - " in ", - $handle->quant($dirs, 'directory'), - "."; - } - -Next to a lexicon full of shorthand code, that sort of sticks out like a -sore thumb -- but this I<is> a special case, after all; and at least -it's possible, if not as concise as usual. - -As to how you'd implement the Russian example from the beginning of -the article, well, There's More Than One Way To Do It, but it could be -something like this (using English words for Russian, just so you know -what's going on): - - "I [quant,_1,directory,accusative] scanned." - -This shifts the burden of complexity off to the quant method. That -method's parameters are: the numeric value it's going to use to -quantify something; the Russian word it's going to quantify; and the -parameter "accusative", which you're using to mean that this -sentence's syntax wants a noun in the accusative case there, although -that quantification method may have to overrule, for grammatical -reasons you may recall from the beginning of this article. - -Now, the Russian quant method here is responsible not only for -implementing the strange logic necessary for figuring out how Russian -number-phrases impose case and number on their noun-phrases, but also -for inflecting the Russian word for "directory". How that inflection -is to be carried out is no small issue, and among the solutions I've -seen, some (like variations on a simple lookup in a hash where all -possible forms are provided for all necessary words) are -straightforward but I<can> become cumbersome when you need to inflect -more than a few dozen words; and other solutions (like using -algorithms to model the inflections, storing only root forms and -irregularities) I<can> involve more overhead than is justifiable for -all but the largest lexicons. - -Mercifully, this design decision becomes crucial only in the hairiest -of inflected languages, of which Russian is by no means the I<worst> case -scenario, but is worse than most. Most languages have simpler -inflection systems; for example, in English or Swahili, there are -generally no more than two possible inflected forms for a given noun -("error/errors"; "kosa/makosa"), and the -rules for producing these forms are fairly simple -- or at least, -simple rules can be formulated that work for most words, and you can -then treat the exceptions as just "irregular", at least relative to -your ad hoc rules. A simpler inflection system (simpler rules, fewer -forms) means that design decisions are less crucial to maintaining -sanity, whereas the same decisions could incur -overhead-versus-scalability problems in languages like Russian. It -may I<also> be likely that code (possibly in Perl, as with -Lingua::EN::Inflect, for English nouns) has already -been written for the language in question, whether simple or complex. - -Moreover, a third possibility may even be simpler than anything -discussed above: "Just require that all possible (or at least -applicable) forms be provided in the call to the given language's quant -method, as in:" - - "I found [quant,_1,file,files]." - -That way, quant just has to chose which form it needs, without having -to look up or generate anything. While possibly not optimal for -Russian, this should work well for most other languages, where -quantification is not as complicated an operation. - -=head2 The Devil in the Details - -There's plenty more to Maketext than described above -- for example, -there's the details of how language tags ("en-US", "i-pwn", "fi", -etc.) or locale IDs ("en_US") interact with actual module naming -("BogoQuery/Locale/en_us.pm"), and what magic can ensue; there's the -details of how to record (and possibly negotiate) what character -encoding Maketext will return text in (UTF8? Latin-1? KOI8?). There's -the interesting fact that Maketext is for localization, but nowhere -actually has a "C<use locale;>" anywhere in it. For the curious, -there's the somewhat frightening details of how I actually -implement something like data inheritance so that searches across -modules' %Lexicon hashes can parallel how Perl implements method -inheritance. - -And, most importantly, there's all the practical details of how to -actually go about deriving from Maketext so you can use it for your -interfaces, and the various tools and conventions for starting out and -maintaining individual language modules. - -That is all covered in the documentation for Locale::Maketext and the -modules that come with it, available in CPAN. After having read this -article, which covers the why's of Maketext, the documentation, -which covers the how's of it, should be quite straightforward. - -=head2 The Proof in the Pudding: Localizing Web Sites - -Maketext and gettext have a notable difference: gettext is in C, -accessible thru C library calls, whereas Maketext is in Perl, and -really can't work without a Perl interpreter (although I suppose -something like it could be written for C). Accidents of history (and -not necessarily lucky ones) have made C++ the most common language for -the implementation of applications like word processors, Web browsers, -and even many in-house applications like custom query systems. Current -conditions make it somewhat unlikely that the next one of any of these -kinds of applications will be written in Perl, albeit clearly more for -reasons of custom and inertia than out of consideration of what is the -right tool for the job. - -However, other accidents of history have made Perl a well-accepted -language for design of server-side programs (generally in CGI form) -for Web site interfaces. Localization of static pages in Web sites is -trivial, feasable either with simple language-negotiation features in -servers like Apache, or with some kind of server-side inclusions of -language-appropriate text into layout templates. However, I think -that the localization of Perl-based search systems (or other kinds of -dynamic content) in Web sites, be they public or access-restricted, -is where Maketext will see the greatest use. - -I presume that it would be only the exceptional Web site that gets -localized for English I<and> Chinese I<and> Italian I<and> Arabic -I<and> Russian, to recall the languages from the beginning of this -article -- to say nothing of German, Spanish, French, Japanese, -Finnish, and Hindi, to name a few languages that benefit from large -numbers of programmers or Web viewers or both. - -However, the ever-increasing internationalization of the Web (whether -measured in terms of amount of content, of numbers of content writers -or programmers, or of size of content audiences) makes it increasingly -likely that the interface to the average Web-based dynamic content -service will be localized for two or maybe three languages. It is my -hope that Maketext will make that task as simple as possible, and will -remove previous barriers to localization for languages dissimilar to -English. - - __END__ - -Sean M. Burke (sburkeE<64>cpan.org) has a Master's in linguistics -from Northwestern University; he specializes in language technology. -Jordan Lachler (lachlerE<64>unm.edu) is a PhD student in the Department of -Linguistics at the University of New Mexico; he specializes in -morphology and pedagogy of North American native languages. - -=head2 References - -Alvestrand, Harald Tveit. 1995. I<RFC 1766: Tags for the -Identification of Languages.> -C<http://www.ietf.org/rfc/rfc1766.txt> -[Now see RFC 3066.] - -Callon, Ross, editor. 1996. I<RFC 1925: The Twelve -Networking Truths.> -C<http://www.ietf.org/rfc/rfc1925.txt> - -Drepper, Ulrich, Peter Miller, -and FranE<ccedil>ois Pinard. 1995-2001. GNU -C<gettext>. Available in C<ftp://prep.ai.mit.edu/pub/gnu/>, with -extensive docs in the distribution tarball. [Since -I wrote this article in 1998, I now see that the -gettext docs are now trying more to come to terms with -plurality. Whether useful conclusions have come from it -is another question altogether. -- SMB, May 2001] - -Forbes, Nevill. 1964. I<Russian Grammar.> Third Edition, revised -by J. C. Dumbreck. Oxford University Press. - -=cut - -#End - |