summaryrefslogtreecommitdiff
path: root/Master/tlpkg/tlperl/lib/pods/perlhack.pod
diff options
context:
space:
mode:
authorSiep Kroonenberg <siepo@cybercomm.nl>2011-02-17 17:31:37 +0000
committerSiep Kroonenberg <siepo@cybercomm.nl>2011-02-17 17:31:37 +0000
commit227723589c9564b9b2b7042dd27eb9aad56a560a (patch)
tree5185d4773197a8bc419342ca26b1db30dc992b5a /Master/tlpkg/tlperl/lib/pods/perlhack.pod
parent98f44e1fb37d3dbed7ffbc066daac04371ed6f3a (diff)
New tlperl part XI
git-svn-id: svn://tug.org/texlive/trunk@21433 c570f23f-e606-0410-a88d-b1316a301751
Diffstat (limited to 'Master/tlpkg/tlperl/lib/pods/perlhack.pod')
-rw-r--r--Master/tlpkg/tlperl/lib/pods/perlhack.pod3401
1 files changed, 3401 insertions, 0 deletions
diff --git a/Master/tlpkg/tlperl/lib/pods/perlhack.pod b/Master/tlpkg/tlperl/lib/pods/perlhack.pod
new file mode 100644
index 00000000000..844b83b7b05
--- /dev/null
+++ b/Master/tlpkg/tlperl/lib/pods/perlhack.pod
@@ -0,0 +1,3401 @@
+=head1 NAME
+
+perlhack - How to hack at the Perl internals
+
+=head1 DESCRIPTION
+
+This document attempts to explain how Perl development takes place,
+and ends with some suggestions for people wanting to become bona fide
+porters.
+
+The perl5-porters mailing list is where the Perl standard distribution
+is maintained and developed. The list can get anywhere from 10 to 150
+messages a day, depending on the heatedness of the debate. Most days
+there are two or three patches, extensions, features, or bugs being
+discussed at a time.
+
+A searchable archive of the list is at either:
+
+ http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/
+
+or
+
+ http://archive.develooper.com/perl5-porters@perl.org/
+
+List subscribers (the porters themselves) come in several flavours.
+Some are quiet curious lurkers, who rarely pitch in and instead watch
+the ongoing development to ensure they're forewarned of new changes or
+features in Perl. Some are representatives of vendors, who are there
+to make sure that Perl continues to compile and work on their
+platforms. Some patch any reported bug that they know how to fix,
+some are actively patching their pet area (threads, Win32, the regexp
+engine), while others seem to do nothing but complain. In other
+words, it's your usual mix of technical people.
+
+Over this group of porters presides Larry Wall. He has the final word
+in what does and does not change in the Perl language. Various
+releases of Perl are shepherded by a "pumpking", a porter
+responsible for gathering patches, deciding on a patch-by-patch,
+feature-by-feature basis what will and will not go into the release.
+For instance, Gurusamy Sarathy was the pumpking for the 5.6 release of
+Perl, and Jarkko Hietaniemi was the pumpking for the 5.8 release, and
+Rafael Garcia-Suarez holds the pumpking crown for the 5.10 release.
+
+In addition, various people are pumpkings for different things. For
+instance, Andy Dougherty and Jarkko Hietaniemi did a grand job as the
+I<Configure> pumpkin up till the 5.8 release. For the 5.10 release
+H.Merijn Brand took over.
+
+Larry sees Perl development along the lines of the US government:
+there's the Legislature (the porters), the Executive branch (the
+pumpkings), and the Supreme Court (Larry). The legislature can
+discuss and submit patches to the executive branch all they like, but
+the executive branch is free to veto them. Rarely, the Supreme Court
+will side with the executive branch over the legislature, or the
+legislature over the executive branch. Mostly, however, the
+legislature and the executive branch are supposed to get along and
+work out their differences without impeachment or court cases.
+
+You might sometimes see reference to Rule 1 and Rule 2. Larry's power
+as Supreme Court is expressed in The Rules:
+
+=over 4
+
+=item 1
+
+Larry is always by definition right about how Perl should behave.
+This means he has final veto power on the core functionality.
+
+=item 2
+
+Larry is allowed to change his mind about any matter at a later date,
+regardless of whether he previously invoked Rule 1.
+
+=back
+
+Got that? Larry is always right, even when he was wrong. It's rare
+to see either Rule exercised, but they are often alluded to.
+
+New features and extensions to the language are contentious, because
+the criteria used by the pumpkings, Larry, and other porters to decide
+which features should be implemented and incorporated are not codified
+in a few small design goals as with some other languages. Instead,
+the heuristics are flexible and often difficult to fathom. Here is
+one person's list, roughly in decreasing order of importance, of
+heuristics that new features have to be weighed against:
+
+=over 4
+
+=item Does concept match the general goals of Perl?
+
+These haven't been written anywhere in stone, but one approximation
+is:
+
+ 1. Keep it fast, simple, and useful.
+ 2. Keep features/concepts as orthogonal as possible.
+ 3. No arbitrary limits (platforms, data sizes, cultures).
+ 4. Keep it open and exciting to use/patch/advocate Perl everywhere.
+ 5. Either assimilate new technologies, or build bridges to them.
+
+=item Where is the implementation?
+
+All the talk in the world is useless without an implementation. In
+almost every case, the person or people who argue for a new feature
+will be expected to be the ones who implement it. Porters capable
+of coding new features have their own agendas, and are not available
+to implement your (possibly good) idea.
+
+=item Backwards compatibility
+
+It's a cardinal sin to break existing Perl programs. New warnings are
+contentious--some say that a program that emits warnings is not
+broken, while others say it is. Adding keywords has the potential to
+break programs, changing the meaning of existing token sequences or
+functions might break programs.
+
+=item Could it be a module instead?
+
+Perl 5 has extension mechanisms, modules and XS, specifically to avoid
+the need to keep changing the Perl interpreter. You can write modules
+that export functions, you can give those functions prototypes so they
+can be called like built-in functions, you can even write XS code to
+mess with the runtime data structures of the Perl interpreter if you
+want to implement really complicated things. If it can be done in a
+module instead of in the core, it's highly unlikely to be added.
+
+=item Is the feature generic enough?
+
+Is this something that only the submitter wants added to the language,
+or would it be broadly useful? Sometimes, instead of adding a feature
+with a tight focus, the porters might decide to wait until someone
+implements the more generalized feature. For instance, instead of
+implementing a "delayed evaluation" feature, the porters are waiting
+for a macro system that would permit delayed evaluation and much more.
+
+=item Does it potentially introduce new bugs?
+
+Radical rewrites of large chunks of the Perl interpreter have the
+potential to introduce new bugs. The smaller and more localized the
+change, the better.
+
+=item Does it preclude other desirable features?
+
+A patch is likely to be rejected if it closes off future avenues of
+development. For instance, a patch that placed a true and final
+interpretation on prototypes is likely to be rejected because there
+are still options for the future of prototypes that haven't been
+addressed.
+
+=item Is the implementation robust?
+
+Good patches (tight code, complete, correct) stand more chance of
+going in. Sloppy or incorrect patches might be placed on the back
+burner until the pumpking has time to fix, or might be discarded
+altogether without further notice.
+
+=item Is the implementation generic enough to be portable?
+
+The worst patches make use of a system-specific features. It's highly
+unlikely that non-portable additions to the Perl language will be
+accepted.
+
+=item Is the implementation tested?
+
+Patches which change behaviour (fixing bugs or introducing new features)
+must include regression tests to verify that everything works as expected.
+Without tests provided by the original author, how can anyone else changing
+perl in the future be sure that they haven't unwittingly broken the behaviour
+the patch implements? And without tests, how can the patch's author be
+confident that his/her hard work put into the patch won't be accidentally
+thrown away by someone in the future?
+
+=item Is there enough documentation?
+
+Patches without documentation are probably ill-thought out or
+incomplete. Nothing can be added without documentation, so submitting
+a patch for the appropriate manpages as well as the source code is
+always a good idea.
+
+=item Is there another way to do it?
+
+Larry said "Although the Perl Slogan is I<There's More Than One Way
+to Do It>, I hesitate to make 10 ways to do something". This is a
+tricky heuristic to navigate, though--one man's essential addition is
+another man's pointless cruft.
+
+=item Does it create too much work?
+
+Work for the pumpking, work for Perl programmers, work for module
+authors, ... Perl is supposed to be easy.
+
+=item Patches speak louder than words
+
+Working code is always preferred to pie-in-the-sky ideas. A patch to
+add a feature stands a much higher chance of making it to the language
+than does a random feature request, no matter how fervently argued the
+request might be. This ties into "Will it be useful?", as the fact
+that someone took the time to make the patch demonstrates a strong
+desire for the feature.
+
+=back
+
+If you're on the list, you might hear the word "core" bandied
+around. It refers to the standard distribution. "Hacking on the
+core" means you're changing the C source code to the Perl
+interpreter. "A core module" is one that ships with Perl.
+
+=head2 Keeping in sync
+
+The source code to the Perl interpreter, in its different versions, is
+kept in a repository managed by the git revision control system. The
+pumpkings and a few others have write access to the repository to check in
+changes.
+
+How to clone and use the git perl repository is described in L<perlrepository>.
+
+You can also choose to use rsync to get a copy of the current source tree
+for the bleadperl branch and all maintenance branches:
+
+ $ rsync -avz rsync://perl5.git.perl.org/perl-current .
+ $ rsync -avz rsync://perl5.git.perl.org/perl-5.12.x .
+ $ rsync -avz rsync://perl5.git.perl.org/perl-5.10.x .
+ $ rsync -avz rsync://perl5.git.perl.org/perl-5.8.x .
+ $ rsync -avz rsync://perl5.git.perl.org/perl-5.6.x .
+ $ rsync -avz rsync://perl5.git.perl.org/perl-5.005xx .
+
+(Add the C<--delete> option to remove leftover files)
+
+To get a full list of the available sync points:
+
+ $ rsync perl5.git.perl.org::
+
+You may also want to subscribe to the perl5-changes mailing list to
+receive a copy of each patch that gets submitted to the maintenance
+and development "branches" of the perl repository. See
+http://lists.perl.org/ for subscription information.
+
+If you are a member of the perl5-porters mailing list, it is a good
+thing to keep in touch with the most recent changes. If not only to
+verify if what you would have posted as a bug report isn't already
+solved in the most recent available perl development branch, also
+known as perl-current, bleading edge perl, bleedperl or bleadperl.
+
+Needless to say, the source code in perl-current is usually in a perpetual
+state of evolution. You should expect it to be very buggy. Do B<not> use
+it for any purpose other than testing and development.
+
+=head2 Perlbug administration
+
+There is a single remote administrative interface for modifying bug status,
+category, open issues etc. using the B<RT> bugtracker system, maintained
+by Robert Spier. Become an administrator, and close any bugs you can get
+your sticky mitts on:
+
+ http://bugs.perl.org/
+
+To email the bug system administrators:
+
+ "perlbug-admin" <perlbug-admin@perl.org>
+
+=head2 Submitting patches
+
+Always submit patches to I<perl5-porters@perl.org>. If you're
+patching a core module and there's an author listed, send the author a
+copy (see L<Patching a core module>). This lets other porters review
+your patch, which catches a surprising number of errors in patches.
+Please patch against the latest B<development> version. (e.g., even if
+you're fixing a bug in the 5.8 track, patch against the C<blead> branch in
+the git repository.)
+
+If changes are accepted, they are applied to the development branch. Then
+the maintenance pumpking decides which of those patches is to be
+backported to the maint branch. Only patches that survive the heat of the
+development branch get applied to maintenance versions.
+
+Your patch should update the documentation and test suite. See
+L<Writing a test>. If you have added or removed files in the distribution,
+edit the MANIFEST file accordingly, sort the MANIFEST file using
+C<make manisort>, and include those changes as part of your patch.
+
+Patching documentation also follows the same order: if accepted, a patch
+is first applied to B<development>, and if relevant then it's backported
+to B<maintenance>. (With an exception for some patches that document
+behaviour that only appears in the maintenance branch, but which has
+changed in the development version.)
+
+To report a bug in Perl, use the program I<perlbug> which comes with
+Perl (if you can't get Perl to work, send mail to the address
+I<perlbug@perl.org> or I<perlbug@perl.com>). Reporting bugs through
+I<perlbug> feeds into the automated bug-tracking system, access to
+which is provided through the web at http://rt.perl.org/rt3/ . It
+often pays to check the archives of the perl5-porters mailing list to
+see whether the bug you're reporting has been reported before, and if
+so whether it was considered a bug. See above for the location of
+the searchable archives.
+
+The CPAN testers ( http://testers.cpan.org/ ) are a group of
+volunteers who test CPAN modules on a variety of platforms. Perl
+Smokers ( http://www.nntp.perl.org/group/perl.daily-build and
+http://www.nntp.perl.org/group/perl.daily-build.reports/ )
+automatically test Perl source releases on platforms with various
+configurations. Both efforts welcome volunteers. In order to get
+involved in smoke testing of the perl itself visit
+L<http://search.cpan.org/dist/Test-Smoke>. In order to start smoke
+testing CPAN modules visit L<http://search.cpan.org/dist/CPANPLUS-YACSmoke/>
+or L<http://search.cpan.org/dist/minismokebox/> or
+L<http://search.cpan.org/dist/CPAN-Reporter/>.
+
+It's a good idea to read and lurk for a while before chipping in.
+That way you'll get to see the dynamic of the conversations, learn the
+personalities of the players, and hopefully be better prepared to make
+a useful contribution when do you speak up.
+
+If after all this you still think you want to join the perl5-porters
+mailing list, send mail to I<perl5-porters-subscribe@perl.org>. To
+unsubscribe, send mail to I<perl5-porters-unsubscribe@perl.org>.
+
+To hack on the Perl guts, you'll need to read the following things:
+
+=over 3
+
+=item L<perlguts>
+
+This is of paramount importance, since it's the documentation of what
+goes where in the Perl source. Read it over a couple of times and it
+might start to make sense - don't worry if it doesn't yet, because the
+best way to study it is to read it in conjunction with poking at Perl
+source, and we'll do that later on.
+
+Gisle Aas's "illustrated perlguts", also known as I<illguts>, has very
+helpful pictures:
+
+L<http://search.cpan.org/dist/illguts/>
+
+=item L<perlxstut> and L<perlxs>
+
+A working knowledge of XSUB programming is incredibly useful for core
+hacking; XSUBs use techniques drawn from the PP code, the portion of the
+guts that actually executes a Perl program. It's a lot gentler to learn
+those techniques from simple examples and explanation than from the core
+itself.
+
+=item L<perlapi>
+
+The documentation for the Perl API explains what some of the internal
+functions do, as well as the many macros used in the source.
+
+=item F<Porting/pumpkin.pod>
+
+This is a collection of words of wisdom for a Perl porter; some of it is
+only useful to the pumpkin holder, but most of it applies to anyone
+wanting to go about Perl development.
+
+=item The perl5-porters FAQ
+
+This should be available from http://dev.perl.org/perl5/docs/p5p-faq.html .
+It contains hints on reading perl5-porters, information on how
+perl5-porters works and how Perl development in general works.
+
+=back
+
+=head2 Finding Your Way Around
+
+Perl maintenance can be split into a number of areas, and certain people
+(pumpkins) will have responsibility for each area. These areas sometimes
+correspond to files or directories in the source kit. Among the areas are:
+
+=over 3
+
+=item Core modules
+
+Modules shipped as part of the Perl core live in various subdirectories, where
+two are dedicated to core-only modules, and two are for the dual-life modules
+which live on CPAN and may be maintained separately with respect to the Perl
+core:
+
+ lib/ is for pure-Perl modules, which exist in the core only.
+
+ ext/ is for XS extensions, and modules with special Makefile.PL
+ requirements, which exist in the core only.
+
+ cpan/ is for dual-life modules, where the CPAN module is
+ canonical (should be patched first).
+
+ dist/ is for dual-life modules, where the blead source is
+ canonical.
+
+For some dual-life modules it has not been discussed if the CPAN version or the
+blead source is canonical. Until that is done, those modules should be in
+F<cpan/>.
+
+=item Tests
+
+There are tests for nearly all the modules, built-ins and major bits
+of functionality. Test files all have a .t suffix. Module tests live
+in the F<lib/> and F<ext/> directories next to the module being
+tested. Others live in F<t/>. See L<Writing a test>
+
+=item Documentation
+
+Documentation maintenance includes looking after everything in the
+F<pod/> directory, (as well as contributing new documentation) and
+the documentation to the modules in core.
+
+=item Configure
+
+The Configure process is the way we make Perl portable across the
+myriad of operating systems it supports. Responsibility for the
+Configure, build and installation process, as well as the overall
+portability of the core code rests with the Configure pumpkin -
+others help out with individual operating systems.
+
+The three files that fall under his/her responsibility are Configure,
+config_h.SH, and Porting/Glossary (and a whole bunch of small related
+files that are less important here). The Configure pumpkin decides how
+patches to these are dealt with. Currently, the Configure pumpkin will
+accept patches in most common formats, even directly to these files.
+Other committers are allowed to commit to these files under the strict
+condition that they will inform the Configure pumpkin, either on IRC
+(if he/she happens to be around) or through (personal) e-mail.
+
+The files involved are the operating system directories, (F<win32/>,
+F<os2/>, F<vms/> and so on) the shell scripts which generate F<config.h>
+and F<Makefile>, as well as the metaconfig files which generate
+F<Configure>. (metaconfig isn't included in the core distribution.)
+
+See http://perl5.git.perl.org/metaconfig.git/blob/HEAD:/README for a
+description of the full process involved.
+
+=item Interpreter
+
+And of course, there's the core of the Perl interpreter itself. Let's
+have a look at that in a little more detail.
+
+=back
+
+Before we leave looking at the layout, though, don't forget that
+F<MANIFEST> contains not only the file names in the Perl distribution,
+but short descriptions of what's in them, too. For an overview of the
+important files, try this:
+
+ perl -lne 'print if /^[^\/]+\.[ch]\s+/' MANIFEST
+
+=head2 Elements of the interpreter
+
+The work of the interpreter has two main stages: compiling the code
+into the internal representation, or bytecode, and then executing it.
+L<perlguts/Compiled code> explains exactly how the compilation stage
+happens.
+
+Here is a short breakdown of perl's operation:
+
+=over 3
+
+=item Startup
+
+The action begins in F<perlmain.c>. (or F<miniperlmain.c> for miniperl)
+This is very high-level code, enough to fit on a single screen, and it
+resembles the code found in L<perlembed>; most of the real action takes
+place in F<perl.c>
+
+F<perlmain.c> is generated by L<writemain> from F<miniperlmain.c> at
+make time, so you should make perl to follow this along.
+
+First, F<perlmain.c> allocates some memory and constructs a Perl
+interpreter, along these lines:
+
+ 1 PERL_SYS_INIT3(&argc,&argv,&env);
+ 2
+ 3 if (!PL_do_undump) {
+ 4 my_perl = perl_alloc();
+ 5 if (!my_perl)
+ 6 exit(1);
+ 7 perl_construct(my_perl);
+ 8 PL_perl_destruct_level = 0;
+ 9 }
+
+Line 1 is a macro, and its definition is dependent on your operating
+system. Line 3 references C<PL_do_undump>, a global variable - all
+global variables in Perl start with C<PL_>. This tells you whether the
+current running program was created with the C<-u> flag to perl and then
+F<undump>, which means it's going to be false in any sane context.
+
+Line 4 calls a function in F<perl.c> to allocate memory for a Perl
+interpreter. It's quite a simple function, and the guts of it looks like
+this:
+
+ my_perl = (PerlInterpreter*)PerlMem_malloc(sizeof(PerlInterpreter));
+
+Here you see an example of Perl's system abstraction, which we'll see
+later: C<PerlMem_malloc> is either your system's C<malloc>, or Perl's
+own C<malloc> as defined in F<malloc.c> if you selected that option at
+configure time.
+
+Next, in line 7, we construct the interpreter using perl_construct,
+also in F<perl.c>; this sets up all the special variables that Perl
+needs, the stacks, and so on.
+
+Now we pass Perl the command line options, and tell it to go:
+
+ exitstatus = perl_parse(my_perl, xs_init, argc, argv, (char **)NULL);
+ if (!exitstatus)
+ perl_run(my_perl);
+
+ exitstatus = perl_destruct(my_perl);
+
+ perl_free(my_perl);
+
+C<perl_parse> is actually a wrapper around C<S_parse_body>, as defined
+in F<perl.c>, which processes the command line options, sets up any
+statically linked XS modules, opens the program and calls C<yyparse> to
+parse it.
+
+=item Parsing
+
+The aim of this stage is to take the Perl source, and turn it into an op
+tree. We'll see what one of those looks like later. Strictly speaking,
+there's three things going on here.
+
+C<yyparse>, the parser, lives in F<perly.c>, although you're better off
+reading the original YACC input in F<perly.y>. (Yes, Virginia, there
+B<is> a YACC grammar for Perl!) The job of the parser is to take your
+code and "understand" it, splitting it into sentences, deciding which
+operands go with which operators and so on.
+
+The parser is nobly assisted by the lexer, which chunks up your input
+into tokens, and decides what type of thing each token is: a variable
+name, an operator, a bareword, a subroutine, a core function, and so on.
+The main point of entry to the lexer is C<yylex>, and that and its
+associated routines can be found in F<toke.c>. Perl isn't much like
+other computer languages; it's highly context sensitive at times, it can
+be tricky to work out what sort of token something is, or where a token
+ends. As such, there's a lot of interplay between the tokeniser and the
+parser, which can get pretty frightening if you're not used to it.
+
+As the parser understands a Perl program, it builds up a tree of
+operations for the interpreter to perform during execution. The routines
+which construct and link together the various operations are to be found
+in F<op.c>, and will be examined later.
+
+=item Optimization
+
+Now the parsing stage is complete, and the finished tree represents
+the operations that the Perl interpreter needs to perform to execute our
+program. Next, Perl does a dry run over the tree looking for
+optimisations: constant expressions such as C<3 + 4> will be computed
+now, and the optimizer will also see if any multiple operations can be
+replaced with a single one. For instance, to fetch the variable C<$foo>,
+instead of grabbing the glob C<*foo> and looking at the scalar
+component, the optimizer fiddles the op tree to use a function which
+directly looks up the scalar in question. The main optimizer is C<peep>
+in F<op.c>, and many ops have their own optimizing functions.
+
+=item Running
+
+Now we're finally ready to go: we have compiled Perl byte code, and all
+that's left to do is run it. The actual execution is done by the
+C<runops_standard> function in F<run.c>; more specifically, it's done by
+these three innocent looking lines:
+
+ while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
+ PERL_ASYNC_CHECK();
+ }
+
+You may be more comfortable with the Perl version of that:
+
+ PERL_ASYNC_CHECK() while $Perl::op = &{$Perl::op->{function}};
+
+Well, maybe not. Anyway, each op contains a function pointer, which
+stipulates the function which will actually carry out the operation.
+This function will return the next op in the sequence - this allows for
+things like C<if> which choose the next op dynamically at run time.
+The C<PERL_ASYNC_CHECK> makes sure that things like signals interrupt
+execution if required.
+
+The actual functions called are known as PP code, and they're spread
+between four files: F<pp_hot.c> contains the "hot" code, which is most
+often used and highly optimized, F<pp_sys.c> contains all the
+system-specific functions, F<pp_ctl.c> contains the functions which
+implement control structures (C<if>, C<while> and the like) and F<pp.c>
+contains everything else. These are, if you like, the C code for Perl's
+built-in functions and operators.
+
+Note that each C<pp_> function is expected to return a pointer to the next
+op. Calls to perl subs (and eval blocks) are handled within the same
+runops loop, and do not consume extra space on the C stack. For example,
+C<pp_entersub> and C<pp_entertry> just push a C<CxSUB> or C<CxEVAL> block
+struct onto the context stack which contain the address of the op
+following the sub call or eval. They then return the first op of that sub
+or eval block, and so execution continues of that sub or block. Later, a
+C<pp_leavesub> or C<pp_leavetry> op pops the C<CxSUB> or C<CxEVAL>,
+retrieves the return op from it, and returns it.
+
+=item Exception handing
+
+Perl's exception handing (i.e. C<die> etc.) is built on top of the low-level
+C<setjmp()>/C<longjmp()> C-library functions. These basically provide a
+way to capture the current PC and SP registers and later restore them; i.e.
+a C<longjmp()> continues at the point in code where a previous C<setjmp()>
+was done, with anything further up on the C stack being lost. This is why
+code should always save values using C<SAVE_FOO> rather than in auto
+variables.
+
+The perl core wraps C<setjmp()> etc in the macros C<JMPENV_PUSH> and
+C<JMPENV_JUMP>. The basic rule of perl exceptions is that C<exit>, and
+C<die> (in the absence of C<eval>) perform a C<JMPENV_JUMP(2)>, while
+C<die> within C<eval> does a C<JMPENV_JUMP(3)>.
+
+At entry points to perl, such as C<perl_parse()>, C<perl_run()> and
+C<call_sv(cv, G_EVAL)> each does a C<JMPENV_PUSH>, then enter a runops
+loop or whatever, and handle possible exception returns. For a 2 return,
+final cleanup is performed, such as popping stacks and calling C<CHECK> or
+C<END> blocks. Amongst other things, this is how scope cleanup still
+occurs during an C<exit>.
+
+If a C<die> can find a C<CxEVAL> block on the context stack, then the
+stack is popped to that level and the return op in that block is assigned
+to C<PL_restartop>; then a C<JMPENV_JUMP(3)> is performed. This normally
+passes control back to the guard. In the case of C<perl_run> and
+C<call_sv>, a non-null C<PL_restartop> triggers re-entry to the runops
+loop. The is the normal way that C<die> or C<croak> is handled within an
+C<eval>.
+
+Sometimes ops are executed within an inner runops loop, such as tie, sort
+or overload code. In this case, something like
+
+ sub FETCH { eval { die } }
+
+would cause a longjmp right back to the guard in C<perl_run>, popping both
+runops loops, which is clearly incorrect. One way to avoid this is for the
+tie code to do a C<JMPENV_PUSH> before executing C<FETCH> in the inner
+runops loop, but for efficiency reasons, perl in fact just sets a flag,
+using C<CATCH_SET(TRUE)>. The C<pp_require>, C<pp_entereval> and
+C<pp_entertry> ops check this flag, and if true, they call C<docatch>,
+which does a C<JMPENV_PUSH> and starts a new runops level to execute the
+code, rather than doing it on the current loop.
+
+As a further optimisation, on exit from the eval block in the C<FETCH>,
+execution of the code following the block is still carried on in the inner
+loop. When an exception is raised, C<docatch> compares the C<JMPENV>
+level of the C<CxEVAL> with C<PL_top_env> and if they differ, just
+re-throws the exception. In this way any inner loops get popped.
+
+Here's an example.
+
+ 1: eval { tie @a, 'A' };
+ 2: sub A::TIEARRAY {
+ 3: eval { die };
+ 4: die;
+ 5: }
+
+To run this code, C<perl_run> is called, which does a C<JMPENV_PUSH> then
+enters a runops loop. This loop executes the eval and tie ops on line 1,
+with the eval pushing a C<CxEVAL> onto the context stack.
+
+The C<pp_tie> does a C<CATCH_SET(TRUE)>, then starts a second runops loop
+to execute the body of C<TIEARRAY>. When it executes the entertry op on
+line 3, C<CATCH_GET> is true, so C<pp_entertry> calls C<docatch> which
+does a C<JMPENV_PUSH> and starts a third runops loop, which then executes
+the die op. At this point the C call stack looks like this:
+
+ Perl_pp_die
+ Perl_runops # third loop
+ S_docatch_body
+ S_docatch
+ Perl_pp_entertry
+ Perl_runops # second loop
+ S_call_body
+ Perl_call_sv
+ Perl_pp_tie
+ Perl_runops # first loop
+ S_run_body
+ perl_run
+ main
+
+and the context and data stacks, as shown by C<-Dstv>, look like:
+
+ STACK 0: MAIN
+ CX 0: BLOCK =>
+ CX 1: EVAL => AV() PV("A"\0)
+ retop=leave
+ STACK 1: MAGIC
+ CX 0: SUB =>
+ retop=(null)
+ CX 1: EVAL => *
+ retop=nextstate
+
+The die pops the first C<CxEVAL> off the context stack, sets
+C<PL_restartop> from it, does a C<JMPENV_JUMP(3)>, and control returns to
+the top C<docatch>. This then starts another third-level runops level,
+which executes the nextstate, pushmark and die ops on line 4. At the point
+that the second C<pp_die> is called, the C call stack looks exactly like
+that above, even though we are no longer within an inner eval; this is
+because of the optimization mentioned earlier. However, the context stack
+now looks like this, ie with the top CxEVAL popped:
+
+ STACK 0: MAIN
+ CX 0: BLOCK =>
+ CX 1: EVAL => AV() PV("A"\0)
+ retop=leave
+ STACK 1: MAGIC
+ CX 0: SUB =>
+ retop=(null)
+
+The die on line 4 pops the context stack back down to the CxEVAL, leaving
+it as:
+
+ STACK 0: MAIN
+ CX 0: BLOCK =>
+
+As usual, C<PL_restartop> is extracted from the C<CxEVAL>, and a
+C<JMPENV_JUMP(3)> done, which pops the C stack back to the docatch:
+
+ S_docatch
+ Perl_pp_entertry
+ Perl_runops # second loop
+ S_call_body
+ Perl_call_sv
+ Perl_pp_tie
+ Perl_runops # first loop
+ S_run_body
+ perl_run
+ main
+
+In this case, because the C<JMPENV> level recorded in the C<CxEVAL>
+differs from the current one, C<docatch> just does a C<JMPENV_JUMP(3)>
+and the C stack unwinds to:
+
+ perl_run
+ main
+
+Because C<PL_restartop> is non-null, C<run_body> starts a new runops loop
+and execution continues.
+
+=back
+
+=head2 Internal Variable Types
+
+You should by now have had a look at L<perlguts>, which tells you about
+Perl's internal variable types: SVs, HVs, AVs and the rest. If not, do
+that now.
+
+These variables are used not only to represent Perl-space variables, but
+also any constants in the code, as well as some structures completely
+internal to Perl. The symbol table, for instance, is an ordinary Perl
+hash. Your code is represented by an SV as it's read into the parser;
+any program files you call are opened via ordinary Perl filehandles, and
+so on.
+
+The core L<Devel::Peek|Devel::Peek> module lets us examine SVs from a
+Perl program. Let's see, for instance, how Perl treats the constant
+C<"hello">.
+
+ % perl -MDevel::Peek -e 'Dump("hello")'
+ 1 SV = PV(0xa041450) at 0xa04ecbc
+ 2 REFCNT = 1
+ 3 FLAGS = (POK,READONLY,pPOK)
+ 4 PV = 0xa0484e0 "hello"\0
+ 5 CUR = 5
+ 6 LEN = 6
+
+Reading C<Devel::Peek> output takes a bit of practise, so let's go
+through it line by line.
+
+Line 1 tells us we're looking at an SV which lives at C<0xa04ecbc> in
+memory. SVs themselves are very simple structures, but they contain a
+pointer to a more complex structure. In this case, it's a PV, a
+structure which holds a string value, at location C<0xa041450>. Line 2
+is the reference count; there are no other references to this data, so
+it's 1.
+
+Line 3 are the flags for this SV - it's OK to use it as a PV, it's a
+read-only SV (because it's a constant) and the data is a PV internally.
+Next we've got the contents of the string, starting at location
+C<0xa0484e0>.
+
+Line 5 gives us the current length of the string - note that this does
+B<not> include the null terminator. Line 6 is not the length of the
+string, but the length of the currently allocated buffer; as the string
+grows, Perl automatically extends the available storage via a routine
+called C<SvGROW>.
+
+You can get at any of these quantities from C very easily; just add
+C<Sv> to the name of the field shown in the snippet, and you've got a
+macro which will return the value: C<SvCUR(sv)> returns the current
+length of the string, C<SvREFCOUNT(sv)> returns the reference count,
+C<SvPV(sv, len)> returns the string itself with its length, and so on.
+More macros to manipulate these properties can be found in L<perlguts>.
+
+Let's take an example of manipulating a PV, from C<sv_catpvn>, in F<sv.c>
+
+ 1 void
+ 2 Perl_sv_catpvn(pTHX_ register SV *sv, register const char *ptr, register STRLEN len)
+ 3 {
+ 4 STRLEN tlen;
+ 5 char *junk;
+
+ 6 junk = SvPV_force(sv, tlen);
+ 7 SvGROW(sv, tlen + len + 1);
+ 8 if (ptr == junk)
+ 9 ptr = SvPVX(sv);
+ 10 Move(ptr,SvPVX(sv)+tlen,len,char);
+ 11 SvCUR(sv) += len;
+ 12 *SvEND(sv) = '\0';
+ 13 (void)SvPOK_only_UTF8(sv); /* validate pointer */
+ 14 SvTAINT(sv);
+ 15 }
+
+This is a function which adds a string, C<ptr>, of length C<len> onto
+the end of the PV stored in C<sv>. The first thing we do in line 6 is
+make sure that the SV B<has> a valid PV, by calling the C<SvPV_force>
+macro to force a PV. As a side effect, C<tlen> gets set to the current
+value of the PV, and the PV itself is returned to C<junk>.
+
+In line 7, we make sure that the SV will have enough room to accommodate
+the old string, the new string and the null terminator. If C<LEN> isn't
+big enough, C<SvGROW> will reallocate space for us.
+
+Now, if C<junk> is the same as the string we're trying to add, we can
+grab the string directly from the SV; C<SvPVX> is the address of the PV
+in the SV.
+
+Line 10 does the actual catenation: the C<Move> macro moves a chunk of
+memory around: we move the string C<ptr> to the end of the PV - that's
+the start of the PV plus its current length. We're moving C<len> bytes
+of type C<char>. After doing so, we need to tell Perl we've extended the
+string, by altering C<CUR> to reflect the new length. C<SvEND> is a
+macro which gives us the end of the string, so that needs to be a
+C<"\0">.
+
+Line 13 manipulates the flags; since we've changed the PV, any IV or NV
+values will no longer be valid: if we have C<$a=10; $a.="6";> we don't
+want to use the old IV of 10. C<SvPOK_only_utf8> is a special UTF-8-aware
+version of C<SvPOK_only>, a macro which turns off the IOK and NOK flags
+and turns on POK. The final C<SvTAINT> is a macro which launders tainted
+data if taint mode is turned on.
+
+AVs and HVs are more complicated, but SVs are by far the most common
+variable type being thrown around. Having seen something of how we
+manipulate these, let's go on and look at how the op tree is
+constructed.
+
+=head2 Op Trees
+
+First, what is the op tree, anyway? The op tree is the parsed
+representation of your program, as we saw in our section on parsing, and
+it's the sequence of operations that Perl goes through to execute your
+program, as we saw in L</Running>.
+
+An op is a fundamental operation that Perl can perform: all the built-in
+functions and operators are ops, and there are a series of ops which
+deal with concepts the interpreter needs internally - entering and
+leaving a block, ending a statement, fetching a variable, and so on.
+
+The op tree is connected in two ways: you can imagine that there are two
+"routes" through it, two orders in which you can traverse the tree.
+First, parse order reflects how the parser understood the code, and
+secondly, execution order tells perl what order to perform the
+operations in.
+
+The easiest way to examine the op tree is to stop Perl after it has
+finished parsing, and get it to dump out the tree. This is exactly what
+the compiler backends L<B::Terse|B::Terse>, L<B::Concise|B::Concise>
+and L<B::Debug|B::Debug> do.
+
+Let's have a look at how Perl sees C<$a = $b + $c>:
+
+ % perl -MO=Terse -e '$a=$b+$c'
+ 1 LISTOP (0x8179888) leave
+ 2 OP (0x81798b0) enter
+ 3 COP (0x8179850) nextstate
+ 4 BINOP (0x8179828) sassign
+ 5 BINOP (0x8179800) add [1]
+ 6 UNOP (0x81796e0) null [15]
+ 7 SVOP (0x80fafe0) gvsv GV (0x80fa4cc) *b
+ 8 UNOP (0x81797e0) null [15]
+ 9 SVOP (0x8179700) gvsv GV (0x80efeb0) *c
+ 10 UNOP (0x816b4f0) null [15]
+ 11 SVOP (0x816dcf0) gvsv GV (0x80fa460) *a
+
+Let's start in the middle, at line 4. This is a BINOP, a binary
+operator, which is at location C<0x8179828>. The specific operator in
+question is C<sassign> - scalar assignment - and you can find the code
+which implements it in the function C<pp_sassign> in F<pp_hot.c>. As a
+binary operator, it has two children: the add operator, providing the
+result of C<$b+$c>, is uppermost on line 5, and the left hand side is on
+line 10.
+
+Line 10 is the null op: this does exactly nothing. What is that doing
+there? If you see the null op, it's a sign that something has been
+optimized away after parsing. As we mentioned in L</Optimization>,
+the optimization stage sometimes converts two operations into one, for
+example when fetching a scalar variable. When this happens, instead of
+rewriting the op tree and cleaning up the dangling pointers, it's easier
+just to replace the redundant operation with the null op. Originally,
+the tree would have looked like this:
+
+ 10 SVOP (0x816b4f0) rv2sv [15]
+ 11 SVOP (0x816dcf0) gv GV (0x80fa460) *a
+
+That is, fetch the C<a> entry from the main symbol table, and then look
+at the scalar component of it: C<gvsv> (C<pp_gvsv> into F<pp_hot.c>)
+happens to do both these things.
+
+The right hand side, starting at line 5 is similar to what we've just
+seen: we have the C<add> op (C<pp_add> also in F<pp_hot.c>) add together
+two C<gvsv>s.
+
+Now, what's this about?
+
+ 1 LISTOP (0x8179888) leave
+ 2 OP (0x81798b0) enter
+ 3 COP (0x8179850) nextstate
+
+C<enter> and C<leave> are scoping ops, and their job is to perform any
+housekeeping every time you enter and leave a block: lexical variables
+are tidied up, unreferenced variables are destroyed, and so on. Every
+program will have those first three lines: C<leave> is a list, and its
+children are all the statements in the block. Statements are delimited
+by C<nextstate>, so a block is a collection of C<nextstate> ops, with
+the ops to be performed for each statement being the children of
+C<nextstate>. C<enter> is a single op which functions as a marker.
+
+That's how Perl parsed the program, from top to bottom:
+
+ Program
+ |
+ Statement
+ |
+ =
+ / \
+ / \
+ $a +
+ / \
+ $b $c
+
+However, it's impossible to B<perform> the operations in this order:
+you have to find the values of C<$b> and C<$c> before you add them
+together, for instance. So, the other thread that runs through the op
+tree is the execution order: each op has a field C<op_next> which points
+to the next op to be run, so following these pointers tells us how perl
+executes the code. We can traverse the tree in this order using
+the C<exec> option to C<B::Terse>:
+
+ % perl -MO=Terse,exec -e '$a=$b+$c'
+ 1 OP (0x8179928) enter
+ 2 COP (0x81798c8) nextstate
+ 3 SVOP (0x81796c8) gvsv GV (0x80fa4d4) *b
+ 4 SVOP (0x8179798) gvsv GV (0x80efeb0) *c
+ 5 BINOP (0x8179878) add [1]
+ 6 SVOP (0x816dd38) gvsv GV (0x80fa468) *a
+ 7 BINOP (0x81798a0) sassign
+ 8 LISTOP (0x8179900) leave
+
+This probably makes more sense for a human: enter a block, start a
+statement. Get the values of C<$b> and C<$c>, and add them together.
+Find C<$a>, and assign one to the other. Then leave.
+
+The way Perl builds up these op trees in the parsing process can be
+unravelled by examining F<perly.y>, the YACC grammar. Let's take the
+piece we need to construct the tree for C<$a = $b + $c>
+
+ 1 term : term ASSIGNOP term
+ 2 { $$ = newASSIGNOP(OPf_STACKED, $1, $2, $3); }
+ 3 | term ADDOP term
+ 4 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); }
+
+If you're not used to reading BNF grammars, this is how it works: You're
+fed certain things by the tokeniser, which generally end up in upper
+case. Here, C<ADDOP>, is provided when the tokeniser sees C<+> in your
+code. C<ASSIGNOP> is provided when C<=> is used for assigning. These are
+"terminal symbols", because you can't get any simpler than them.
+
+The grammar, lines one and three of the snippet above, tells you how to
+build up more complex forms. These complex forms, "non-terminal symbols"
+are generally placed in lower case. C<term> here is a non-terminal
+symbol, representing a single expression.
+
+The grammar gives you the following rule: you can make the thing on the
+left of the colon if you see all the things on the right in sequence.
+This is called a "reduction", and the aim of parsing is to completely
+reduce the input. There are several different ways you can perform a
+reduction, separated by vertical bars: so, C<term> followed by C<=>
+followed by C<term> makes a C<term>, and C<term> followed by C<+>
+followed by C<term> can also make a C<term>.
+
+So, if you see two terms with an C<=> or C<+>, between them, you can
+turn them into a single expression. When you do this, you execute the
+code in the block on the next line: if you see C<=>, you'll do the code
+in line 2. If you see C<+>, you'll do the code in line 4. It's this code
+which contributes to the op tree.
+
+ | term ADDOP term
+ { $$ = newBINOP($2, 0, scalar($1), scalar($3)); }
+
+What this does is creates a new binary op, and feeds it a number of
+variables. The variables refer to the tokens: C<$1> is the first token in
+the input, C<$2> the second, and so on - think regular expression
+backreferences. C<$$> is the op returned from this reduction. So, we
+call C<newBINOP> to create a new binary operator. The first parameter to
+C<newBINOP>, a function in F<op.c>, is the op type. It's an addition
+operator, so we want the type to be C<ADDOP>. We could specify this
+directly, but it's right there as the second token in the input, so we
+use C<$2>. The second parameter is the op's flags: 0 means "nothing
+special". Then the things to add: the left and right hand side of our
+expression, in scalar context.
+
+=head2 Stacks
+
+When perl executes something like C<addop>, how does it pass on its
+results to the next op? The answer is, through the use of stacks. Perl
+has a number of stacks to store things it's currently working on, and
+we'll look at the three most important ones here.
+
+=over 3
+
+=item Argument stack
+
+Arguments are passed to PP code and returned from PP code using the
+argument stack, C<ST>. The typical way to handle arguments is to pop
+them off the stack, deal with them how you wish, and then push the result
+back onto the stack. This is how, for instance, the cosine operator
+works:
+
+ NV value;
+ value = POPn;
+ value = Perl_cos(value);
+ XPUSHn(value);
+
+We'll see a more tricky example of this when we consider Perl's macros
+below. C<POPn> gives you the NV (floating point value) of the top SV on
+the stack: the C<$x> in C<cos($x)>. Then we compute the cosine, and push
+the result back as an NV. The C<X> in C<XPUSHn> means that the stack
+should be extended if necessary - it can't be necessary here, because we
+know there's room for one more item on the stack, since we've just
+removed one! The C<XPUSH*> macros at least guarantee safety.
+
+Alternatively, you can fiddle with the stack directly: C<SP> gives you
+the first element in your portion of the stack, and C<TOP*> gives you
+the top SV/IV/NV/etc. on the stack. So, for instance, to do unary
+negation of an integer:
+
+ SETi(-TOPi);
+
+Just set the integer value of the top stack entry to its negation.
+
+Argument stack manipulation in the core is exactly the same as it is in
+XSUBs - see L<perlxstut>, L<perlxs> and L<perlguts> for a longer
+description of the macros used in stack manipulation.
+
+=item Mark stack
+
+I say "your portion of the stack" above because PP code doesn't
+necessarily get the whole stack to itself: if your function calls
+another function, you'll only want to expose the arguments aimed for the
+called function, and not (necessarily) let it get at your own data. The
+way we do this is to have a "virtual" bottom-of-stack, exposed to each
+function. The mark stack keeps bookmarks to locations in the argument
+stack usable by each function. For instance, when dealing with a tied
+variable, (internally, something with "P" magic) Perl has to call
+methods for accesses to the tied variables. However, we need to separate
+the arguments exposed to the method to the argument exposed to the
+original function - the store or fetch or whatever it may be. Here's
+roughly how the tied C<push> is implemented; see C<av_push> in F<av.c>:
+
+ 1 PUSHMARK(SP);
+ 2 EXTEND(SP,2);
+ 3 PUSHs(SvTIED_obj((SV*)av, mg));
+ 4 PUSHs(val);
+ 5 PUTBACK;
+ 6 ENTER;
+ 7 call_method("PUSH", G_SCALAR|G_DISCARD);
+ 8 LEAVE;
+
+Let's examine the whole implementation, for practice:
+
+ 1 PUSHMARK(SP);
+
+Push the current state of the stack pointer onto the mark stack. This is
+so that when we've finished adding items to the argument stack, Perl
+knows how many things we've added recently.
+
+ 2 EXTEND(SP,2);
+ 3 PUSHs(SvTIED_obj((SV*)av, mg));
+ 4 PUSHs(val);
+
+We're going to add two more items onto the argument stack: when you have
+a tied array, the C<PUSH> subroutine receives the object and the value
+to be pushed, and that's exactly what we have here - the tied object,
+retrieved with C<SvTIED_obj>, and the value, the SV C<val>.
+
+ 5 PUTBACK;
+
+Next we tell Perl to update the global stack pointer from our internal
+variable: C<dSP> only gave us a local copy, not a reference to the global.
+
+ 6 ENTER;
+ 7 call_method("PUSH", G_SCALAR|G_DISCARD);
+ 8 LEAVE;
+
+C<ENTER> and C<LEAVE> localise a block of code - they make sure that all
+variables are tidied up, everything that has been localised gets
+its previous value returned, and so on. Think of them as the C<{> and
+C<}> of a Perl block.
+
+To actually do the magic method call, we have to call a subroutine in
+Perl space: C<call_method> takes care of that, and it's described in
+L<perlcall>. We call the C<PUSH> method in scalar context, and we're
+going to discard its return value. The call_method() function
+removes the top element of the mark stack, so there is nothing for
+the caller to clean up.
+
+=item Save stack
+
+C doesn't have a concept of local scope, so perl provides one. We've
+seen that C<ENTER> and C<LEAVE> are used as scoping braces; the save
+stack implements the C equivalent of, for example:
+
+ {
+ local $foo = 42;
+ ...
+ }
+
+See L<perlguts/Localising Changes> for how to use the save stack.
+
+=back
+
+=head2 Millions of Macros
+
+One thing you'll notice about the Perl source is that it's full of
+macros. Some have called the pervasive use of macros the hardest thing
+to understand, others find it adds to clarity. Let's take an example,
+the code which implements the addition operator:
+
+ 1 PP(pp_add)
+ 2 {
+ 3 dSP; dATARGET; tryAMAGICbin(add,opASSIGN);
+ 4 {
+ 5 dPOPTOPnnrl_ul;
+ 6 SETn( left + right );
+ 7 RETURN;
+ 8 }
+ 9 }
+
+Every line here (apart from the braces, of course) contains a macro. The
+first line sets up the function declaration as Perl expects for PP code;
+line 3 sets up variable declarations for the argument stack and the
+target, the return value of the operation. Finally, it tries to see if
+the addition operation is overloaded; if so, the appropriate subroutine
+is called.
+
+Line 5 is another variable declaration - all variable declarations start
+with C<d> - which pops from the top of the argument stack two NVs (hence
+C<nn>) and puts them into the variables C<right> and C<left>, hence the
+C<rl>. These are the two operands to the addition operator. Next, we
+call C<SETn> to set the NV of the return value to the result of adding
+the two values. This done, we return - the C<RETURN> macro makes sure
+that our return value is properly handled, and we pass the next operator
+to run back to the main run loop.
+
+Most of these macros are explained in L<perlapi>, and some of the more
+important ones are explained in L<perlxs> as well. Pay special attention
+to L<perlguts/Background and PERL_IMPLICIT_CONTEXT> for information on
+the C<[pad]THX_?> macros.
+
+=head2 The .i Targets
+
+You can expand the macros in a F<foo.c> file by saying
+
+ make foo.i
+
+which will expand the macros using cpp. Don't be scared by the results.
+
+=head1 SOURCE CODE STATIC ANALYSIS
+
+Various tools exist for analysing C source code B<statically>, as
+opposed to B<dynamically>, that is, without executing the code.
+It is possible to detect resource leaks, undefined behaviour, type
+mismatches, portability problems, code paths that would cause illegal
+memory accesses, and other similar problems by just parsing the C code
+and looking at the resulting graph, what does it tell about the
+execution and data flows. As a matter of fact, this is exactly
+how C compilers know to give warnings about dubious code.
+
+=head2 lint, splint
+
+The good old C code quality inspector, C<lint>, is available in
+several platforms, but please be aware that there are several
+different implementations of it by different vendors, which means that
+the flags are not identical across different platforms.
+
+There is a lint variant called C<splint> (Secure Programming Lint)
+available from http://www.splint.org/ that should compile on any
+Unix-like platform.
+
+There are C<lint> and <splint> targets in Makefile, but you may have
+to diddle with the flags (see above).
+
+=head2 Coverity
+
+Coverity (http://www.coverity.com/) is a product similar to lint and
+as a testbed for their product they periodically check several open
+source projects, and they give out accounts to open source developers
+to the defect databases.
+
+=head2 cpd (cut-and-paste detector)
+
+The cpd tool detects cut-and-paste coding. If one instance of the
+cut-and-pasted code changes, all the other spots should probably be
+changed, too. Therefore such code should probably be turned into a
+subroutine or a macro.
+
+cpd (http://pmd.sourceforge.net/cpd.html) is part of the pmd project
+(http://pmd.sourceforge.net/). pmd was originally written for static
+analysis of Java code, but later the cpd part of it was extended to
+parse also C and C++.
+
+Download the pmd-bin-X.Y.zip () from the SourceForge site, extract the
+pmd-X.Y.jar from it, and then run that on source code thusly:
+
+ java -cp pmd-X.Y.jar net.sourceforge.pmd.cpd.CPD --minimum-tokens 100 --files /some/where/src --language c > cpd.txt
+
+You may run into memory limits, in which case you should use the -Xmx option:
+
+ java -Xmx512M ...
+
+=head2 gcc warnings
+
+Though much can be written about the inconsistency and coverage
+problems of gcc warnings (like C<-Wall> not meaning "all the
+warnings", or some common portability problems not being covered by
+C<-Wall>, or C<-ansi> and C<-pedantic> both being a poorly defined
+collection of warnings, and so forth), gcc is still a useful tool in
+keeping our coding nose clean.
+
+The C<-Wall> is by default on.
+
+The C<-ansi> (and its sidekick, C<-pedantic>) would be nice to be on
+always, but unfortunately they are not safe on all platforms, they can
+for example cause fatal conflicts with the system headers (Solaris
+being a prime example). If Configure C<-Dgccansipedantic> is used,
+the C<cflags> frontend selects C<-ansi -pedantic> for the platforms
+where they are known to be safe.
+
+Starting from Perl 5.9.4 the following extra flags are added:
+
+=over 4
+
+=item *
+
+C<-Wendif-labels>
+
+=item *
+
+C<-Wextra>
+
+=item *
+
+C<-Wdeclaration-after-statement>
+
+=back
+
+The following flags would be nice to have but they would first need
+their own Augean stablemaster:
+
+=over 4
+
+=item *
+
+C<-Wpointer-arith>
+
+=item *
+
+C<-Wshadow>
+
+=item *
+
+C<-Wstrict-prototypes>
+
+=back
+
+The C<-Wtraditional> is another example of the annoying tendency of
+gcc to bundle a lot of warnings under one switch (it would be
+impossible to deploy in practice because it would complain a lot) but
+it does contain some warnings that would be beneficial to have available
+on their own, such as the warning about string constants inside macros
+containing the macro arguments: this behaved differently pre-ANSI
+than it does in ANSI, and some C compilers are still in transition,
+AIX being an example.
+
+=head2 Warnings of other C compilers
+
+Other C compilers (yes, there B<are> other C compilers than gcc) often
+have their "strict ANSI" or "strict ANSI with some portability extensions"
+modes on, like for example the Sun Workshop has its C<-Xa> mode on
+(though implicitly), or the DEC (these days, HP...) has its C<-std1>
+mode on.
+
+=head2 DEBUGGING
+
+You can compile a special debugging version of Perl, which allows you
+to use the C<-D> option of Perl to tell more about what Perl is doing.
+But sometimes there is no alternative than to dive in with a debugger,
+either to see the stack trace of a core dump (very useful in a bug
+report), or trying to figure out what went wrong before the core dump
+happened, or how did we end up having wrong or unexpected results.
+
+=head2 Poking at Perl
+
+To really poke around with Perl, you'll probably want to build Perl for
+debugging, like this:
+
+ ./Configure -d -D optimize=-g
+ make
+
+C<-g> is a flag to the C compiler to have it produce debugging
+information which will allow us to step through a running program,
+and to see in which C function we are at (without the debugging
+information we might see only the numerical addresses of the functions,
+which is not very helpful).
+
+F<Configure> will also turn on the C<DEBUGGING> compilation symbol which
+enables all the internal debugging code in Perl. There are a whole bunch
+of things you can debug with this: L<perlrun> lists them all, and the
+best way to find out about them is to play about with them. The most
+useful options are probably
+
+ l Context (loop) stack processing
+ t Trace execution
+ o Method and overloading resolution
+ c String/numeric conversions
+
+Some of the functionality of the debugging code can be achieved using XS
+modules.
+
+ -Dr => use re 'debug'
+ -Dx => use O 'Debug'
+
+=head2 Using a source-level debugger
+
+If the debugging output of C<-D> doesn't help you, it's time to step
+through perl's execution with a source-level debugger.
+
+=over 3
+
+=item *
+
+We'll use C<gdb> for our examples here; the principles will apply to
+any debugger (many vendors call their debugger C<dbx>), but check the
+manual of the one you're using.
+
+=back
+
+To fire up the debugger, type
+
+ gdb ./perl
+
+Or if you have a core dump:
+
+ gdb ./perl core
+
+You'll want to do that in your Perl source tree so the debugger can read
+the source code. You should see the copyright message, followed by the
+prompt.
+
+ (gdb)
+
+C<help> will get you into the documentation, but here are the most
+useful commands:
+
+=over 3
+
+=item run [args]
+
+Run the program with the given arguments.
+
+=item break function_name
+
+=item break source.c:xxx
+
+Tells the debugger that we'll want to pause execution when we reach
+either the named function (but see L<perlguts/Internal Functions>!) or the given
+line in the named source file.
+
+=item step
+
+Steps through the program a line at a time.
+
+=item next
+
+Steps through the program a line at a time, without descending into
+functions.
+
+=item continue
+
+Run until the next breakpoint.
+
+=item finish
+
+Run until the end of the current function, then stop again.
+
+=item 'enter'
+
+Just pressing Enter will do the most recent operation again - it's a
+blessing when stepping through miles of source code.
+
+=item print
+
+Execute the given C code and print its results. B<WARNING>: Perl makes
+heavy use of macros, and F<gdb> does not necessarily support macros
+(see later L</"gdb macro support">). You'll have to substitute them
+yourself, or to invoke cpp on the source code files
+(see L</"The .i Targets">)
+So, for instance, you can't say
+
+ print SvPV_nolen(sv)
+
+but you have to say
+
+ print Perl_sv_2pv_nolen(sv)
+
+=back
+
+You may find it helpful to have a "macro dictionary", which you can
+produce by saying C<cpp -dM perl.c | sort>. Even then, F<cpp> won't
+recursively apply those macros for you.
+
+=head2 gdb macro support
+
+Recent versions of F<gdb> have fairly good macro support, but
+in order to use it you'll need to compile perl with macro definitions
+included in the debugging information. Using F<gcc> version 3.1, this
+means configuring with C<-Doptimize=-g3>. Other compilers might use a
+different switch (if they support debugging macros at all).
+
+=head2 Dumping Perl Data Structures
+
+One way to get around this macro hell is to use the dumping functions in
+F<dump.c>; these work a little like an internal
+L<Devel::Peek|Devel::Peek>, but they also cover OPs and other structures
+that you can't get at from Perl. Let's take an example. We'll use the
+C<$a = $b + $c> we used before, but give it a bit of context:
+C<$b = "6XXXX"; $c = 2.3;>. Where's a good place to stop and poke around?
+
+What about C<pp_add>, the function we examined earlier to implement the
+C<+> operator:
+
+ (gdb) break Perl_pp_add
+ Breakpoint 1 at 0x46249f: file pp_hot.c, line 309.
+
+Notice we use C<Perl_pp_add> and not C<pp_add> - see L<perlguts/Internal Functions>.
+With the breakpoint in place, we can run our program:
+
+ (gdb) run -e '$b = "6XXXX"; $c = 2.3; $a = $b + $c'
+
+Lots of junk will go past as gdb reads in the relevant source files and
+libraries, and then:
+
+ Breakpoint 1, Perl_pp_add () at pp_hot.c:309
+ 309 dSP; dATARGET; tryAMAGICbin(add,opASSIGN);
+ (gdb) step
+ 311 dPOPTOPnnrl_ul;
+ (gdb)
+
+We looked at this bit of code before, and we said that C<dPOPTOPnnrl_ul>
+arranges for two C<NV>s to be placed into C<left> and C<right> - let's
+slightly expand it:
+
+ #define dPOPTOPnnrl_ul NV right = POPn; \
+ SV *leftsv = TOPs; \
+ NV left = USE_LEFT(leftsv) ? SvNV(leftsv) : 0.0
+
+C<POPn> takes the SV from the top of the stack and obtains its NV either
+directly (if C<SvNOK> is set) or by calling the C<sv_2nv> function.
+C<TOPs> takes the next SV from the top of the stack - yes, C<POPn> uses
+C<TOPs> - but doesn't remove it. We then use C<SvNV> to get the NV from
+C<leftsv> in the same way as before - yes, C<POPn> uses C<SvNV>.
+
+Since we don't have an NV for C<$b>, we'll have to use C<sv_2nv> to
+convert it. If we step again, we'll find ourselves there:
+
+ Perl_sv_2nv (sv=0xa0675d0) at sv.c:1669
+ 1669 if (!sv)
+ (gdb)
+
+We can now use C<Perl_sv_dump> to investigate the SV:
+
+ SV = PV(0xa057cc0) at 0xa0675d0
+ REFCNT = 1
+ FLAGS = (POK,pPOK)
+ PV = 0xa06a510 "6XXXX"\0
+ CUR = 5
+ LEN = 6
+ $1 = void
+
+We know we're going to get C<6> from this, so let's finish the
+subroutine:
+
+ (gdb) finish
+ Run till exit from #0 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1671
+ 0x462669 in Perl_pp_add () at pp_hot.c:311
+ 311 dPOPTOPnnrl_ul;
+
+We can also dump out this op: the current op is always stored in
+C<PL_op>, and we can dump it with C<Perl_op_dump>. This'll give us
+similar output to L<B::Debug|B::Debug>.
+
+ {
+ 13 TYPE = add ===> 14
+ TARG = 1
+ FLAGS = (SCALAR,KIDS)
+ {
+ TYPE = null ===> (12)
+ (was rv2sv)
+ FLAGS = (SCALAR,KIDS)
+ {
+ 11 TYPE = gvsv ===> 12
+ FLAGS = (SCALAR)
+ GV = main::b
+ }
+ }
+
+# finish this later #
+
+=head2 Patching
+
+All right, we've now had a look at how to navigate the Perl sources and
+some things you'll need to know when fiddling with them. Let's now get
+on and create a simple patch. Here's something Larry suggested: if a
+C<U> is the first active format during a C<pack>, (for example,
+C<pack "U3C8", @stuff>) then the resulting string should be treated as
+UTF-8 encoded.
+
+If you are working with a git clone of the Perl repository, you will want to
+create a branch for your changes. This will make creating a proper patch much
+simpler. See the L<perlrepository> for details on how to do this.
+
+How do we prepare to fix this up? First we locate the code in question -
+the C<pack> happens at runtime, so it's going to be in one of the F<pp>
+files. Sure enough, C<pp_pack> is in F<pp.c>. Since we're going to be
+altering this file, let's copy it to F<pp.c~>.
+
+[Well, it was in F<pp.c> when this tutorial was written. It has now been
+split off with C<pp_unpack> to its own file, F<pp_pack.c>]
+
+Now let's look over C<pp_pack>: we take a pattern into C<pat>, and then
+loop over the pattern, taking each format character in turn into
+C<datum_type>. Then for each possible format character, we swallow up
+the other arguments in the pattern (a field width, an asterisk, and so
+on) and convert the next chunk input into the specified format, adding
+it onto the output SV C<cat>.
+
+How do we know if the C<U> is the first format in the C<pat>? Well, if
+we have a pointer to the start of C<pat> then, if we see a C<U> we can
+test whether we're still at the start of the string. So, here's where
+C<pat> is set up:
+
+ STRLEN fromlen;
+ register char *pat = SvPVx(*++MARK, fromlen);
+ register char *patend = pat + fromlen;
+ register I32 len;
+ I32 datumtype;
+ SV *fromstr;
+
+We'll have another string pointer in there:
+
+ STRLEN fromlen;
+ register char *pat = SvPVx(*++MARK, fromlen);
+ register char *patend = pat + fromlen;
+ + char *patcopy;
+ register I32 len;
+ I32 datumtype;
+ SV *fromstr;
+
+And just before we start the loop, we'll set C<patcopy> to be the start
+of C<pat>:
+
+ items = SP - MARK;
+ MARK++;
+ sv_setpvn(cat, "", 0);
+ + patcopy = pat;
+ while (pat < patend) {
+
+Now if we see a C<U> which was at the start of the string, we turn on
+the C<UTF8> flag for the output SV, C<cat>:
+
+ + if (datumtype == 'U' && pat==patcopy+1)
+ + SvUTF8_on(cat);
+ if (datumtype == '#') {
+ while (pat < patend && *pat != '\n')
+ pat++;
+
+Remember that it has to be C<patcopy+1> because the first character of
+the string is the C<U> which has been swallowed into C<datumtype!>
+
+Oops, we forgot one thing: what if there are spaces at the start of the
+pattern? C<pack(" U*", @stuff)> will have C<U> as the first active
+character, even though it's not the first thing in the pattern. In this
+case, we have to advance C<patcopy> along with C<pat> when we see spaces:
+
+ if (isSPACE(datumtype))
+ continue;
+
+needs to become
+
+ if (isSPACE(datumtype)) {
+ patcopy++;
+ continue;
+ }
+
+OK. That's the C part done. Now we must do two additional things before
+this patch is ready to go: we've changed the behaviour of Perl, and so
+we must document that change. We must also provide some more regression
+tests to make sure our patch works and doesn't create a bug somewhere
+else along the line.
+
+The regression tests for each operator live in F<t/op/>, and so we
+make a copy of F<t/op/pack.t> to F<t/op/pack.t~>. Now we can add our
+tests to the end. First, we'll test that the C<U> does indeed create
+Unicode strings.
+
+t/op/pack.t has a sensible ok() function, but if it didn't we could
+use the one from t/test.pl.
+
+ require './test.pl';
+ plan( tests => 159 );
+
+so instead of this:
+
+ print 'not ' unless "1.20.300.4000" eq sprintf "%vd",
+ pack("U*",1,20,300,4000);
+ print "ok $test\n"; $test++;
+
+we can write the more sensible (see L<Test::More> for a full
+explanation of is() and other testing functions).
+
+ is( "1.20.300.4000", sprintf "%vd", pack("U*",1,20,300,4000),
+ "U* produces Unicode" );
+
+Now we'll test that we got that space-at-the-beginning business right:
+
+ is( "1.20.300.4000", sprintf "%vd", pack(" U*",1,20,300,4000),
+ " with spaces at the beginning" );
+
+And finally we'll test that we don't make Unicode strings if C<U> is B<not>
+the first active format:
+
+ isnt( v1.20.300.4000, sprintf "%vd", pack("C0U*",1,20,300,4000),
+ "U* not first isn't Unicode" );
+
+Mustn't forget to change the number of tests which appears at the top,
+or else the automated tester will get confused. This will either look
+like this:
+
+ print "1..156\n";
+
+or this:
+
+ plan( tests => 156 );
+
+We now compile up Perl, and run it through the test suite. Our new
+tests pass, hooray!
+
+Finally, the documentation. The job is never done until the paperwork is
+over, so let's describe the change we've just made. The relevant place
+is F<pod/perlfunc.pod>; again, we make a copy, and then we'll insert
+this text in the description of C<pack>:
+
+ =item *
+
+ If the pattern begins with a C<U>, the resulting string will be treated
+ as UTF-8-encoded Unicode. You can force UTF-8 encoding on in a string
+ with an initial C<U0>, and the bytes that follow will be interpreted as
+ Unicode characters. If you don't want this to happen, you can begin
+ your pattern with C<C0> (or anything else) to force Perl not to UTF-8
+ encode your string, and then follow this with a C<U*> somewhere in your
+ pattern.
+
+=head2 Patching a core module
+
+This works just like patching anything else, with an extra
+consideration. Many core modules also live on CPAN. If this is so,
+patch the CPAN version instead of the core and send the patch off to
+the module maintainer (with a copy to p5p). This will help the module
+maintainer keep the CPAN version in sync with the core version without
+constantly scanning p5p.
+
+The list of maintainers of core modules is usefully documented in
+F<Porting/Maintainers.pl>.
+
+=head2 Adding a new function to the core
+
+If, as part of a patch to fix a bug, or just because you have an
+especially good idea, you decide to add a new function to the core,
+discuss your ideas on p5p well before you start work. It may be that
+someone else has already attempted to do what you are considering and
+can give lots of good advice or even provide you with bits of code
+that they already started (but never finished).
+
+You have to follow all of the advice given above for patching. It is
+extremely important to test any addition thoroughly and add new tests
+to explore all boundary conditions that your new function is expected
+to handle. If your new function is used only by one module (e.g. toke),
+then it should probably be named S_your_function (for static); on the
+other hand, if you expect it to accessible from other functions in
+Perl, you should name it Perl_your_function. See L<perlguts/Internal Functions>
+for more details.
+
+The location of any new code is also an important consideration. Don't
+just create a new top level .c file and put your code there; you would
+have to make changes to Configure (so the Makefile is created properly),
+as well as possibly lots of include files. This is strictly pumpking
+business.
+
+It is better to add your function to one of the existing top level
+source code files, but your choice is complicated by the nature of
+the Perl distribution. Only the files that are marked as compiled
+static are located in the perl executable. Everything else is located
+in the shared library (or DLL if you are running under WIN32). So,
+for example, if a function was only used by functions located in
+toke.c, then your code can go in toke.c. If, however, you want to call
+the function from universal.c, then you should put your code in another
+location, for example util.c.
+
+In addition to writing your c-code, you will need to create an
+appropriate entry in embed.pl describing your function, then run
+'make regen_headers' to create the entries in the numerous header
+files that perl needs to compile correctly. See L<perlguts/Internal Functions>
+for information on the various options that you can set in embed.pl.
+You will forget to do this a few (or many) times and you will get
+warnings during the compilation phase. Make sure that you mention
+this when you post your patch to P5P; the pumpking needs to know this.
+
+When you write your new code, please be conscious of existing code
+conventions used in the perl source files. See L<perlstyle> for
+details. Although most of the guidelines discussed seem to focus on
+Perl code, rather than c, they all apply (except when they don't ;).
+Also see L<perlrepository> for lots of details about both formatting and
+submitting patches of your changes.
+
+Lastly, TEST TEST TEST TEST TEST any code before posting to p5p.
+Test on as many platforms as you can find. Test as many perl
+Configure options as you can (e.g. MULTIPLICITY). If you have
+profiling or memory tools, see L<EXTERNAL TOOLS FOR DEBUGGING PERL>
+below for how to use them to further test your code. Remember that
+most of the people on P5P are doing this on their own time and
+don't have the time to debug your code.
+
+=head2 Writing a test
+
+Every module and built-in function has an associated test file (or
+should...). If you add or change functionality, you have to write a
+test. If you fix a bug, you have to write a test so that bug never
+comes back. If you alter the docs, it would be nice to test what the
+new documentation says.
+
+In short, if you submit a patch you probably also have to patch the
+tests.
+
+For modules, the test file is right next to the module itself.
+F<lib/strict.t> tests F<lib/strict.pm>. This is a recent innovation,
+so there are some snags (and it would be wonderful for you to brush
+them out), but it basically works that way. Everything else lives in
+F<t/>.
+
+If you add a new test directory under F<t/>, it is imperative that you
+add that directory to F<t/HARNESS> and F<t/TEST>.
+
+=over 3
+
+=item F<t/base/>
+
+Testing of the absolute basic functionality of Perl. Things like
+C<if>, basic file reads and writes, simple regexes, etc. These are
+run first in the test suite and if any of them fail, something is
+I<really> broken.
+
+=item F<t/cmd/>
+
+These test the basic control structures, C<if/else>, C<while>,
+subroutines, etc.
+
+=item F<t/comp/>
+
+Tests basic issues of how Perl parses and compiles itself.
+
+=item F<t/io/>
+
+Tests for built-in IO functions, including command line arguments.
+
+=item F<t/lib/>
+
+The old home for the module tests, you shouldn't put anything new in
+here. There are still some bits and pieces hanging around in here
+that need to be moved. Perhaps you could move them? Thanks!
+
+=item F<t/mro/>
+
+Tests for perl's method resolution order implementations
+(see L<mro>).
+
+=item F<t/op/>
+
+Tests for perl's built in functions that don't fit into any of the
+other directories.
+
+=item F<t/re/>
+
+Tests for regex related functions or behaviour. (These used to live
+in t/op).
+
+=item F<t/run/>
+
+Testing features of how perl actually runs, including exit codes and
+handling of PERL* environment variables.
+
+=item F<t/uni/>
+
+Tests for the core support of Unicode.
+
+=item F<t/win32/>
+
+Windows-specific tests.
+
+=item F<t/x2p>
+
+A test suite for the s2p converter.
+
+=back
+
+The core uses the same testing style as the rest of Perl, a simple
+"ok/not ok" run through Test::Harness, but there are a few special
+considerations.
+
+There are three ways to write a test in the core. Test::More,
+t/test.pl and ad hoc C<print $test ? "ok 42\n" : "not ok 42\n">. The
+decision of which to use depends on what part of the test suite you're
+working on. This is a measure to prevent a high-level failure (such
+as Config.pm breaking) from causing basic functionality tests to fail.
+If you write your own test, use the L<Test Anything Protocol|TAP>.
+
+=over 4
+
+=item t/base t/comp
+
+Since we don't know if require works, or even subroutines, use ad hoc
+tests for these two. Step carefully to avoid using the feature being
+tested.
+
+=item t/cmd t/run t/io t/op
+
+Now that basic require() and subroutines are tested, you can use the
+t/test.pl library which emulates the important features of Test::More
+while using a minimum of core features.
+
+You can also conditionally use certain libraries like Config, but be
+sure to skip the test gracefully if it's not there.
+
+=item t/lib ext lib
+
+Now that the core of Perl is tested, Test::More can be used. You can
+also use the full suite of core modules in the tests.
+
+=back
+
+When you say "make test" Perl uses the F<t/TEST> program to run the
+test suite (except under Win32 where it uses F<t/harness> instead.)
+All tests are run from the F<t/> directory, B<not> the directory
+which contains the test. This causes some problems with the tests
+in F<lib/>, so here's some opportunity for some patching.
+
+You must be triply conscious of cross-platform concerns. This usually
+boils down to using File::Spec and avoiding things like C<fork()> and
+C<system()> unless absolutely necessary.
+
+=head2 Special Make Test Targets
+
+There are various special make targets that can be used to test Perl
+slightly differently than the standard "test" target. Not all them
+are expected to give a 100% success rate. Many of them have several
+aliases, and many of them are not available on certain operating
+systems.
+
+=over 4
+
+=item coretest
+
+Run F<perl> on all core tests (F<t/*> and F<lib/[a-z]*> pragma tests).
+
+(Not available on Win32)
+
+=item test.deparse
+
+Run all the tests through B::Deparse. Not all tests will succeed.
+
+(Not available on Win32)
+
+=item test.taintwarn
+
+Run all tests with the B<-t> command-line switch. Not all tests
+are expected to succeed (until they're specifically fixed, of course).
+
+(Not available on Win32)
+
+=item minitest
+
+Run F<miniperl> on F<t/base>, F<t/comp>, F<t/cmd>, F<t/run>, F<t/io>,
+F<t/op>, F<t/uni> and F<t/mro> tests.
+
+=item test.valgrind check.valgrind utest.valgrind ucheck.valgrind
+
+(Only in Linux) Run all the tests using the memory leak + naughty
+memory access tool "valgrind". The log files will be named
+F<testname.valgrind>.
+
+=item test.third check.third utest.third ucheck.third
+
+(Only in Tru64) Run all the tests using the memory leak + naughty
+memory access tool "Third Degree". The log files will be named
+F<perl.3log.testname>.
+
+=item test.torture torturetest
+
+Run all the usual tests and some extra tests. As of Perl 5.8.0 the
+only extra tests are Abigail's JAPHs, F<t/japh/abigail.t>.
+
+You can also run the torture test with F<t/harness> by giving
+C<-torture> argument to F<t/harness>.
+
+=item utest ucheck test.utf8 check.utf8
+
+Run all the tests with -Mutf8. Not all tests will succeed.
+
+(Not available on Win32)
+
+=item minitest.utf16 test.utf16
+
+Runs the tests with UTF-16 encoded scripts, encoded with different
+versions of this encoding.
+
+C<make utest.utf16> runs the test suite with a combination of C<-utf8> and
+C<-utf16> arguments to F<t/TEST>.
+
+(Not available on Win32)
+
+=item test_harness
+
+Run the test suite with the F<t/harness> controlling program, instead of
+F<t/TEST>. F<t/harness> is more sophisticated, and uses the
+L<Test::Harness> module, thus using this test target supposes that perl
+mostly works. The main advantage for our purposes is that it prints a
+detailed summary of failed tests at the end. Also, unlike F<t/TEST>, it
+doesn't redirect stderr to stdout.
+
+Note that under Win32 F<t/harness> is always used instead of F<t/TEST>, so
+there is no special "test_harness" target.
+
+Under Win32's "test" target you may use the TEST_SWITCHES and TEST_FILES
+environment variables to control the behaviour of F<t/harness>. This means
+you can say
+
+ nmake test TEST_FILES="op/*.t"
+ nmake test TEST_SWITCHES="-torture" TEST_FILES="op/*.t"
+
+=item Parallel tests
+
+The core distribution can now run its regression tests in parallel on
+Unix-like platforms. Instead of running C<make test>, set C<TEST_JOBS> in
+your environment to the number of tests to run in parallel, and run
+C<make test_harness>. On a Bourne-like shell, this can be done as
+
+ TEST_JOBS=3 make test_harness # Run 3 tests in parallel
+
+An environment variable is used, rather than parallel make itself, because
+L<TAP::Harness> needs to be able to schedule individual non-conflicting test
+scripts itself, and there is no standard interface to C<make> utilities to
+interact with their job schedulers.
+
+Note that currently some test scripts may fail when run in parallel (most
+notably C<ext/IO/t/io_dir.t>). If necessary run just the failing scripts
+again sequentially and see if the failures go away.
+=item test-notty test_notty
+
+Sets PERL_SKIP_TTY_TEST to true before running normal test.
+
+=back
+
+=head2 Running tests by hand
+
+You can run part of the test suite by hand by using one the following
+commands from the F<t/> directory :
+
+ ./perl -I../lib TEST list-of-.t-files
+
+or
+
+ ./perl -I../lib harness list-of-.t-files
+
+(if you don't specify test scripts, the whole test suite will be run.)
+
+=head3 Using t/harness for testing
+
+If you use C<harness> for testing you have several command line options
+available to you. The arguments are as follows, and are in the order
+that they must appear if used together.
+
+ harness -v -torture -re=pattern LIST OF FILES TO TEST
+ harness -v -torture -re LIST OF PATTERNS TO MATCH
+
+If C<LIST OF FILES TO TEST> is omitted the file list is obtained from
+the manifest. The file list may include shell wildcards which will be
+expanded out.
+
+=over 4
+
+=item -v
+
+Run the tests under verbose mode so you can see what tests were run,
+and debug output.
+
+=item -torture
+
+Run the torture tests as well as the normal set.
+
+=item -re=PATTERN
+
+Filter the file list so that all the test files run match PATTERN.
+Note that this form is distinct from the B<-re LIST OF PATTERNS> form below
+in that it allows the file list to be provided as well.
+
+=item -re LIST OF PATTERNS
+
+Filter the file list so that all the test files run match
+/(LIST|OF|PATTERNS)/. Note that with this form the patterns
+are joined by '|' and you cannot supply a list of files, instead
+the test files are obtained from the MANIFEST.
+
+=back
+
+You can run an individual test by a command similar to
+
+ ./perl -I../lib patho/to/foo.t
+
+except that the harnesses set up some environment variables that may
+affect the execution of the test :
+
+=over 4
+
+=item PERL_CORE=1
+
+indicates that we're running this test part of the perl core test suite.
+This is useful for modules that have a dual life on CPAN.
+
+=item PERL_DESTRUCT_LEVEL=2
+
+is set to 2 if it isn't set already (see L</PERL_DESTRUCT_LEVEL>)
+
+=item PERL
+
+(used only by F<t/TEST>) if set, overrides the path to the perl executable
+that should be used to run the tests (the default being F<./perl>).
+
+=item PERL_SKIP_TTY_TEST
+
+if set, tells to skip the tests that need a terminal. It's actually set
+automatically by the Makefile, but can also be forced artificially by
+running 'make test_notty'.
+
+=back
+
+=head3 Other environment variables that may influence tests
+
+=over 4
+
+=item PERL_TEST_Net_Ping
+
+Setting this variable runs all the Net::Ping modules tests,
+otherwise some tests that interact with the outside world are skipped.
+See L<perl58delta>.
+
+=item PERL_TEST_NOVREXX
+
+Setting this variable skips the vrexx.t tests for OS2::REXX.
+
+=item PERL_TEST_NUMCONVERTS
+
+This sets a variable in op/numconvert.t.
+
+=back
+
+See also the documentation for the Test and Test::Harness modules,
+for more environment variables that affect testing.
+
+=head2 Common problems when patching Perl source code
+
+Perl source plays by ANSI C89 rules: no C99 (or C++) extensions. In
+some cases we have to take pre-ANSI requirements into consideration.
+You don't care about some particular platform having broken Perl?
+I hear there is still a strong demand for J2EE programmers.
+
+=head2 Perl environment problems
+
+=over 4
+
+=item *
+
+Not compiling with threading
+
+Compiling with threading (-Duseithreads) completely rewrites
+the function prototypes of Perl. You better try your changes
+with that. Related to this is the difference between "Perl_-less"
+and "Perl_-ly" APIs, for example:
+
+ Perl_sv_setiv(aTHX_ ...);
+ sv_setiv(...);
+
+The first one explicitly passes in the context, which is needed for e.g.
+threaded builds. The second one does that implicitly; do not get them
+mixed. If you are not passing in a aTHX_, you will need to do a dTHX
+(or a dVAR) as the first thing in the function.
+
+See L<perlguts/"How multiple interpreters and concurrency are supported">
+for further discussion about context.
+
+=item *
+
+Not compiling with -DDEBUGGING
+
+The DEBUGGING define exposes more code to the compiler,
+therefore more ways for things to go wrong. You should try it.
+
+=item *
+
+Introducing (non-read-only) globals
+
+Do not introduce any modifiable globals, truly global or file static.
+They are bad form and complicate multithreading and other forms of
+concurrency. The right way is to introduce them as new interpreter
+variables, see F<intrpvar.h> (at the very end for binary compatibility).
+
+Introducing read-only (const) globals is okay, as long as you verify
+with e.g. C<nm libperl.a|egrep -v ' [TURtr] '> (if your C<nm> has
+BSD-style output) that the data you added really is read-only.
+(If it is, it shouldn't show up in the output of that command.)
+
+If you want to have static strings, make them constant:
+
+ static const char etc[] = "...";
+
+If you want to have arrays of constant strings, note carefully
+the right combination of C<const>s:
+
+ static const char * const yippee[] =
+ {"hi", "ho", "silver"};
+
+There is a way to completely hide any modifiable globals (they are all
+moved to heap), the compilation setting C<-DPERL_GLOBAL_STRUCT_PRIVATE>.
+It is not normally used, but can be used for testing, read more
+about it in L<perlguts/"Background and PERL_IMPLICIT_CONTEXT">.
+
+=item *
+
+Not exporting your new function
+
+Some platforms (Win32, AIX, VMS, OS/2, to name a few) require any
+function that is part of the public API (the shared Perl library)
+to be explicitly marked as exported. See the discussion about
+F<embed.pl> in L<perlguts>.
+
+=item *
+
+Exporting your new function
+
+The new shiny result of either genuine new functionality or your
+arduous refactoring is now ready and correctly exported. So what
+could possibly go wrong?
+
+Maybe simply that your function did not need to be exported in the
+first place. Perl has a long and not so glorious history of exporting
+functions that it should not have.
+
+If the function is used only inside one source code file, make it
+static. See the discussion about F<embed.pl> in L<perlguts>.
+
+If the function is used across several files, but intended only for
+Perl's internal use (and this should be the common case), do not
+export it to the public API. See the discussion about F<embed.pl>
+in L<perlguts>.
+
+=back
+
+=head2 Portability problems
+
+The following are common causes of compilation and/or execution
+failures, not common to Perl as such. The C FAQ is good bedtime
+reading. Please test your changes with as many C compilers and
+platforms as possible; we will, anyway, and it's nice to save
+oneself from public embarrassment.
+
+If using gcc, you can add the C<-std=c89> option which will hopefully
+catch most of these unportabilities. (However it might also catch
+incompatibilities in your system's header files.)
+
+Use the Configure C<-Dgccansipedantic> flag to enable the gcc
+C<-ansi -pedantic> flags which enforce stricter ANSI rules.
+
+If using the C<gcc -Wall> note that not all the possible warnings
+(like C<-Wunitialized>) are given unless you also compile with C<-O>.
+
+Note that if using gcc, starting from Perl 5.9.5 the Perl core source
+code files (the ones at the top level of the source code distribution,
+but not e.g. the extensions under ext/) are automatically compiled
+with as many as possible of the C<-std=c89>, C<-ansi>, C<-pedantic>,
+and a selection of C<-W> flags (see cflags.SH).
+
+Also study L<perlport> carefully to avoid any bad assumptions
+about the operating system, filesystems, and so forth.
+
+You may once in a while try a "make microperl" to see whether we
+can still compile Perl with just the bare minimum of interfaces.
+(See README.micro.)
+
+Do not assume an operating system indicates a certain compiler.
+
+=over 4
+
+=item *
+
+Casting pointers to integers or casting integers to pointers
+
+ void castaway(U8* p)
+ {
+ IV i = p;
+
+or
+
+ void castaway(U8* p)
+ {
+ IV i = (IV)p;
+
+Both are bad, and broken, and unportable. Use the PTR2IV()
+macro that does it right. (Likewise, there are PTR2UV(), PTR2NV(),
+INT2PTR(), and NUM2PTR().)
+
+=item *
+
+Casting between data function pointers and data pointers
+
+Technically speaking casting between function pointers and data
+pointers is unportable and undefined, but practically speaking
+it seems to work, but you should use the FPTR2DPTR() and DPTR2FPTR()
+macros. Sometimes you can also play games with unions.
+
+=item *
+
+Assuming sizeof(int) == sizeof(long)
+
+There are platforms where longs are 64 bits, and platforms where ints
+are 64 bits, and while we are out to shock you, even platforms where
+shorts are 64 bits. This is all legal according to the C standard.
+(In other words, "long long" is not a portable way to specify 64 bits,
+and "long long" is not even guaranteed to be any wider than "long".)
+
+Instead, use the definitions IV, UV, IVSIZE, I32SIZE, and so forth.
+Avoid things like I32 because they are B<not> guaranteed to be
+I<exactly> 32 bits, they are I<at least> 32 bits, nor are they
+guaranteed to be B<int> or B<long>. If you really explicitly need
+64-bit variables, use I64 and U64, but only if guarded by HAS_QUAD.
+
+=item *
+
+Assuming one can dereference any type of pointer for any type of data
+
+ char *p = ...;
+ long pony = *p; /* BAD */
+
+Many platforms, quite rightly so, will give you a core dump instead
+of a pony if the p happens not be correctly aligned.
+
+=item *
+
+Lvalue casts
+
+ (int)*p = ...; /* BAD */
+
+Simply not portable. Get your lvalue to be of the right type,
+or maybe use temporary variables, or dirty tricks with unions.
+
+=item *
+
+Assume B<anything> about structs (especially the ones you
+don't control, like the ones coming from the system headers)
+
+=over 8
+
+=item *
+
+That a certain field exists in a struct
+
+=item *
+
+That no other fields exist besides the ones you know of
+
+=item *
+
+That a field is of certain signedness, sizeof, or type
+
+=item *
+
+That the fields are in a certain order
+
+=over 8
+
+=item *
+
+While C guarantees the ordering specified in the struct definition,
+between different platforms the definitions might differ
+
+=back
+
+=item *
+
+That the sizeof(struct) or the alignments are the same everywhere
+
+=over 8
+
+=item *
+
+There might be padding bytes between the fields to align the fields -
+the bytes can be anything
+
+=item *
+
+Structs are required to be aligned to the maximum alignment required
+by the fields - which for native types is for usually equivalent to
+sizeof() of the field
+
+=back
+
+=back
+
+=item *
+
+Assuming the character set is ASCIIish
+
+Perl can compile and run under EBCDIC platforms. See L<perlebcdic>.
+This is transparent for the most part, but because the character sets
+differ, you shouldn't use numeric (decimal, octal, nor hex) constants
+to refer to characters. You can safely say 'A', but not 0x41.
+You can safely say '\n', but not \012.
+If a character doesn't have a trivial input form, you can
+create a #define for it in both C<utfebcdic.h> and C<utf8.h>, so that
+it resolves to different values depending on the character set being used.
+(There are three different EBCDIC character sets defined in C<utfebcdic.h>,
+so it might be best to insert the #define three times in that file.)
+
+Also, the range 'A' - 'Z' in ASCII is an unbroken sequence of 26 upper case
+alphabetic characters. That is not true in EBCDIC. Nor for 'a' to 'z'.
+But '0' - '9' is an unbroken range in both systems. Don't assume anything
+about other ranges.
+
+Many of the comments in the existing code ignore the possibility of EBCDIC,
+and may be wrong therefore, even if the code works.
+This is actually a tribute to the successful transparent insertion of being
+able to handle EBCDIC without having to change pre-existing code.
+
+UTF-8 and UTF-EBCDIC are two different encodings used to represent Unicode
+code points as sequences of bytes. Macros
+with the same names (but different definitions)
+in C<utf8.h> and C<utfebcdic.h>
+are used to allow the calling code to think that there is only one such
+encoding.
+This is almost always referred to as C<utf8>, but it means the EBCDIC version
+as well. Again, comments in the code may well be wrong even if the code itself
+is right.
+For example, the concept of C<invariant characters> differs between ASCII and
+EBCDIC.
+On ASCII platforms, only characters that do not have the high-order
+bit set (i.e. whose ordinals are strict ASCII, 0 - 127)
+are invariant, and the documentation and comments in the code
+may assume that,
+often referring to something like, say, C<hibit>.
+The situation differs and is not so simple on EBCDIC machines, but as long as
+the code itself uses the C<NATIVE_IS_INVARIANT()> macro appropriately, it
+works, even if the comments are wrong.
+
+=item *
+
+Assuming the character set is just ASCII
+
+ASCII is a 7 bit encoding, but bytes have 8 bits in them. The 128 extra
+characters have different meanings depending on the locale. Absent a locale,
+currently these extra characters are generally considered to be unassigned,
+and this has presented some problems.
+This is being changed starting in 5.12 so that these characters will
+be considered to be Latin-1 (ISO-8859-1).
+
+=item *
+
+Mixing #define and #ifdef
+
+ #define BURGLE(x) ... \
+ #ifdef BURGLE_OLD_STYLE /* BAD */
+ ... do it the old way ... \
+ #else
+ ... do it the new way ... \
+ #endif
+
+You cannot portably "stack" cpp directives. For example in the above
+you need two separate BURGLE() #defines, one for each #ifdef branch.
+
+=item *
+
+Adding non-comment stuff after #endif or #else
+
+ #ifdef SNOSH
+ ...
+ #else !SNOSH /* BAD */
+ ...
+ #endif SNOSH /* BAD */
+
+The #endif and #else cannot portably have anything non-comment after
+them. If you want to document what is going (which is a good idea
+especially if the branches are long), use (C) comments:
+
+ #ifdef SNOSH
+ ...
+ #else /* !SNOSH */
+ ...
+ #endif /* SNOSH */
+
+The gcc option C<-Wendif-labels> warns about the bad variant
+(by default on starting from Perl 5.9.4).
+
+=item *
+
+Having a comma after the last element of an enum list
+
+ enum color {
+ CERULEAN,
+ CHARTREUSE,
+ CINNABAR, /* BAD */
+ };
+
+is not portable. Leave out the last comma.
+
+Also note that whether enums are implicitly morphable to ints
+varies between compilers, you might need to (int).
+
+=item *
+
+Using //-comments
+
+ // This function bamfoodles the zorklator. /* BAD */
+
+That is C99 or C++. Perl is C89. Using the //-comments is silently
+allowed by many C compilers but cranking up the ANSI C89 strictness
+(which we like to do) causes the compilation to fail.
+
+=item *
+
+Mixing declarations and code
+
+ void zorklator()
+ {
+ int n = 3;
+ set_zorkmids(n); /* BAD */
+ int q = 4;
+
+That is C99 or C++. Some C compilers allow that, but you shouldn't.
+
+The gcc option C<-Wdeclaration-after-statements> scans for such problems
+(by default on starting from Perl 5.9.4).
+
+=item *
+
+Introducing variables inside for()
+
+ for(int i = ...; ...; ...) { /* BAD */
+
+That is C99 or C++. While it would indeed be awfully nice to have that
+also in C89, to limit the scope of the loop variable, alas, we cannot.
+
+=item *
+
+Mixing signed char pointers with unsigned char pointers
+
+ int foo(char *s) { ... }
+ ...
+ unsigned char *t = ...; /* Or U8* t = ... */
+ foo(t); /* BAD */
+
+While this is legal practice, it is certainly dubious, and downright
+fatal in at least one platform: for example VMS cc considers this a
+fatal error. One cause for people often making this mistake is that a
+"naked char" and therefore dereferencing a "naked char pointer" have
+an undefined signedness: it depends on the compiler and the flags of
+the compiler and the underlying platform whether the result is signed
+or unsigned. For this very same reason using a 'char' as an array
+index is bad.
+
+=item *
+
+Macros that have string constants and their arguments as substrings of
+the string constants
+
+ #define FOO(n) printf("number = %d\n", n) /* BAD */
+ FOO(10);
+
+Pre-ANSI semantics for that was equivalent to
+
+ printf("10umber = %d\10");
+
+which is probably not what you were expecting. Unfortunately at least
+one reasonably common and modern C compiler does "real backward
+compatibility" here, in AIX that is what still happens even though the
+rest of the AIX compiler is very happily C89.
+
+=item *
+
+Using printf formats for non-basic C types
+
+ IV i = ...;
+ printf("i = %d\n", i); /* BAD */
+
+While this might by accident work in some platform (where IV happens
+to be an C<int>), in general it cannot. IV might be something larger.
+Even worse the situation is with more specific types (defined by Perl's
+configuration step in F<config.h>):
+
+ Uid_t who = ...;
+ printf("who = %d\n", who); /* BAD */
+
+The problem here is that Uid_t might be not only not C<int>-wide
+but it might also be unsigned, in which case large uids would be
+printed as negative values.
+
+There is no simple solution to this because of printf()'s limited
+intelligence, but for many types the right format is available as
+with either 'f' or '_f' suffix, for example:
+
+ IVdf /* IV in decimal */
+ UVxf /* UV is hexadecimal */
+
+ printf("i = %"IVdf"\n", i); /* The IVdf is a string constant. */
+
+ Uid_t_f /* Uid_t in decimal */
+
+ printf("who = %"Uid_t_f"\n", who);
+
+Or you can try casting to a "wide enough" type:
+
+ printf("i = %"IVdf"\n", (IV)something_very_small_and_signed);
+
+Also remember that the C<%p> format really does require a void pointer:
+
+ U8* p = ...;
+ printf("p = %p\n", (void*)p);
+
+The gcc option C<-Wformat> scans for such problems.
+
+=item *
+
+Blindly using variadic macros
+
+gcc has had them for a while with its own syntax, and C99 brought
+them with a standardized syntax. Don't use the former, and use
+the latter only if the HAS_C99_VARIADIC_MACROS is defined.
+
+=item *
+
+Blindly passing va_list
+
+Not all platforms support passing va_list to further varargs (stdarg)
+functions. The right thing to do is to copy the va_list using the
+Perl_va_copy() if the NEED_VA_COPY is defined.
+
+=item *
+
+Using gcc statement expressions
+
+ val = ({...;...;...}); /* BAD */
+
+While a nice extension, it's not portable. The Perl code does
+admittedly use them if available to gain some extra speed
+(essentially as a funky form of inlining), but you shouldn't.
+
+=item *
+
+Binding together several statements in a macro
+
+Use the macros STMT_START and STMT_END.
+
+ STMT_START {
+ ...
+ } STMT_END
+
+=item *
+
+Testing for operating systems or versions when should be testing for features
+
+ #ifdef __FOONIX__ /* BAD */
+ foo = quux();
+ #endif
+
+Unless you know with 100% certainty that quux() is only ever available
+for the "Foonix" operating system B<and> that is available B<and>
+correctly working for B<all> past, present, B<and> future versions of
+"Foonix", the above is very wrong. This is more correct (though still
+not perfect, because the below is a compile-time check):
+
+ #ifdef HAS_QUUX
+ foo = quux();
+ #endif
+
+How does the HAS_QUUX become defined where it needs to be? Well, if
+Foonix happens to be Unixy enough to be able to run the Configure
+script, and Configure has been taught about detecting and testing
+quux(), the HAS_QUUX will be correctly defined. In other platforms,
+the corresponding configuration step will hopefully do the same.
+
+In a pinch, if you cannot wait for Configure to be educated,
+or if you have a good hunch of where quux() might be available,
+you can temporarily try the following:
+
+ #if (defined(__FOONIX__) || defined(__BARNIX__))
+ # define HAS_QUUX
+ #endif
+
+ ...
+
+ #ifdef HAS_QUUX
+ foo = quux();
+ #endif
+
+But in any case, try to keep the features and operating systems separate.
+
+=back
+
+=head2 Problematic System Interfaces
+
+=over 4
+
+=item *
+
+malloc(0), realloc(0), calloc(0, 0) are non-portable. To be portable
+allocate at least one byte. (In general you should rarely need to
+work at this low level, but instead use the various malloc wrappers.)
+
+=item *
+
+snprintf() - the return type is unportable. Use my_snprintf() instead.
+
+=back
+
+=head2 Security problems
+
+Last but not least, here are various tips for safer coding.
+
+=over 4
+
+=item *
+
+Do not use gets()
+
+Or we will publicly ridicule you. Seriously.
+
+=item *
+
+Do not use strcpy() or strcat() or strncpy() or strncat()
+
+Use my_strlcpy() and my_strlcat() instead: they either use the native
+implementation, or Perl's own implementation (borrowed from the public
+domain implementation of INN).
+
+=item *
+
+Do not use sprintf() or vsprintf()
+
+If you really want just plain byte strings, use my_snprintf()
+and my_vsnprintf() instead, which will try to use snprintf() and
+vsnprintf() if those safer APIs are available. If you want something
+fancier than a plain byte string, use SVs and Perl_sv_catpvf().
+
+=back
+
+=head1 EXTERNAL TOOLS FOR DEBUGGING PERL
+
+Sometimes it helps to use external tools while debugging and
+testing Perl. This section tries to guide you through using
+some common testing and debugging tools with Perl. This is
+meant as a guide to interfacing these tools with Perl, not
+as any kind of guide to the use of the tools themselves.
+
+B<NOTE 1>: Running under memory debuggers such as Purify, valgrind, or
+Third Degree greatly slows down the execution: seconds become minutes,
+minutes become hours. For example as of Perl 5.8.1, the
+ext/Encode/t/Unicode.t takes extraordinarily long to complete under
+e.g. Purify, Third Degree, and valgrind. Under valgrind it takes more
+than six hours, even on a snappy computer. The said test must be
+doing something that is quite unfriendly for memory debuggers. If you
+don't feel like waiting, that you can simply kill away the perl
+process.
+
+B<NOTE 2>: To minimize the number of memory leak false alarms (see
+L</PERL_DESTRUCT_LEVEL> for more information), you have to set the
+environment variable PERL_DESTRUCT_LEVEL to 2.
+
+For csh-like shells:
+
+ setenv PERL_DESTRUCT_LEVEL 2
+
+For Bourne-type shells:
+
+ PERL_DESTRUCT_LEVEL=2
+ export PERL_DESTRUCT_LEVEL
+
+In Unixy environments you can also use the C<env> command:
+
+ env PERL_DESTRUCT_LEVEL=2 valgrind ./perl -Ilib ...
+
+B<NOTE 3>: There are known memory leaks when there are compile-time
+errors within eval or require, seeing C<S_doeval> in the call stack
+is a good sign of these. Fixing these leaks is non-trivial,
+unfortunately, but they must be fixed eventually.
+
+B<NOTE 4>: L<DynaLoader> will not clean up after itself completely
+unless Perl is built with the Configure option
+C<-Accflags=-DDL_UNLOAD_ALL_AT_EXIT>.
+
+=head2 Rational Software's Purify
+
+Purify is a commercial tool that is helpful in identifying
+memory overruns, wild pointers, memory leaks and other such
+badness. Perl must be compiled in a specific way for
+optimal testing with Purify. Purify is available under
+Windows NT, Solaris, HP-UX, SGI, and Siemens Unix.
+
+=head2 Purify on Unix
+
+On Unix, Purify creates a new Perl binary. To get the most
+benefit out of Purify, you should create the perl to Purify
+using:
+
+ sh Configure -Accflags=-DPURIFY -Doptimize='-g' \
+ -Uusemymalloc -Dusemultiplicity
+
+where these arguments mean:
+
+=over 4
+
+=item -Accflags=-DPURIFY
+
+Disables Perl's arena memory allocation functions, as well as
+forcing use of memory allocation functions derived from the
+system malloc.
+
+=item -Doptimize='-g'
+
+Adds debugging information so that you see the exact source
+statements where the problem occurs. Without this flag, all
+you will see is the source filename of where the error occurred.
+
+=item -Uusemymalloc
+
+Disable Perl's malloc so that Purify can more closely monitor
+allocations and leaks. Using Perl's malloc will make Purify
+report most leaks in the "potential" leaks category.
+
+=item -Dusemultiplicity
+
+Enabling the multiplicity option allows perl to clean up
+thoroughly when the interpreter shuts down, which reduces the
+number of bogus leak reports from Purify.
+
+=back
+
+Once you've compiled a perl suitable for Purify'ing, then you
+can just:
+
+ make pureperl
+
+which creates a binary named 'pureperl' that has been Purify'ed.
+This binary is used in place of the standard 'perl' binary
+when you want to debug Perl memory problems.
+
+As an example, to show any memory leaks produced during the
+standard Perl testset you would create and run the Purify'ed
+perl as:
+
+ make pureperl
+ cd t
+ ../pureperl -I../lib harness
+
+which would run Perl on test.pl and report any memory problems.
+
+Purify outputs messages in "Viewer" windows by default. If
+you don't have a windowing environment or if you simply
+want the Purify output to unobtrusively go to a log file
+instead of to the interactive window, use these following
+options to output to the log file "perl.log":
+
+ setenv PURIFYOPTIONS "-chain-length=25 -windows=no \
+ -log-file=perl.log -append-logfile=yes"
+
+If you plan to use the "Viewer" windows, then you only need this option:
+
+ setenv PURIFYOPTIONS "-chain-length=25"
+
+In Bourne-type shells:
+
+ PURIFYOPTIONS="..."
+ export PURIFYOPTIONS
+
+or if you have the "env" utility:
+
+ env PURIFYOPTIONS="..." ../pureperl ...
+
+=head2 Purify on NT
+
+Purify on Windows NT instruments the Perl binary 'perl.exe'
+on the fly. There are several options in the makefile you
+should change to get the most use out of Purify:
+
+=over 4
+
+=item DEFINES
+
+You should add -DPURIFY to the DEFINES line so the DEFINES
+line looks something like:
+
+ DEFINES = -DWIN32 -D_CONSOLE -DNO_STRICT $(CRYPT_FLAG) -DPURIFY=1
+
+to disable Perl's arena memory allocation functions, as
+well as to force use of memory allocation functions derived
+from the system malloc.
+
+=item USE_MULTI = define
+
+Enabling the multiplicity option allows perl to clean up
+thoroughly when the interpreter shuts down, which reduces the
+number of bogus leak reports from Purify.
+
+=item #PERL_MALLOC = define
+
+Disable Perl's malloc so that Purify can more closely monitor
+allocations and leaks. Using Perl's malloc will make Purify
+report most leaks in the "potential" leaks category.
+
+=item CFG = Debug
+
+Adds debugging information so that you see the exact source
+statements where the problem occurs. Without this flag, all
+you will see is the source filename of where the error occurred.
+
+=back
+
+As an example, to show any memory leaks produced during the
+standard Perl testset you would create and run Purify as:
+
+ cd win32
+ make
+ cd ../t
+ purify ../perl -I../lib harness
+
+which would instrument Perl in memory, run Perl on test.pl,
+then finally report any memory problems.
+
+=head2 valgrind
+
+The excellent valgrind tool can be used to find out both memory leaks
+and illegal memory accesses. As of version 3.3.0, Valgrind only
+supports Linux on x86, x86-64 and PowerPC. The special "test.valgrind"
+target can be used to run the tests under valgrind. Found errors
+and memory leaks are logged in files named F<testfile.valgrind>.
+
+Valgrind also provides a cachegrind tool, invoked on perl as:
+
+ VG_OPTS=--tool=cachegrind make test.valgrind
+
+As system libraries (most notably glibc) are also triggering errors,
+valgrind allows to suppress such errors using suppression files. The
+default suppression file that comes with valgrind already catches a lot
+of them. Some additional suppressions are defined in F<t/perl.supp>.
+
+To get valgrind and for more information see
+
+ http://developer.kde.org/~sewardj/
+
+=head2 Compaq's/Digital's/HP's Third Degree
+
+Third Degree is a tool for memory leak detection and memory access checks.
+It is one of the many tools in the ATOM toolkit. The toolkit is only
+available on Tru64 (formerly known as Digital UNIX formerly known as
+DEC OSF/1).
+
+When building Perl, you must first run Configure with -Doptimize=-g
+and -Uusemymalloc flags, after that you can use the make targets
+"perl.third" and "test.third". (What is required is that Perl must be
+compiled using the C<-g> flag, you may need to re-Configure.)
+
+The short story is that with "atom" you can instrument the Perl
+executable to create a new executable called F<perl.third>. When the
+instrumented executable is run, it creates a log of dubious memory
+traffic in file called F<perl.3log>. See the manual pages of atom and
+third for more information. The most extensive Third Degree
+documentation is available in the Compaq "Tru64 UNIX Programmer's
+Guide", chapter "Debugging Programs with Third Degree".
+
+The "test.third" leaves a lot of files named F<foo_bar.3log> in the t/
+subdirectory. There is a problem with these files: Third Degree is so
+effective that it finds problems also in the system libraries.
+Therefore you should used the Porting/thirdclean script to cleanup
+the F<*.3log> files.
+
+There are also leaks that for given certain definition of a leak,
+aren't. See L</PERL_DESTRUCT_LEVEL> for more information.
+
+=head2 PERL_DESTRUCT_LEVEL
+
+If you want to run any of the tests yourself manually using e.g.
+valgrind, or the pureperl or perl.third executables, please note that
+by default perl B<does not> explicitly cleanup all the memory it has
+allocated (such as global memory arenas) but instead lets the exit()
+of the whole program "take care" of such allocations, also known as
+"global destruction of objects".
+
+There is a way to tell perl to do complete cleanup: set the
+environment variable PERL_DESTRUCT_LEVEL to a non-zero value.
+The t/TEST wrapper does set this to 2, and this is what you
+need to do too, if you don't want to see the "global leaks":
+For example, for "third-degreed" Perl:
+
+ env PERL_DESTRUCT_LEVEL=2 ./perl.third -Ilib t/foo/bar.t
+
+(Note: the mod_perl apache module uses also this environment variable
+for its own purposes and extended its semantics. Refer to the mod_perl
+documentation for more information. Also, spawned threads do the
+equivalent of setting this variable to the value 1.)
+
+If, at the end of a run you get the message I<N scalars leaked>, you can
+recompile with C<-DDEBUG_LEAKING_SCALARS>, which will cause the addresses
+of all those leaked SVs to be dumped along with details as to where each
+SV was originally allocated. This information is also displayed by
+Devel::Peek. Note that the extra details recorded with each SV increases
+memory usage, so it shouldn't be used in production environments. It also
+converts C<new_SV()> from a macro into a real function, so you can use
+your favourite debugger to discover where those pesky SVs were allocated.
+
+If you see that you're leaking memory at runtime, but neither valgrind
+nor C<-DDEBUG_LEAKING_SCALARS> will find anything, you're probably
+leaking SVs that are still reachable and will be properly cleaned up
+during destruction of the interpreter. In such cases, using the C<-Dm>
+switch can point you to the source of the leak. If the executable was
+built with C<-DDEBUG_LEAKING_SCALARS>, C<-Dm> will output SV allocations
+in addition to memory allocations. Each SV allocation has a distinct
+serial number that will be written on creation and destruction of the SV.
+So if you're executing the leaking code in a loop, you need to look for
+SVs that are created, but never destroyed between each cycle. If such an
+SV is found, set a conditional breakpoint within C<new_SV()> and make it
+break only when C<PL_sv_serial> is equal to the serial number of the
+leaking SV. Then you will catch the interpreter in exactly the state
+where the leaking SV is allocated, which is sufficient in many cases to
+find the source of the leak.
+
+As C<-Dm> is using the PerlIO layer for output, it will by itself
+allocate quite a bunch of SVs, which are hidden to avoid recursion.
+You can bypass the PerlIO layer if you use the SV logging provided
+by C<-DPERL_MEM_LOG> instead.
+
+=head2 PERL_MEM_LOG
+
+If compiled with C<-DPERL_MEM_LOG>, both memory and SV allocations go
+through logging functions, which is handy for breakpoint setting.
+
+Unless C<-DPERL_MEM_LOG_NOIMPL> is also compiled, the logging
+functions read $ENV{PERL_MEM_LOG} to determine whether to log the
+event, and if so how:
+
+ $ENV{PERL_MEM_LOG} =~ /m/ Log all memory ops
+ $ENV{PERL_MEM_LOG} =~ /s/ Log all SV ops
+ $ENV{PERL_MEM_LOG} =~ /t/ include timestamp in Log
+ $ENV{PERL_MEM_LOG} =~ /^(\d+)/ write to FD given (default is 2)
+
+Memory logging is somewhat similar to C<-Dm> but is independent of
+C<-DDEBUGGING>, and at a higher level; all uses of Newx(), Renew(),
+and Safefree() are logged with the caller's source code file and line
+number (and C function name, if supported by the C compiler). In
+contrast, C<-Dm> is directly at the point of C<malloc()>. SV logging
+is similar.
+
+Since the logging doesn't use PerlIO, all SV allocations are logged
+and no extra SV allocations are introduced by enabling the logging.
+If compiled with C<-DDEBUG_LEAKING_SCALARS>, the serial number for
+each SV allocation is also logged.
+
+=head2 Profiling
+
+Depending on your platform there are various of profiling Perl.
+
+There are two commonly used techniques of profiling executables:
+I<statistical time-sampling> and I<basic-block counting>.
+
+The first method takes periodically samples of the CPU program
+counter, and since the program counter can be correlated with the code
+generated for functions, we get a statistical view of in which
+functions the program is spending its time. The caveats are that very
+small/fast functions have lower probability of showing up in the
+profile, and that periodically interrupting the program (this is
+usually done rather frequently, in the scale of milliseconds) imposes
+an additional overhead that may skew the results. The first problem
+can be alleviated by running the code for longer (in general this is a
+good idea for profiling), the second problem is usually kept in guard
+by the profiling tools themselves.
+
+The second method divides up the generated code into I<basic blocks>.
+Basic blocks are sections of code that are entered only in the
+beginning and exited only at the end. For example, a conditional jump
+starts a basic block. Basic block profiling usually works by
+I<instrumenting> the code by adding I<enter basic block #nnnn>
+book-keeping code to the generated code. During the execution of the
+code the basic block counters are then updated appropriately. The
+caveat is that the added extra code can skew the results: again, the
+profiling tools usually try to factor their own effects out of the
+results.
+
+=head2 Gprof Profiling
+
+gprof is a profiling tool available in many Unix platforms,
+it uses F<statistical time-sampling>.
+
+You can build a profiled version of perl called "perl.gprof" by
+invoking the make target "perl.gprof" (What is required is that Perl
+must be compiled using the C<-pg> flag, you may need to re-Configure).
+Running the profiled version of Perl will create an output file called
+F<gmon.out> is created which contains the profiling data collected
+during the execution.
+
+The gprof tool can then display the collected data in various ways.
+Usually gprof understands the following options:
+
+=over 4
+
+=item -a
+
+Suppress statically defined functions from the profile.
+
+=item -b
+
+Suppress the verbose descriptions in the profile.
+
+=item -e routine
+
+Exclude the given routine and its descendants from the profile.
+
+=item -f routine
+
+Display only the given routine and its descendants in the profile.
+
+=item -s
+
+Generate a summary file called F<gmon.sum> which then may be given
+to subsequent gprof runs to accumulate data over several runs.
+
+=item -z
+
+Display routines that have zero usage.
+
+=back
+
+For more detailed explanation of the available commands and output
+formats, see your own local documentation of gprof.
+
+quick hint:
+
+ $ sh Configure -des -Dusedevel -Doptimize='-pg' && make perl.gprof
+ $ ./perl.gprof someprog # creates gmon.out in current directory
+ $ gprof ./perl.gprof > out
+ $ view out
+
+=head2 GCC gcov Profiling
+
+Starting from GCC 3.0 I<basic block profiling> is officially available
+for the GNU CC.
+
+You can build a profiled version of perl called F<perl.gcov> by
+invoking the make target "perl.gcov" (what is required that Perl must
+be compiled using gcc with the flags C<-fprofile-arcs
+-ftest-coverage>, you may need to re-Configure).
+
+Running the profiled version of Perl will cause profile output to be
+generated. For each source file an accompanying ".da" file will be
+created.
+
+To display the results you use the "gcov" utility (which should
+be installed if you have gcc 3.0 or newer installed). F<gcov> is
+run on source code files, like this
+
+ gcov sv.c
+
+which will cause F<sv.c.gcov> to be created. The F<.gcov> files
+contain the source code annotated with relative frequencies of
+execution indicated by "#" markers.
+
+Useful options of F<gcov> include C<-b> which will summarise the
+basic block, branch, and function call coverage, and C<-c> which
+instead of relative frequencies will use the actual counts. For
+more information on the use of F<gcov> and basic block profiling
+with gcc, see the latest GNU CC manual, as of GCC 3.0 see
+
+ http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc.html
+
+and its section titled "8. gcov: a Test Coverage Program"
+
+ http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html#SEC132
+
+quick hint:
+
+ $ sh Configure -des -Doptimize='-g' -Accflags='-fprofile-arcs -ftest-coverage' \
+ -Aldflags='-fprofile-arcs -ftest-coverage' && make perl.gcov
+ $ rm -f regexec.c.gcov regexec.gcda
+ $ ./perl.gcov
+ $ gcov regexec.c
+ $ view regexec.c.gcov
+
+=head2 Pixie Profiling
+
+Pixie is a profiling tool available on IRIX and Tru64 (aka Digital
+UNIX aka DEC OSF/1) platforms. Pixie does its profiling using
+I<basic-block counting>.
+
+You can build a profiled version of perl called F<perl.pixie> by
+invoking the make target "perl.pixie" (what is required is that Perl
+must be compiled using the C<-g> flag, you may need to re-Configure).
+
+In Tru64 a file called F<perl.Addrs> will also be silently created,
+this file contains the addresses of the basic blocks. Running the
+profiled version of Perl will create a new file called "perl.Counts"
+which contains the counts for the basic block for that particular
+program execution.
+
+To display the results you use the F<prof> utility. The exact
+incantation depends on your operating system, "prof perl.Counts" in
+IRIX, and "prof -pixie -all -L. perl" in Tru64.
+
+In IRIX the following prof options are available:
+
+=over 4
+
+=item -h
+
+Reports the most heavily used lines in descending order of use.
+Useful for finding the hotspot lines.
+
+=item -l
+
+Groups lines by procedure, with procedures sorted in descending order of use.
+Within a procedure, lines are listed in source order.
+Useful for finding the hotspots of procedures.
+
+=back
+
+In Tru64 the following options are available:
+
+=over 4
+
+=item -p[rocedures]
+
+Procedures sorted in descending order by the number of cycles executed
+in each procedure. Useful for finding the hotspot procedures.
+(This is the default option.)
+
+=item -h[eavy]
+
+Lines sorted in descending order by the number of cycles executed in
+each line. Useful for finding the hotspot lines.
+
+=item -i[nvocations]
+
+The called procedures are sorted in descending order by number of calls
+made to the procedures. Useful for finding the most used procedures.
+
+=item -l[ines]
+
+Grouped by procedure, sorted by cycles executed per procedure.
+Useful for finding the hotspots of procedures.
+
+=item -testcoverage
+
+The compiler emitted code for these lines, but the code was unexecuted.
+
+=item -z[ero]
+
+Unexecuted procedures.
+
+=back
+
+For further information, see your system's manual pages for pixie and prof.
+
+=head2 Miscellaneous tricks
+
+=over 4
+
+=item *
+
+Those debugging perl with the DDD frontend over gdb may find the
+following useful:
+
+You can extend the data conversion shortcuts menu, so for example you
+can display an SV's IV value with one click, without doing any typing.
+To do that simply edit ~/.ddd/init file and add after:
+
+ ! Display shortcuts.
+ Ddd*gdbDisplayShortcuts: \
+ /t () // Convert to Bin\n\
+ /d () // Convert to Dec\n\
+ /x () // Convert to Hex\n\
+ /o () // Convert to Oct(\n\
+
+the following two lines:
+
+ ((XPV*) (())->sv_any )->xpv_pv // 2pvx\n\
+ ((XPVIV*) (())->sv_any )->xiv_iv // 2ivx
+
+so now you can do ivx and pvx lookups or you can plug there the
+sv_peek "conversion":
+
+ Perl_sv_peek(my_perl, (SV*)()) // sv_peek
+
+(The my_perl is for threaded builds.)
+Just remember that every line, but the last one, should end with \n\
+
+Alternatively edit the init file interactively via:
+3rd mouse button -> New Display -> Edit Menu
+
+Note: you can define up to 20 conversion shortcuts in the gdb
+section.
+
+=item *
+
+If you see in a debugger a memory area mysteriously full of 0xABABABAB
+or 0xEFEFEFEF, you may be seeing the effect of the Poison() macros,
+see L<perlclib>.
+
+=item *
+
+Under ithreads the optree is read only. If you want to enforce this, to check
+for write accesses from buggy code, compile with C<-DPL_OP_SLAB_ALLOC> to
+enable the OP slab allocator and C<-DPERL_DEBUG_READONLY_OPS> to enable code
+that allocates op memory via C<mmap>, and sets it read-only at run time.
+Any write access to an op results in a C<SIGBUS> and abort.
+
+This code is intended for development only, and may not be portable even to
+all Unix variants. Also, it is an 80% solution, in that it isn't able to make
+all ops read only. Specifically it
+
+=over
+
+=item 1
+
+Only sets read-only on all slabs of ops at C<CHECK> time, hence ops allocated
+later via C<require> or C<eval> will be re-write
+
+=item 2
+
+Turns an entire slab of ops read-write if the refcount of any op in the slab
+needs to be decreased.
+
+=item 3
+
+Turns an entire slab of ops read-write if any op from the slab is freed.
+
+=back
+
+It's not possible to turn the slabs to read-only after an action requiring
+read-write access, as either can happen during op tree building time, so
+there may still be legitimate write access.
+
+However, as an 80% solution it is still effective, as currently it catches
+a write access during the generation of F<Config.pm>, which means that we
+can't yet build F<perl> with this enabled.
+
+=back
+
+
+=head1 CONCLUSION
+
+We've had a brief look around the Perl source, how to maintain quality
+of the source code, an overview of the stages F<perl> goes through
+when it's running your code, how to use debuggers to poke at the Perl
+guts, and finally how to analyse the execution of Perl. We took a very
+simple problem and demonstrated how to solve it fully - with
+documentation, regression tests, and finally a patch for submission to
+p5p. Finally, we talked about how to use external tools to debug and
+test Perl.
+
+I'd now suggest you read over those references again, and then, as soon
+as possible, get your hands dirty. The best way to learn is by doing,
+so:
+
+=over 3
+
+=item *
+
+Subscribe to perl5-porters, follow the patches and try and understand
+them; don't be afraid to ask if there's a portion you're not clear on -
+who knows, you may unearth a bug in the patch...
+
+=item *
+
+Keep up to date with the bleeding edge Perl distributions and get
+familiar with the changes. Try and get an idea of what areas people are
+working on and the changes they're making.
+
+=item *
+
+Do read the README associated with your operating system, e.g. README.aix
+on the IBM AIX OS. Don't hesitate to supply patches to that README if
+you find anything missing or changed over a new OS release.
+
+=item *
+
+Find an area of Perl that seems interesting to you, and see if you can
+work out how it works. Scan through the source, and step over it in the
+debugger. Play, poke, investigate, fiddle! You'll probably get to
+understand not just your chosen area but a much wider range of F<perl>'s
+activity as well, and probably sooner than you'd think.
+
+=back
+
+=over 3
+
+=item I<The Road goes ever on and on, down from the door where it began.>
+
+=back
+
+If you can do these things, you've started on the long road to Perl porting.
+Thanks for wanting to help make Perl better - and happy hacking!
+
+=head2 Metaphoric Quotations
+
+If you recognized the quote about the Road above, you're in luck.
+
+Most software projects begin each file with a literal description of each
+file's purpose. Perl instead begins each with a literary allusion to that
+file's purpose.
+
+Like chapters in many books, all top-level Perl source files (along with a
+few others here and there) begin with an epigramic inscription that alludes,
+indirectly and metaphorically, to the material you're about to read.
+
+Quotations are taken from writings of J.R.R Tolkien pertaining to his
+Legendarium, almost always from I<The Lord of the Rings>. Chapters and
+page numbers are given using the following editions:
+
+=over 4
+
+=item *
+
+I<The Hobbit>, by J.R.R. Tolkien. The hardcover, 70th-anniversary
+edition of 2007 was used, published in the UK by Harper Collins Publishers
+and in the US by the Houghton Mifflin Company.
+
+=item *
+
+I<The Lord of the Rings>, by J.R.R. Tolkien. The hardcover,
+50th-anniversary edition of 2004 was used, published in the UK by Harper
+Collins Publishers and in the US by the Houghton Mifflin Company.
+
+=item *
+
+I<The Lays of Beleriand>, by J.R.R. Tolkien and published posthumously by his
+son and literary executor, C.J.R. Tolkien, being the 3rd of the 12 volumes
+in Christopher's mammoth I<History of Middle Earth>. Page numbers derive
+from the hardcover edition, first published in 1983 by George Allen &
+Unwin; no page numbers changed for the special 3-volume omnibus edition of
+2002 or the various trade-paper editions, all again now by Harper Collins
+or Houghton Mifflin.
+
+=back
+
+Other JRRT books fair game for quotes would thus include I<The Adventures of
+Tom Bombadil>, I<The Silmarillion>, I<Unfinished Tales>, and I<The Tale of
+the Children of Hurin>, all but the first posthumously assembled by CJRT.
+But I<The Lord of the Rings> itself is perfectly fine and probably best to
+quote from, provided you can find a suitable quote there.
+
+So if you were to supply a new, complete, top-level source file to add to
+Perl, you should conform to this peculiar practice by yourself selecting an
+appropriate quotation from Tolkien, retaining the original spelling and
+punctuation and using the same format the rest of the quotes are in.
+Indirect and oblique is just fine; remember, it's a metaphor, so being meta
+is, after all, what it's for.
+
+=head1 AUTHOR
+
+This document was written by Nathan Torkington, and is maintained by
+the perl5-porters mailing list.
+
+=head1 SEE ALSO
+
+L<perlrepository>