This is pokeint.info, produced by makeinfo version 7.2 from
pokeint.texi.

This manual describes the internals of GNU poke (version 5.0-modified,
20 May 2025).

   Copyright © 2024, 2025 The poke authors.

     You can redistribute it and/or modify this manual under the terms
     of the GNU General Public License as published by the Free Software
     Foundation, either version 3 of the License, or (at your option)
     any later version.
INFO-DIR-SECTION GNU poke
START-INFO-DIR-ENTRY
* pokeint: (poke). Internals of the GNU poke binary editor.
END-INFO-DIR-ENTRY


File: pokeint.info,  Node: Top,  Next: Nomenclature,  Up: (dir)

GNU poke Internals
******************

This manual describes the internals of GNU poke (version 5.0-modified,
20 May 2025).

   Copyright © 2024, 2025 The poke authors.

     You can redistribute it and/or modify this manual under the terms
     of the GNU General Public License as published by the Free Software
     Foundation, either version 3 of the License, or (at your option)
     any later version.

   GNU poke is an interactive, extensible editor for binary data.  Not
limited to editing basic entities such as bits and bytes, it provides a
full-fledged procedural, interactive programming language designed to
describe data structures and to operate on them.

   This manual is mainly a reference manual rather than a tutorial.  It
discusses how to contribute to GNU poke, the source tree structure and
build system, the overall architecture and a detailed description of the
internals of the program.  The target audience for this manual are GNU
poke developers, testers, and people interested in extending or
customizing the program to their needs.

* Menu:

* Nomenclature::

Contributing
* Development environment::
* Version control system::
* Running an uninstalled poke::
* Picking something to work on::
* Submitting a patch::
* Notifying about pushed commits::

Building poke
* Building basics::
* Partial builds::
* Building after a git pull::
* Building with local changes in gnulib or Jitter::
* Updating gettext::

Testing poke
* Writing tests::
* Continuous Integration::

Fuzzing poke
* Grammarinator::

Debugging poke
* Building with debugging support::
* Invoking GDB to debug an uninstalled poke::
* Debugging poke with GDB::
* Valgrind and poke::
* Debugging PVM assembly code::

Profiling poke
* Building poke with profiling support::

Hacking poke
* Maintenance targets::
* Writing compiler diagnostics::

Coding Style and Conventions
* Writing C::
* Writing Poke::
* Writing RAS::

Writing Documentation
* Documenting pickles::

Internals
* The Poke compiler::
* The Poke Virtual Machine::
* Memory management::

Indices
* Concept Index::               General concept index


File: pokeint.info,  Node: Nomenclature,  Next: Development environment,  Prev: Top,  Up: Top

1 Nomenclature
**************

We call “poke” the program.  When the context may induce confusion
(since "poke" is a pretty common word) then we use “GNU poke”.

   “Poke” (with upper case P) is the name of the domain-specific
language implemented by poke, the program.

   A “pickle” is a Poke source file containing definitions of types,
variables, functions, etc, that conceptually apply to some definite
domain.  For example, ‘elf.pk’ is a pickle that provides facilities to
poke ELF object files.  Pickles are not necessarily related to file
formats: a set of functions to work with bit patterns, for example,
could be implemented in a pickle ‘bitpatterns.pk’.


File: pokeint.info,  Node: Development environment,  Next: Version control system,  Prev: Nomenclature,  Up: Top

2 Development environment
*************************

The packages listed in the file ‘DEPENDENCIES’ are needed.
Additionally:

* Menu:

* Autotools::
* Dejagnu::
* Flex::
* Bison::
* GNU Jitter::


File: pokeint.info,  Node: Autotools,  Next: Dejagnu,  Up: Development environment

2.1 Autotools
=============

This distribution uses whatever versions of Automake, Autoconf, and
Gettext are listed in NEWS; usually the latest ones released.  If you
are getting the sources from git (or change configure.ac), you'll need
to have these tools installed to (re)build.  You'll also need help2man.
All of these programs are available from <ftp://ftp.gnu.org/gnu>.


File: pokeint.info,  Node: Dejagnu,  Next: Flex,  Prev: Autotools,  Up: Development environment

2.2 Dejagnu
===========

The poke testsuite uses DejaGNU. Please install it if you intend to run
the tests.  If you want to hack poke, you definitely want to run the
tests :)


File: pokeint.info,  Node: Flex,  Next: Bison,  Prev: Dejagnu,  Up: Development environment

2.3 Flex
========

You will need a recent version of flex, since we are using some recent
options like ‘reentrant’ or ‘bison-bridge’.  flex version 2.6.1 works
fine.


File: pokeint.info,  Node: Bison,  Next: GNU Jitter,  Prev: Flex,  Up: Development environment

2.4 Bison
=========

In order to improve the user's experience, we are using fairly recent
versions of bison.  Bison 3.6.0 or later is required to bootstrap and
build poke from git.


File: pokeint.info,  Node: GNU Jitter,  Prev: Bison,  Up: Development environment

2.5 GNU Jitter
==============

In order to build and run poke, you need Luca Saiu's GNU Jitter.
Jitter's git source repository, publicly readable over HTTP, is at
<http://git.ageinghacker.net/jitter> and its home page is
<https://www.gnu.org/software/jitter/>.

   The appropriate version of Jitter is downloaded and bootstrapped
automatically by Poke's ‘bootstrap’ script, which frees the user from
the annoyance of installing Jitter as a dependency.

   Configuring and compiling Poke will also compile and configure Jitter
in a subdirectory.  Jitter, when configured in “sub-package mode” as
Poke does, only generates static libraries and requires no installation.


File: pokeint.info,  Node: Version control system,  Next: Running an uninstalled poke,  Prev: Development environment,  Up: Top

3 Version control system
************************

* Menu:

* Git repos::
* Personal branches::


File: pokeint.info,  Node: Git repos,  Next: Personal branches,  Up: Version control system

3.1 Git repos
=============

This section lists the different repositories used in the development of
GNU poke.  The ‘https’ handlers provide read-only.  The ssh handlers are
for write access, where USERNAME is the handler of your user in
<https://savannah.gnu.org>.

<https://https.git.savannah.gnu.org/git/poke.git>
<USERNAME@git.savannah.gnu.org/srv/git/poke.git>
     Main repository, containing the source code of libpoke, the poke
     application, documentation, poked, pokefmt, and testsuite.  The GNU
     poke source distribution is compiled from this repo.

<https://https.git.savannah.gnu.org/git/poke/poke-mode.git>
<USERNAME@git.savannah.gnu.org/srv/git/poke/poke-mode.git>
     Repository for ‘poke-mode’, an Emacs major mode for editing Poke
     source code.  The poke-mode ELPA package is built from this repo.

<https://https.git.savannah.gnu.org/git/poke/poke-el.git>
<USERNAME@git.savannah.gnu.org/srv/git/poke/poke-el.git>
     Repository for ‘poke.el’, an Emacs interface to GNU poke.  The poke
     ELPA package is built from this repo.

<https://https.git.savannah.gnu.org/git/poke/poke/poke-elf.git>
<USERNAME@git.savannah.gnu.org/srv/git/poke/poke-elf.git>
     Repository for the poke ELF pickles project(1).

<https://https.git.savannah.gnu.org/git/poke/poke/poke-dwarf.git>
<USERNAME@git.savannah.gnu.org/srv/git/poke/poke-dwarf.git>
     Repository for the poke DWARF pickles project(2).

   ---------- Footnotes ----------

   (1) https://jemarch.net/poke-elf

   (2) https://jemarch.net/poke-dwarf


File: pokeint.info,  Node: Personal branches,  Prev: Git repos,  Up: Version control system

3.2 Personal branches
=====================

Anyone having write access to the git repository is allowed to push and
maintain personal branches.  These branches should be called ‘WHO/WHAT’,
where ‘WHO’ is the nick identifying the owner of the branch and ‘WHAT’ a
description of what it contains.

   Example:

     jemarch/hyperlinks-server

   Personal branches are intended to ease the interaction between
developers, and to provide a convenient basis for testing large changes.

   Personal branches can be rebased, and deleted.  Please do not write
into a personal branch unless you have the explicit approval of the
branch owner.


File: pokeint.info,  Node: Running an uninstalled poke,  Next: Picking something to work on,  Prev: Version control system,  Up: Top

4 Running an uninstalled poke
*****************************

Once poke is compiled, you can run it before installing by using the
‘run’ script built into the root of your build tree; this takes care of
setting variables such as ‘PATH’, ‘POKEDATADIR’, ‘POKESTYLESDIR’, and
‘POKEPICKLESDIR’ to point at appropriate locations.

   For example:

     $ pwd
     /home/jemarch/gnu/hacks/poke/build/
     $ ./run poke


File: pokeint.info,  Node: Picking something to work on,  Next: Submitting a patch,  Prev: Running an uninstalled poke,  Up: Top

5 Picking something to work on
******************************

We maintain a categorized and quite big list of tasks and ideas in the
‘TODO’ file which is located at the top level of the sources
distribution.  Some of the tasks listed there are simple, well defined,
and straightforward: these are ideal for first contributions.  Other
entries, however, describe full-fledged projects or even half baked
ideas that are only sketched barely.  These later will require
additional discussion in the mailing list.

   In any case, if you start working on some project, be it listed in
‘TODO’ or not, we highly recommend you let us know in the poke-devel
mailing list.

   If you would prefer to work on bug fixing, you can browse the open
bugs at
<https://sourceware.org/bugzilla/describecomponents.cgi?product=poke>.


File: pokeint.info,  Node: Submitting a patch,  Next: Notifying about pushed commits,  Prev: Picking something to work on,  Up: Top

6 Submitting a patch
********************

If you hack a feature/improvement/bugfix for poke and want to get it
integrated upstream, please keep the following points in mind:

   − If your patch changes the user-visible characteristics of poke,
     please include an update for the user manual.
   − If your patch adds or changes the way poke works internally, in a
     significant way, please consider including an update for the
     ‘HACKING’ file.
   − Please include a GNU-style ChangeLog in the patch description, but
     do not include it in the thunks.  This is to ease reviewers to
     apply your patch for testing.  Of course, include the thunk in the
     final push!  (We will get rid of manual ChangeLog entries soon.)
   − Make sure to run =make syntax-check= before submitting the patch,
     and fix any reported problem.  Note that the maintainer reviewing
     your patch will also do this, so this is a great time to save an
     iteration ;)
   − Let's keep poke.git master linear...  no merges please.  Pull with
     ‘--ff-only’.
   − Send the patch to the <poke-devel@gnu.org> mailing list.
   − Use text email only.  No html please.
   − Inline the patch in the body of your email, or alternatively attach
     it as ‘text/x-diff’ or ‘text/x-patch’.  This is to ease reviewers
     to quote parts of the patch.


File: pokeint.info,  Node: Notifying about pushed commits,  Next: Building basics,  Prev: Submitting a patch,  Up: Top

7 Notifying about pushed commits
********************************

Once you push one or more commits to some branch in the public git repo,
please send each patch as an email to the <poke-devel@gnu.org> mailing
list with a subject line like:

     [COMMITTED BRANCH] Summary of the change

where BRANCH is the name of the branch to which you pushed the
associated commit.  This applies to both patches applies as obvious and
to commits pushed by maintainers.

   This will help the other hackers subscribed to the mailing list to
follow the on-going developments, and even to catch some problem in any
of the pushed commits.


File: pokeint.info,  Node: Building basics,  Next: Partial builds,  Prev: Notifying about pushed commits,  Up: Top

8 Building basics
*****************

After getting the git sources, and installing the tools above, you can
run:

     $ ./bootstrap --skip-po

   Then, you can run ‘configure’, as described in the files ‘INSTALL’
and ‘INSTALL.generic’.  For example:

     $ mkdir build/ && cd build
     $ ../configure

Finally:

     $ make
     $ make check


File: pokeint.info,  Node: Partial builds,  Next: Building after a git pull,  Prev: Building basics,  Up: Top

9 Partial builds
****************

The poke source distribution encompasses several components: libpoke,
the poke application, the poked daemon, etc.  By default all the
components will be built provided their requirements are found at
configure time, but it is also possible to explicitly disable them when
invoking configure.

   For example, this is how you would build a poke without ‘poked’ and
without ‘pokefmt’:

     $ ../configure --disable-poked --disable-pokefmt

   Note however that if some of the components are disabled, the
resulting tree cannot be used to run ‘make dist’ on it.  It will fail.


File: pokeint.info,  Node: Building after a git pull,  Next: Building with local changes in gnulib or Jitter,  Prev: Partial builds,  Up: Top

10 Building after a git pull
****************************

Usually issuing a ‘make’ at the top build directory is enough to rebuild
poke after changes have been fetched from the source repo.  The Automake
maintainer-mode rules will make sure that autoconf and friends are run
again and do the right thing.

   However, when the Jitter submodule is updated, you have to first
bootstrap:

     $ ./bootstrap

   Then go to the top build directory, and issue a configure explicitly
before rebuilding:

     $ cd build && ../configure
     $ make

   You can't rely on the maintainer-mode rules there, because of certain
problem with the sub-package mode of Jitter.


File: pokeint.info,  Node: Building with local changes in gnulib or Jitter,  Next: Updating gettext,  Prev: Building after a git pull,  Up: Top

11 Building with local changes in gnulib or Jitter
**************************************************

If you have local changes to gnulib or jitter, that are not yet
registered as submodule reference commits, the way to invoke bootstrap
such that it doesn't delete your changes is:

     $ ./bootstrap --skip-po --no-git --gnulib-srcdir=gnulib --jitter-srcdir=jitter


File: pokeint.info,  Node: Updating gettext,  Next: Writing tests,  Prev: Building with local changes in gnulib or Jitter,  Up: Top

12 Updating gettext
*******************

When updating gettext, besides the normal installation on the system, it
is necessary to run ‘gettextize -f’ in this hierarchy to update the
‘po/’ infrastructure.  After doing so, rerun ‘gnulib-tool --import’
since otherwise older files will have been imported.  See the Gnulib
manual for more information.


File: pokeint.info,  Node: Writing tests,  Next: Continuous Integration,  Prev: Updating gettext,  Up: Top

13 Writing tests
****************

The poke testsuites live in the ‘testsuite/’ subdirectory.  This section
contains useful hints for adding tests there.

* Menu:

* Test framework::
* Run selected tests::
* Unit tests::
* Naming tests::
* Always set obase::
* Put each test in its own file::
* List the test files in testsuite/Makefile.am::
* dg-output may require a newline::
* Using data files in tests::
* Using NBD connections in tests::
* Writing tests that depend on a certain capability::
* Writing REPL tests::
* Command REPL tests::
* General REPL tests::
* Testing pickles::


File: pokeint.info,  Node: Test framework,  Next: Run selected tests,  Up: Writing tests

13.1 Test framework
===================

The test suite is based on DejaGnu, for which you find the general
documentation at
<https://www.gnu.org/software/dejagnu/manual/index.html>.

   The documentation of specific DejaGnu directives is at
<https://gcc.gnu.org/onlinedocs/gccint/Directives.html> and
<https://gcc.gnu.org/wiki/HowToPrepareATestcase>.


File: pokeint.info,  Node: Run selected tests,  Next: Unit tests,  Prev: Test framework,  Up: Writing tests

13.2 Run selected tests
=======================

Tests are categorized in different sub-directories.  You can run run
tests of a single category by running

     make check RUNTESTFLAGS=CATEGORY.exp

   For example, to run tests of =pkl= in =poke.pkl= sub-directory, run:

     make check RUNTESTFLAGS=pkl.exp

   You can also select tests using ‘TESTS’ environment variable.  As an
example, to run all test files which start with ‘array’, you can run:

     make check TESTS=array*.pk


File: pokeint.info,  Node: Unit tests,  Next: Naming tests,  Prev: Run selected tests,  Up: Writing tests

13.3 Unit tests
===============

We are using DejaGnu's support for unit tests.  See
‘testsuite/poke.libpoke’ for an example on how to write C unit tests.


File: pokeint.info,  Node: Naming tests,  Next: Always set obase,  Prev: Unit tests,  Up: Writing tests

13.4 Naming tests
=================

For testing a functionality =foo=, name your test =foo.pk= or ‘foo-N.pk’
where N is a number.

   If the test is a =do-compile= whose compilation is expected to fail,
name the test ‘func-diag.pk’ or ‘func-diag-N.pk’.  Here "diag" means
diagnostic.


File: pokeint.info,  Node: Always set obase,  Next: Put each test in its own file,  Prev: Naming tests,  Up: Writing tests

13.5 Always set obase
=====================

If your test relies on printing integer values in the REPL (or using the
‘%v’ formatting tag in a ‘printf’) please make sure to set an explicit
output numerical base, like in:

     /* { dg-command {.set obase 10} }  */

   This way, we won't have to change the tests if at some point we
change the default obase.


File: pokeint.info,  Node: Put each test in its own file,  Next: List the test files in testsuite/Makefile.am,  Prev: Always set obase,  Up: Writing tests

13.6 Put each test in its own file
==================================

If you are writing tests for a specific functionality, like for example
a standard function ‘foo’, it may seem logical to put all the tests in a
single file ‘foo.pk’ like:

     /* { dg-do run } */

     /* { dg-command {foo (1)} } */
     /* { dg-output "expected result" } */

     /* { dg-command {foo (1)} } */
     /* { dg-output "\nexpected result" } */

     [... and so on ...]

   However, this is not a good idea.  If some of the "subtests" fail, it
becomes difficult to determine which one is the culprit looking at the
test log file.

   It is better to put each test in its own file: ‘foo-1.pk’, ‘foo-2.pk’
and so on.


File: pokeint.info,  Node: List the test files in testsuite/Makefile.am,  Next: dg-output may require a newline,  Prev: Put each test in its own file,  Up: Writing tests

13.7 List the test files in ‘testsuite/Makefile.am’
===================================================

So that the tests get distributed, you need to list each newly added
test file in ‘testsuite/Makefile.am’.

   We have a syntax-check target that checks that ‘EXTRA_DIST’ contains
all the test files under ‘testsuite/’, and that there is no test listed
in ‘EXTRA_DIST’ that doesn't exist in the file system.


File: pokeint.info,  Node: dg-output may require a newline,  Next: Using data files in tests,  Prev: List the test files in testsuite/Makefile.am,  Up: Writing tests

13.8 dg-output may require a newline
====================================

If despite the advise above you really need to put more than dg-output
in a dg-run test file, please be aware you need to prefix all of them
(but the first one) with a newline, like in:

     /* { dg-output "foo" } */
     /* { dg-output "\nbar" } */
     /* { dg-output "\n baz" } */


File: pokeint.info,  Node: Using data files in tests,  Next: Using NBD connections in tests,  Prev: dg-output may require a newline,  Up: Writing tests

13.9 Using data files in tests
==============================

If your test requires mapping data, the dg-data directive is what you
need.  It has two forms.

   The first form looks like:

     /* { dg-data {c*} {0x10 0x20 0x30 0x40 ...} } */

   This creates a temporary file (whose name is irrelevant) and makes it
the current IO space at poke initialization time.  The test body, and
dg-commands, can then map on it.

   However, sometimes you need to map on several files.  In this case,
you can have several dg-data entries with an additional argument, which
is the name of the temporary file to create:

     /* { dg-data {c*} {0x01 0x02 ...} foo.data } */
     /* { dg-data {c*} {0x11 0x12 ...} bar.data } */

     [...]

     /* { dg-command { var foo = open ("foo.data") } } */

   The file created by the last dg-data (be it anonymous or named) is
the current IO space.


File: pokeint.info,  Node: Using NBD connections in tests,  Next: Writing tests that depend on a certain capability,  Prev: Using data files in tests,  Up: Writing tests

13.10 Using NBD connections in tests
====================================

If your test requires an NBD server (only useful when poke is compiled
with libnbd), the dg-nbd directive is what you need.  It has one form:

     /* { dg-nbd { 0x1 0x2 ...} [dg-tmpdir]/sock } */

   This utilizes nbdkit to serve an in-memory disk with initial contents
over a named Unix socket.  Note that the data argument is not the same
as for dg-data (which uses tcl's binary command), but rather the syntax
accepted by nbdkit-data-plugin's data= argument, although the two are
similar.  nbdkit then creates a Unix socket server for the data, and
will be shut down gracefully when the testsuite completes.  Use of the
utility directive [dg-tmpdir] ensures that the socket name will not be
too long while still respecting ‘$TMPDIR’ (defaulting to a new
subdirectory of ‘/tmp’), since ‘$objdir’ may be arbitrarily deep.

   To use the server as an IO space, your test will then follow up with:

     /* { dg-command "open (\"nbd+unix:///?socket=[dg-tmpdir]/sock\")" } */


File: pokeint.info,  Node: Writing tests that depend on a certain capability,  Next: Writing REPL tests,  Prev: Using NBD connections in tests,  Up: Writing tests

13.11 Writing tests that depend on a certain capability
=======================================================

Sometimes the presence of the functionality tested may be optional.  An
example is testing the support for styling, which depends on poke to be
built with libtextstyle support.

   The dg-require directive can be used for this purpose.  For example,
the following test will be skipped (with status UNSUPPORTED) if the
capability ‘libtextstyle’ is not found in poke:

     /* { dg-do run } */
     /* { dg-options "--color=html" } */
     /* { dg-require libtextstyle } */

     /* { dg-command {printf "%<foo:%i32d%>", 10} } */
     /* { dg-output "<span class=\"foo\">10</span>" } */

   IMPORTANT NOTE: dg-require should appear before any dg-command
directive in the test file.

   The supported capabilities are:

‘libtextstyle’
     poke is built with libtextstyle support.
‘nbd’
     poke is built with NBD io space support, and dg-nbd works.


File: pokeint.info,  Node: Writing REPL tests,  Next: Command REPL tests,  Prev: Writing tests that depend on a certain capability,  Up: Writing tests

13.12 Writing REPL tests
========================

The ‘poke.repl’ testsuite is intended to test features in the
interactive usage of poke.  Therefore, it is not dg-based.  Instead, it
uses the services provided by ‘testsuite/lib/poke.exp’.  In a nutshell,
these services are:

‘poke_start’
     Run a new poke process and wait at the prompt.
‘poke_exit’
     Exit poke.
‘poke_test_cmd CMD PATTERN’
     Send CMD to poke, and expect the result PATTERN.  CMD is sent as
     virtual keystrokes.  Therefore, sending ‘\t’ or ‘\n’ has the same
     effect on the REPL than typing <TAB> or <RET> in the keyboard,
     respectively.  Note that PATTERN shouldn't include the prompt.
‘poke_send INPUT PATTERN’
     Send INPUT to poke, and expect PATTERN as output.
‘poke_send_signal SIGNAL’
     Send SIGNAL to poke.

   Adding a new test to ‘poke.repl’ involves editing
‘poke.repl/repl.exp’ and adding some content there.  The following
subsections detail how.


File: pokeint.info,  Node: Command REPL tests,  Next: General REPL tests,  Prev: Writing REPL tests,  Up: Writing tests

13.13 Command REPL tests
========================

Some REPL tests need to check whether poke replies properly to some sent
command.  The ‘poke_test_cmd’ procedure defined in
‘testsuite/lib/poke.exp’ provides a convenient interface for this.

   For example:

     set test "slashes are preserved"
     poke_start
     poke_test_cmd {4 / 2} {2}
     poke_exit

   The snippet above implements a test named "slashes are preserved"
that runs poke and sends the command ‘4 / 2’ with expected result ‘2’.

   The dialogue when the test above is executed is:

     (poke) 4 / 2
     2
     (poke)


File: pokeint.info,  Node: General REPL tests,  Next: Testing pickles,  Prev: Command REPL tests,  Up: Writing tests

13.14 General REPL tests
========================

Other REPL tests are not about executing commands.  Suppose for example
we want to test whether tab-completion works.  We would write something
like:

     set test "tab-completion-2"
     poke_start
     poke_send ".e\t\t" "\r\n.editor  .exit *\r\n$poke_prompt .e"
     poke_exit

   The test above uses the ‘poke_send’ procedure, defined in
‘testsuite/lib/poke.exp’.  This procedure gets two arguments: the input
that is sent to poke, and the expected output.  Note how using
‘poke_send’ doesn't require poke to execute any command.

   Using ‘poke_send’ is more laborious than using ‘poke_test_cmd’: it is
necessary to explicitly include the prompt in the expected output
whenever needed.

   Note also how newlines are perceived by expect as the sequence
‘\r\n’.


File: pokeint.info,  Node: Testing pickles,  Prev: General REPL tests,  Up: Writing tests

13.15 Testing pickles
=====================

Each pickle in ‘pickles/FOO.pk’ shall have a test file
‘testsuite/poke.pickles/FOO-test.pk’ written using pickle pktest.

   If there are some features in the pickle that cannot be tested using
pktest (e.g., pickle uses ‘print’), that pickle also shall have a
testsuite in ‘testsuite/poke.FOO’ with a driver
‘testsuite/poke.FOO/FOO.exp’.


File: pokeint.info,  Node: Continuous Integration,  Next: Grammarinator,  Prev: Writing tests,  Up: Top

14 Continuous Integration
*************************

The package is built automatically, at regular intervals.  You find the
latest build results at
<https://gitlab.com/gnu-poke/ci-distcheck/pipelines>.

   We also have a set of "buildbots" hosted at Sourceware.  See
<https://builder.sourceware.org/>.  The sources for this are in the
Sourceware ‘builder.git’ repository at
<https://sourceware.org/git/?p=builder.git>.  The ‘README’ file contains
a good reference of how to hack on the build machinery.  The poke
homepage contains little graphics indicating the last execution results
of these buildbots.

   Parts relevant to poke are named ‘/.*gnupoke.*/’ in the Sourceware
buildbot configuration.  Please see the repository linked above for
details.


File: pokeint.info,  Node: Grammarinator,  Next: Building with debugging support,  Prev: Continuous Integration,  Up: Top

15 Grammarinator
****************

“grammarinator” is a random test generator/fuzzer that creates test
cases according to an input antlr4 grammar.

   In order to generate Poke programs with grammarinator, follow the
following steps.

   First of all, install grammarinator:

     $ git clone https://github.com/renatahodovan/grammarinator
     $ cd grammarinator
     $ sudo python3 setup.py install

   Then, use ‘grammarinator-process’ in order to create the "unparser"
and "unlexer" for poke:

     $ grammarinator-process ../path/to/poke.g4

   This will create two files in the current working directory:
‘pokeUnlexer.py’ and ‘pokeUnparser.py’.  Next step is to use
‘grammarinator-generate’ in order to create tests:

     $ grammarinator-generate -l pokeUnlexer.py -p pokeUnparser.py \
                              -r program -n 100 -d 10 \
                              -t grammarinator.runtime.simple_space_transformer

   The option ‘-n’ specifies the number of tests (Poke programs) to
generate.

   The option ‘-d’ specifies the maximum recursive level.  The bigger
the recursive level, the bigger the test files will be.  If you don't
specify a maximum recursive level then ‘grammarinator-generate’ will
crash.

   The option ‘-t grammarinator.runtime.simple_space_transformer’ makes
sure that whitespaces get inserted between literal tokens.


File: pokeint.info,  Node: Building with debugging support,  Next: Invoking GDB to debug an uninstalled poke,  Prev: Grammarinator,  Up: Top

16 Building with debugging support
**********************************

Short summary: at the present time Poke and its Jittery VM are not
especially difficult to debug; however the situation is going to change
as Jitter evolves and the following details will become more important
to Poke hackers.

   In order to make debugging easier you may want to disable advanced
dispatches in Jitter, which make the generated code harder to follow and
confuse GDB.

   Jitter by default will use the most efficient dispatch which is both
stable and available for the current configuration; the most efficient
dispatch is no-threading, which requires assembly support in Jitter for
the specific architecture and a supported binary format.
minimal-threading is more portable than no-threading, even if less
efficient.  Jitter supports two alternative dispatches, slower but very
stable, portable and friendly to debugging: “switch” and
direct-threading.

   Any dispatch can be selectively enabled or disabled from the Poke
‘configure’ script, by passing the following options which will be
automatically relayed to Jitter's ‘configure’ script:

‘--enable-dispatch-switch’
‘--disable-dispatch-switch’
‘--enable-dispatch-direct-threading’
‘--disable-dispatch-direct-threading’
‘--enable-dispatch-minimal-threading’
‘--disable-dispatch-minimal-threading’
‘--enable-dispatch-no-threading’
‘--disable-dispatch-no-threading’

   When configured in sub-package mode, as is the case when used with
Poke, Jitter will only actually compile the single most efficient
enabled dispatch.  This means that in order to select some sub-optimal
dispatch method, one has to disable the more efficient ones.  For
example, this is the best options for debugging:

     $ ../configure [...] --disable-dispatch-no-threading \
                          --disable-dispatch-minimal-threading \
                          --disable-dispatch-direct-threading

   Other than the configure options above, you will probably want to
configure the build to generate debugging information, and disable
compiler optimizations, using something like:

     ../configure CFLAGS="-O0 -g3" [...]

   Note that ‘-g3’ is particularly useful to debug the Poke compiler,
because it allows to refer by cpp macros by name in GDB and the compiler
uses macros very extensively.


File: pokeint.info,  Node: Invoking GDB to debug an uninstalled poke,  Next: Debugging poke with GDB,  Prev: Building with debugging support,  Up: Top

17 Invoking GDB to debug an uninstalled poke
********************************************

Since we are using libtool, ‘$(builddir)/poke/poke’ is a shell script.
In order to debug the uninstalled poke, you can do:

     $ ./libtool --mode=execute ./run gdb poke/poke


File: pokeint.info,  Node: Debugging poke with GDB,  Next: Valgrind and poke,  Prev: Invoking GDB to debug an uninstalled poke,  Up: Top

18 Debugging poke with GDB
**************************

Once GDB is invoked in either an installed or uninstalled poke, you can
debug it normally.

   Something to keep in mind, however, is that the Boehm GC we use uses
SIGPWR and SIGXCPU for internal purposes, in order to stop all threads
before starting collecting memory.  Proper handlers are installed by the
GC for these signals.  However, when you run poke under GDB, GDB will
stop once SIGPWR or SIGXCPU are signaled to the current thread.

   You can teach GDB to ignore these signals by issuing this command:

     handle SIGPWR noprint pass
     handle SIGXCPU noprint pass

   Alternatively, you can just ‘(c)ontinue’.


File: pokeint.info,  Node: Valgrind and poke,  Next: Debugging PVM assembly code,  Prev: Debugging poke with GDB,  Up: Top

19 Valgrind and poke
********************

The PVM uses the Boehm conservative garbage collector in order to manage
the memory used by the VM values.  Other parts of poke, such as the PKL
compiler, manage their own memory.

   Valgrind gets easily confused by the GCs tampering with the stack,
and emits a lot of spurious warnings.  Fortunately it is possible to
tell memcheck to omit these warnings: the file
‘etc/boehm-gc.suppressions’ contains a list of suppressions.

   Invoke valgrind with ‘--suppressions=etc/boehm-gc.suppressions’.

   If despite using the suppressions file you see some spurious warning,
please use:

     $ valgrind --tool=memcheck --gen-suppressions=all --log-file=raw.log

   Then process ‘raw.log’ with the ‘etc/parse-valgrind-suppressions.sh’
script, wildcard the result as much as you can, and append the results
to ‘etc/boehm-gc.suppressions’.

   In order to run the testsuite with valgrind, edit ‘lib/poke-dg.exp’
and uncomment the lines:

     set VALGRIND "valgrind --quiet \
        --suppressions=${srcdir}/../etc/boehm-gc.suppressions \
        --tool=memcheck --gen-suppressions=all"

   Finally run ‘make check’ as usual.


File: pokeint.info,  Node: Debugging PVM assembly code,  Next: Building poke with profiling support,  Prev: Valgrind and poke,  Up: Top

20 Debugging PVM assembly code
******************************

Hacking some areas of the compiler, such as the code generator pass,
involves meta-programming PVM assembler.  It is easy to find examples
anywhere in ‘libpoke/pkl-gen.c’:

     pkl_asm_insn (pasm, PKL_INSN_ROT);
     pkl_asm_insn (pasm, PKL_INSN_MULLU);
     pkl_asm_insn (pasm, PKL_INSN_NIP2);

   Or, alternatively, the code may be written in RAS in a ‘.pks’ file.
Like:

     .loop:
        bz @type, .endloop      ; ... A B
        mod @type               ; ... A B A%B
        rot                     ; ... B A%B A
        drop                    ; ... B A%B
        ba .loop

   Often, a run-time problem becomes apparent while the PVM executes the
generated code.  Typical cases are when a PVM value doesn't contain what
it's supposed to contain, and accessing the wrong boxed value causes a
segmentation fault (if we are lucky) or a non-crashing invalid memory
access (if we are very unlucky.)  Whenever that kind of crap happens, we
find ourselves in the need of debugging the PVM code, which is a big
PITA.

   Bad news are: we don't have a PVM debugger (yet).  Good news are: we
have a couple of tools that may help.

   The first of such tools is the ‘prints’ instruction.  This
instructions basically prints in the standard output the string value on
the TOS, and then drops it.  It is a wonderful way to trace PVM code.

   For example, let's say we are trying to find out how many times the
loop above gets executed.  We can install traces like:

           push "XXX entering loop\n"
           prints
     .loop:
           push "XXX in loop\n"
           prints
           mod @type
           rot
           drop
           ba .loop

   The other tool is the ‘strace’ instruction.  It prints the contents
of the stack (one value per line) from the TOS. It gets the number of
stack values to print as an argument, 0 meaning all of them.  It is very
useful in many situations, like when a loop is composing values in the
stack and something is going banana.  It is also useful to determine
what kind of value is being accessed by a given instruction.

   For example, let's say that we are hunting some segmentation fault.
We highly suspect the code generated in the first example in this
section, above.  Of the three instructions, ‘mullu’ is the only one that
could conceivably generate a segfault, so we add a stack trace
instruction right before it to inspect its two arguments:

     pkl_asm_insn (pasm, PKL_INSN_ROT);
     pkl_asm_insn (pasm, PKL_INSN_STRACE, 2); /* XXX remove me */
     pkl_asm_insn (pasm, PKL_INSN_MULLU);
     pkl_asm_insn (pasm, PKL_INSN_NIP2);

   We recompile, re-run, and we find out that the elements at the TOS
when ‘mullu’ is executed are a pair of stupid signed integers, which are
not boxed and not what the instruction expects.  Mystery solved.

20.1 PVM definition tips
========================

When using advanced Jitter dispatches it becomes important that every C
function or global variable accessed by a relocatable VM instruction is
“wrapped”: see the Jitter documentation [not yet existing at the time of
writing] for more information.  Non-relocatable instructions can access
any global without need for wrapping: the problem is strictly with VM
instruction whose native executable code gets copied to different memory
addresses, therefore breaking PC-relative access to globals.  Wrapping
prevents this kind of access, replacing it with a indirect access thru a
local pointer.

   It is important to remark that macros cannot be wrapped as functions:
rather it is necessary to consider any global variable or function
containsed in a macro expansion.

   Notice that high-latency VM instructions (for example instructions
performing I/O) and infrequently run instructions (for example an
instruction which raises an error) can be made non-relocatable without
loss of efficiency; in fact making large VM instructions non-relocatable
might improve code locality at run time.


File: pokeint.info,  Node: Building poke with profiling support,  Next: Maintenance targets,  Prev: Debugging PVM assembly code,  Up: Top

21 Building poke with profiling support
***************************************

poke supports a couple of commands that can be used to profile Poke
programs:

     .vm profile reset
     .vm profile show

   For these commands to work you need to build poke with profiling
support.  This is done by passing the ‘--enable-pvm-profiling’ command
line option to ‘configure’.


File: pokeint.info,  Node: Maintenance targets,  Next: Writing compiler diagnostics,  Prev: Building poke with profiling support,  Up: Top

22 Maintenance targets
**********************

This section describes ‘make’ targets that performs several maintenance
tasks.

‘syntax-check’
     Run several syntax-related checks in the source files.  It is
     useful to run this target before submitting code to be reviewed,
     and while reviewing other people's code.

     Note that sometimes the results have to be taken with a pinch of
     salt.  This happens, for example, when a rule oriented to C is
     applied to, say, an AWK file.  In these cases, consider adding a
     ‘.x-sc_*’ fine-tuning file.  But please ask in poke-devel first.

     We provide a pre-push git hook that performs the syntax check right
     before pushing.  If the check fails, the push is aborted.  You
     should consider enabling this check by coping
     ‘etc/git-hooks/pre-push’ to ‘.git/hooks/’.

‘coverage’
     This target builds *poke* with code coverage support, runs the
     testsuite, and generates a nice html report under
     ‘$(top_builddir)/doc/coverage/’.  It is necessary to have the
     ‘lcov’ program for this to work.  Note also that you need to have
     jitter configured to use a dispatch model other than no-threading.

‘cyclo-poke.html’
     This target runs the pmccabe utility to calculate the modified
     cyclomatic complexity of the functions composing the poke sources,
     and generates a nice html report.

‘refresh-po’
     This target download the latest available translations from the
     translation project and installs them in the source tree.

‘update-copyright’
     Run this rule once per year (usually early in January) to update
     all the copyright years in the project.  By default this excludes
     all variants of COPYING. Exceptions to this procedure (such as
     ‘ChangeLog..*’ for rotated change logs) can be added in the file
     ‘.x-update-copyright’.


File: pokeint.info,  Node: Writing compiler diagnostics,  Next: Writing C,  Prev: Maintenance targets,  Up: Top

23 Writing compiler diagnostics
*******************************

Please follow the following guidelines when adding new errors or
warnings to the poke compiler:

   − Make the diagnostic messages self-contained, i.e.  do not rely on
     the extra location information to make the error intelligible.
     This is mainly for the sake of accessibility, and also for when
     poke is run in quiet mode.

     This is an example of a bad diagnostic:

          error: field doesn't exist in struct

     Write something like this instead:

          error: field `foo' doesn't exist in struct

   − When checking for the type of some ENTITY, please use a two-lines
     error message like:

          error: invalid ENTITY
          error: expected EXPECTED_TYPE, got ACTUAL_TYPE

     this is to keep some uniformity.  You can get the written
     representation of a type using the =pkl_type_str= utility function.

   − When referring to integer types in diagnostic messages, please
     refer to them as “integrals”.

   − Always pass 1 for ‘use_given_name’ when invoking ‘pkl_type_str’
     unless you need the expanded definition of the type for whatever
     reason (like to show the arguments taken by a function type.)


File: pokeint.info,  Node: Writing C,  Next: Writing Poke,  Prev: Writing compiler diagnostics,  Up: Top

24 Writing C
************

In Poke we follow the GNU Coding Standards.  Please see
<https://www.gnu.org/prep/standards/>.

   Additionally, we have a few conventions that we ask you to follow
when hacking poke, in order to keep a coherent style when possible.
These are documented in the following subsections.

24.1 Avoid Tabs
===============

Please format your code using spaces.  Tabs (Ascii 9) should not appear
anywhere except in Makefiles.

24.2 Avoid ‘bool’
=================

C99 defines the type ‘bool’.  However, in poke we prefer to use
variables of type ‘int’ in order to hold truth values.  We kindly ask
you to do the same.

   When documenting functions that receive or return boolean values
stored in integer variables, please remember that the interpretation of
the truth values in C is: zero means false, and any value other than
zero means true.  Avoid referring to 1 to mean true.

   Also, please never rely on a boolean to hold exactly 1 or any other
particular non-zero value, to be interpreted as true.

24.3 Use ‘_p’ for predicates
============================

We follow the venerable convention of using a suffix ‘_p’ (for
predicate) when naming variables and struct fields that contain
booleans.

   We do not always do this, but it is especially important in cases
where the name of the variable is not indicative enough.

24.4 Documenting functions in public headers
============================================

Function prototypes in headers must be documented using a comment block
preceding the prototype.

   Start the comment block with a terse short sentence that states what
the function does.  Then, document the arguments that the function
takes, and the value it returns, if any.  When referring to arguments
and other variables in the comment block, please use caps.

   It is ok for single comment block to document more than one
prototype.

   Please leave an empty line between the comment block and the function
prototype.

   Example:

     /* Frobnicate a frob.

        FROB is a frob that should have been previously initialized using
        frob_new.

        LAZY_P is a boolean.  If true, FROB won't be frobnicated
        immediately, but instead marked for later frobnication.

        If the frob cannot be frobnicated, this function returns NULL.
        Otherwise the function returns the given frob.  */

     frob_t frobnicate (frob_t frob, int lazy_p);
     frob_t forbnicate_r (frob_t frob, int lazy_p);


File: pokeint.info,  Node: Writing Poke,  Next: Writing RAS,  Prev: Writing C,  Up: Top

25 Writing Poke
***************

We recommend to use the Emacs mode in ‘etc/poke-mode.el’ to write ‘.pk’
files.

25.1 Writing offset units
=========================

Do not separate magnitudes and units when writing offsets.  Do it like
this:

     16#B

instead of:

     16 #B

25.2 Use ‘Camel_Case’ for type names
====================================

Use ‘Camel_Case’ for type names, but do not use ‘Camel_Case’ for
variable/function names!

25.3 Formatting conventions in pretty-printers
==============================================

Surround pretty-printed values with ‘#<’ and ‘>’.  This is to notify the
reader that the value has been pretty-printed.


File: pokeint.info,  Node: Writing RAS,  Next: Documenting pickles,  Prev: Writing Poke,  Up: Top

26 Writing RAS
**************

We recommend to use the Emacs mode in ‘etc/poke-ras-mode.el’ to write
‘.pks’ files.

26.1 Calling compiled Poke from RAS
===================================

A big part of the code generated by the PKL code generator is written by
hand, in ‘.pks’ files.  Sometimes it is convenient to call compiled Poke
code from that code; for example, to execute complex formulae or to
perform some operation defined recursively.

   In these situations, the solution is to first write a Poke function
in the compiler's runtime library, ‘pkl-rt.pk’, like:

     fun _pkl_foo = (uint<64> ival, uint<64> eval,
                     uint<32> ivalw, uint<32> fieldw) uint<64>:
     {
       return ival | (eval <<. (ivalw - fieldw));
     }

   Then, the function can be invoked from the hand-written RAS file
using the ‘.call’ directive:

     .call _pkl_foo

   Note how the compiler internal functions use the ‘_pkl_’ prefix.
Also note that ‘.call’ can only invoke functions defined in the global
environment.


File: pokeint.info,  Node: Documenting pickles,  Next: The Poke compiler,  Prev: Writing RAS,  Up: Top

27 Documenting pickles
**********************

The chapter ‘Pickles’ in the poke user manual contains several sections
organized in thematic areas.  When adding a new pickle, find the area
appropriate for the abstractions provided by the pickle (or create one
if it doesn't exist already) and add a section for it.

   Please make sure to document any dependency of your pickle to other
pickles.

   Keep a tutorial-like tone: for fine details, the user can always go
and look to the pickle directly.


File: pokeint.info,  Node: The Poke compiler,  Next: The Poke Virtual Machine,  Prev: Documenting pickles,  Up: Top

28 The Poke compiler
********************

* Menu:

* Compiler overview::
* The bison parser in pkl-tab.y::
* The AST::
* Lexical handling::
* Compiler passes and phases::


File: pokeint.info,  Node: Compiler overview,  Next: The bison parser in pkl-tab.y,  Up: The Poke compiler

28.1 Compiler overview
======================

This figure depicts the architecture of the compiler:

           /--------\
           | source |
           \---+----/
               |
               v
       +-----------------+
       |      Parser     |
       +-----------------+
       |  analysis and   |
       | transformation  |
       |     phases      |
       +-----------------+
       | code generation |
       |      phase      |
       +-----------------+
       | Macro assembler |
       +-----------------+
               |
               v
          /---------\
          | program |
          \---------/


File: pokeint.info,  Node: The bison parser in pkl-tab.y,  Next: The AST,  Prev: Compiler overview,  Up: The Poke compiler

28.2 The bison parser in ‘pkl-tab.y’
====================================

The only purpose of the bison parser in pkl-tab.y is to do the syntactic
analysis, build the initial AST, and set the locations of the AST nodes.

   Unfortunately, currently it also does some extra work, due to
limitations in the LARL parser:

   - It builds the compile-time environment and register type, variable
and function declarations.  - It links return statements with their
containing functions.

   As we shall see below, any further analysis and transformations on
the AST are performed by the compiler phases, which are implemented
elsewhere.  This greatly helps to keep the parser code clean and easy to
read, and also eases changing the syntactic structure of poke programs.


File: pokeint.info,  Node: The AST,  Next: Lexical handling,  Prev: The bison parser in pkl-tab.y,  Up: The Poke compiler

28.3 The AST
============

The compiler operates on an abstract syntax tree that represent the
program being compiled.  The following subsections discuss some useful
aspects of the structure of the AST.

* Menu:

* loop_stmt::
* try_stmt::
* exp::
* cond_exp::


File: pokeint.info,  Node: loop_stmt,  Next: try_stmt,  Up: The AST

28.3.1 loop_stmt
----------------

Loop statements are represented with trees like:

     loop_stmt
     |
     +-- [iterator]
     |       |
     |       +-- decl
     |       +-- container
     |
     +-- [condition]
     +-- body

   Different kind of loop statements supported in Poke are represented
including or not including optional fields.

   ‘while’ statements with a condition:

     while (CONDITION) BODY

are represented as:

     loop_stmt
         |
         +-- condition
         +-- body

   ‘for’ statements without a condition:

     for (;;) BODY

are represented as:

     loop_stmt
         |
         +-- body

   ‘for-in’ statements:

     for (VAR in CONTAINER) BODY

are represented as:

     loop_stmt
         |
         +-- iterator
         |      |
         |      +-- decl (VAR
         |      +-- container
         |
         +-- body

   ‘for-in-where’ statements:

     for (VAR in CONTAINER where SELECTOR) BODY

are represented as:

     loop_stmt
         |
         +-- iterator
         |      |
         |      +-- decl (VAR)
         |      +-- container
         |
         +-- condition (SELECTOR)
         +-- body


File: pokeint.info,  Node: try_stmt,  Next: exp,  Prev: loop_stmt,  Up: The AST

28.3.2 try_stmt
---------------

There are several syntactic forms of this statement:

     try CODE catch HANDLER
     try CODE catch if EXP HANDLER
     try CODE catch (ARG) HANDLER
     try CODE until EXP

   Try statements are represented with trees like:

     try_stmt
     |
     +-- try_stmt_body
     |   |
     |   +-- code
     |
     +-- try_stmt_handler
     |   |
     |   +-- code
     |
     +-- [arg]
     +-- [exp]

   where ‘arg’ and ‘exp’ are exclusive.


File: pokeint.info,  Node: exp,  Next: cond_exp,  Prev: try_stmt,  Up: The AST

28.3.3 exp
----------

Expressions are the application of some “operator” to one or more
operators.  The Poke language supports unary, binary and ternary
operators.

   Expressions are represented with trees like:

     exp
     |
     +-- code
     +-- attr
     +-- flag
     +-- numops
     +-- operands[3]
         |
         +-- [operand]
         +-- [operand]
         +-- [operand]

   Where ‘code’ identifies the operator of the expression.  Operators
are defined in the ‘libpoke/pkl-ops.def’ file.

   If the operator is an attribute, then ‘attr’ identifies the invoked
attribute.  Attributes are defined in the ‘libpoke/pkl-attrs.def’ file.
This field ‘attr’ is initialized in the ‘trans1’ phase.

   The ‘exp’ AST node is used for all the Poke unary and binary
operators, which only use two of the operands.

   Additionally, the application of an attribute to which an argument is
passed, like in:

     foo'elem (1)

   Is encoded as an ‘exp’ node with three operands: the first operand
holds ‘foo’, the second operand an identifier for the attribute ‘elem’
and the third operand the expression passed to the attribute (in this
case 1).  A handler in ‘trans1’ transforms this ternary expression node
in a binary expression, filling ‘attr’ in the process.

   Note that the Poke ternary conditional operator ‘? :’ is _not_
implemented using ‘exp’ nodes, but ‘cond_exp’ AST nodes.


File: pokeint.info,  Node: cond_exp,  Prev: exp,  Up: The AST

28.3.4 cond_exp
---------------

This AST node is used to encode Poke's ternary conditional operator,
which looks like:

     cond_exp ? then_exp : else_exp

   It uses trees like this:

     cond_exp
     |
     +-- cond
     +-- thenexp
     +-- elseexp


File: pokeint.info,  Node: Lexical handling,  Next: Compiler passes and phases,  Prev: The AST,  Up: The Poke compiler

28.4 Lexical handling
=====================

The parser needs to build the lexical enviroment as it works through the
source code.  This is because it needs to find the declarations of all
the entities referenced by variable names.  Also, the Poke syntax
requires to determine whether a given identifier is a type name or not.
It uses a ‘pkl_env’ for this, and it pushes and pops frames according to
the language rules.  The parser annotates each variable reference with
the declaration of the entity being referred, but does not set the
variable's lexical address.  This is because this address may not be
final, as we shall see below.  Once the parsing is done successfully the
lexical environment should consist on just the top-level definitions,
which is kept for the rest of the compilation.  But the inner frames are
no longer available.

   After the parser, the front-end compiler passes are run.  This
includes analysis and checking of language rules, transformation, type
analysis, constant folding, etc.  These passes can do arbitrary
transformations of the tree, and some of these transformations can alter
the lexical structure of the program.  The only exception is the
top-level environment.  Therefore, these passes shall not rely on the
lexical structure.

   Finally, just before the code generator, a lexical pass is run.  This
pass contains a ‘transl’ phase.  At this point no further
transformations of the AST will be performed and it is final.  The
‘transl’ phase reconstructs the lexical environment (the top-level
environment is preserved) and assigns the lexical addresses (back, over)
to all variable references.  It also performs a few language rules
checks that rely on the relative lexical position of entities.


File: pokeint.info,  Node: Compiler passes and phases,  Prev: Lexical handling,  Up: The Poke compiler

28.5 Compiler passes and phases
===============================

* Menu:

* The pkl-passes.def file::
* Naming conventions for phases::
* Naming conventions for phase handlers::


File: pokeint.info,  Node: The pkl-passes.def file,  Next: Naming conventions for phases,  Up: Compiler passes and phases

28.5.1 The ‘pkl-passes.def’ file
--------------------------------

The ‘pkl-passes.def’ file describes the different “passes” performed by
the compiler front-end, along with its constituent “phases”.  This is an
example of a pass definition:

     PKL_PASS(PKL_PASS_F_TYPES,1)
          PKL_PHASE(trans1)
          PKL_PHASE(anal1)
          PKL_PHASE(typify1)
          PKL_PHASE(promo)
          PKL_PHASE(trans2)
          PKL_PHASE(fold)
          PKL_PHASE(trans3)
          PKL_PHASE(typify2)
          PKL_PHASE(anal2)
     PKL_END_PASS

   When running a pass, each phase will be applied in order to each node
of the abstract syntax tree.  See the documentation in ‘pkl-pass.h’ for
more detailed information about the compiler's pass manager.


File: pokeint.info,  Node: Naming conventions for phases,  Next: Naming conventions for phase handlers,  Prev: The pkl-passes.def file,  Up: Compiler passes and phases

28.5.2 Naming conventions for phases
------------------------------------

We use the following convention to name phases:

     {NAME}{SUFFIX}

   Where ‘NAME’ reflects a phase category (see below) and ‘SUFFIX’ is
usually an integer that specifies the order in which the phases are
applied.  Thus, for example, ‘name4’ is performed after ‘name1’.
Sometimes, ‘SUFFIX’ is ‘f’ (meaning "final").

   The suffix is not used if there is only one phase in the given
category.

   We use the following phase categories:

‘anal’
     For phases whose main purpose is to perform checks on the AST,
     and/or the contents of the AST nodes, and emit errors/warnings.
‘trans’
     For phases whose main purpose is to alter the structure of the AST,
     and/or the contents of the AST nodes.
‘typify’
     For phases whose main purpose is to perform type checks, and
     otherwise do work on types.
‘promo’
     For phases whose main purpose is to perform coercions wherever
     appropriate.  Currently there is only one phase in this category.
‘fold’
     For phases whose main purpose is to pre-compute areas of the AST
     whenever it is possible to do so at compile-time.  Currently there
     is only one phase in this category, that performs constant folding.
‘gen’
     For phases whose main purpose is to generate PVM code.  Currently
     there is only one phase in this category.

   The phases in category ‘NAME’ are implemented in the source files
‘libpoke/pkl-NAME.[c]’.


File: pokeint.info,  Node: Naming conventions for phase handlers,  Prev: Naming conventions for phases,  Up: Compiler passes and phases

28.5.3 Naming conventions for phase handlers
--------------------------------------------

We use the following convention to name phase handlers:

     pkl_PHASE_{ps,pr}_NODE

   Where ‘PHASE’ can be a complete phase name (like ‘typify1’) if the
handler is to be installed in that phase only, or a phase category name
(like ‘typify’) if the handler is to be installed in several phases in
that category.  If the phase is to be executed in pre-order, ‘pr’
follows, otherwise, ‘ps’.  Finally, ‘NODE’ is the name of the AST node.

   For example, the handler:

     pkl_anal1_ps_comp_stmt

is installed in the phase ‘anal1’, executes in post-order, and serves
the AST nodes with code ‘PKL_AST_COMP_STMT’.


File: pokeint.info,  Node: The Poke Virtual Machine,  Next: Memory management,  Prev: The Poke compiler,  Up: Top

29 The Poke Virtual Machine
***************************

* Menu:

* PVM types::
* Exception handling::
* Signal handling::
* Offsets and bit-offsets in the PVM::
* Writing relocatable instructions::
* Avoiding branching instructions::
* Wrapping globals::


File: pokeint.info,  Node: PVM types,  Next: Exception handling,  Up: The Poke Virtual Machine

29.1 PVM types
==============

29.1.1 How ‘any’ is handled in the PVM
--------------------------------------

The Poke type ‘any’ is not implemented in the PVM as a PVM type value.
Instead, ‘null’ is used to denote it, by convention.

   Note that it is not possible to use ‘void’ to denote the Poke ‘any’
type, because of closure types: a closure returning ‘void’ is not the
same than a closure returning ‘any’.


File: pokeint.info,  Node: Exception handling,  Next: Signal handling,  Prev: PVM types,  Up: The Poke Virtual Machine

29.2 Exception handling
=======================

Exception types or codes are signed 32-bit integers, and are defined in
‘libpoke/pkl-rt.pk’ as ‘EC_*’ variables.

   The exceptions themselves are struct values of type ‘Exception’, also
defined in ‘libpoke/pkl-rt.pk’.

   There are two ways an exception can be raised in the PVM:

   − Explicitly, when the instruction =raise= is executed.
   − Implicitly, when some instruction needs to fail.  For example, an
     integer division instruction divides by zero.

   In either case, the treatment of a raised exception is the same:

  1. Pop an exception handler from the exception handler stack.
  2. If the exception handler matches the raised exception type, then
       1. Restore the heights of the main and return stacks.
       2. Restore the dynamic environment.
       3. Push the cached exception type to the stack.
       4. Branch to the exception handler.
  3. Repeat.

   The default exception handler, which catches "unhandled" exceptions,
is installed by the macro-assembler in ‘libpoke/pkl-asm.c:pkl_asm_new’
and ‘libpoke/pkl-asm.c:pkl_asm_finish’.  It calls the function
‘_pkl_exception_handler’, that is defined in the compiler runtime in
‘libpoke/pkl-rt.pk’.


File: pokeint.info,  Node: Signal handling,  Next: Offsets and bit-offsets in the PVM,  Prev: Exception handling,  Up: The Poke Virtual Machine

29.3 Signal handling
====================

When the PVM is about to execute a program, it installs a signal handler
that, at the moment, handles ‘SIGINT’.  This signal handler is defined
in the ‘late-c’ block in pvm.jitter, and records the signal in the PVM
internal state.

   Then, when a ‘sync’ instruction is executed, it checks the PVM
internal state and raises a ‘PVM_E_SIGNAL’ exception if there is some
pending signal.  The default signal handler just ignores this signal,
but the user can also intercept it if desired.  The PKL compiler
generates ‘sync’ instructions in strategic places, such as before
backward jumps in loops.

   The ‘exit’ PVM instruction clears the pending signals in the internal
PVM state.


File: pokeint.info,  Node: Offsets and bit-offsets in the PVM,  Next: Writing relocatable instructions,  Prev: Signal handling,  Up: The Poke Virtual Machine

29.4 Offsets and bit-offsets in the PVM
=======================================

The PVM supports a ‘pvm_off’ boxed value, to denote pairs of magnitudes
and units.  Both accessor macros (in ‘pvm-val.h’) and PVM instructions
(‘ogetm’, ‘ogetu’) are provided to access their components.

   Many other PVM entities need to denote offsets in a way or another.
For example, struct fields in ‘pvm_struct’ values need to record their
relative offset with respect the beginning of the struct.

   It may come to mind, quite naturally, to use ‘pvm_off’ values to
denote these offsets.  It is very elegant.  However, we decided to use
“bit offsets” instead, stored in 64-bit ‘pvm_long’ values.

   There are two reasons for this:

   − First of all, performance.  It is fairly common to operate with the
     absolute value of these offsets, in bits.  In fact, in most cases
     that is the only purpose of maintaining them.  Having them stored
     in ‘pvm_off’ values means we have to multiply every time we want to
     get their magnitude.  This is a waste, for no good reason.
   − To avoid code coupling.  PVM offsets are very cool, but they are
     also complex: the unit is arbitrary.  This means in many cases we
     have to assume the nature of the unit, mainly bits.  This is very
     fragile.

   So, the take-home message is: in the PVM, restrict the presence of
‘pvm_off’ values to the ones generated by the code generator.  Whenever
an offset is needed in some internal PVM structure, use bit-offsets
instead encoded as ‘ulong<64>’ values.


File: pokeint.info,  Node: Writing relocatable instructions,  Next: Avoiding branching instructions,  Prev: Offsets and bit-offsets in the PVM,  Up: The Poke Virtual Machine

29.5 Writing relocatable instructions
=====================================

When operating in non-threaded mode, Jitter likes to move the code
compiled for instruction bodies around.  This imposes several
restrictions on what can you have in the instruction bodies:

   − If you call a global function from your instruction, please add it
     to the ‘wrapped-functions’ section in the file.
   − Likewise, if you use a global variable, please add the variable to
     the ‘wrapped-globals’ section.
   − Do not use any constant C string like ‘"foo"’; these are global
     variables in disguise and you cannot wrap them unless you give them
     a name.

   If you cannot follow the guidelines above for whatever reason, please
mark the instruction as ‘non-relocatable’ like this:

     instruction foo ()
       non-relocatable
       code
         ...
       end
     end

   This will prevent Jitter to relocate the instruction body and crash
in many platforms and non-threading modes.


File: pokeint.info,  Node: Avoiding branching instructions,  Next: Wrapping globals,  Prev: Writing relocatable instructions,  Up: The Poke Virtual Machine

29.6 Avoiding branching instructions
====================================

Another cause for a Jitter instruction to be non-relocatable is when it
branches mid-instructions.  In practice this often happens when an
instruction can raise an exception.  The typical example would be a
division instruction ‘div’, which may be defined as raising a "division
by zero" exception if division by zero is found.  Jitter requires such
an instruction to be marked as “branching” like this:

     instruction div
       branching
       code
        ...
       end
     end

   In the PVM we always must avoid having branching instructions like
the above, since it has a performance impact.  A way to avoid it is
having the instruction push some value on the stack that signifies the
execution found a problem, then get the compiler check for that value
and raise the exception.  As an example, if the ‘div’ instruction pushes
‘PVM_NULL’ on the stack if division by zero is attempted, the compiler
can generate this to check for it and raise the corresponding exception:

       div
       bnn .ok
       pushe E_division_by_zero
       raise
     .ok:
       drop


File: pokeint.info,  Node: Wrapping globals,  Prev: Avoiding branching instructions,  Up: The Poke Virtual Machine

29.7 Wrapping globals
=====================

The non-threaded Jitter operation modes require every global (variable
or function) to be wrapped, i.e.  specified in sections
‘wrapped-functions’ or ‘wrapped-globals’ in ‘pvm.jitter’.

   Since it is not trivial to keep track of every used global, having
hundred of instructions, we are providing a syntax-check that looks for
this automatically.

   In order to avoid false positives, there are two files that we
maintain by hand:

‘etc/pvm-wraps-whitelist’
     This file contains a list of _global_ symbols that are used in
     pvm.jitter but are not required to be listed in neither
     ‘wrapped-function’ nor ‘wrapped-globals’.  These are variables and
     functions that, despite being global and undefined, are not used
     _within_ PVM instructions.
‘etc/pvm-wraps-static-whitelist’
     This file contains a list of _static_ symbols that are defined in
     ‘pvm.jitter’ and also listed in either ‘wrapped-functions’ or
     ‘wrapped-globals’.  This includes for example the ‘pvm_*’ wrappers
     we define in the file, to be used in PVM instructions.

29.7.1 Wrapping standard function calls
---------------------------------------

If we need to call to a standard library function provided by the system
(such as ‘printf’, ‘strstr’, or the like) from the body of a PVM
instruction, we have to consider the possibility of gnulib replacing
such function with another having a name like ‘rpl_printf’ or
‘rpl_strstr’.

   Since the ‘wrapped-functions’ list in ‘pvm.jitter’ is static and the
referred functions must exist, just wrapping the ‘rpl_*’ symbols doesn't
work.

   So instead, for every standard function ‘FOO’ we call from PVM
instructions, we:

   − Create a wrapper for it, called ‘pvm_FOO’, and put it in the
     ‘pvm-wrappers.[ch]’ source files.
   − Use ‘pvm_FOO’ in the PVM instruction.
   − Add ‘pvm_FOO’ to the wrapped-functions section in ‘pvm.jitter’.


File: pokeint.info,  Node: Memory management,  Next: Concept Index,  Prev: The Poke Virtual Machine,  Up: Top

30 Memory management
********************

Different parts of poke use different strategies for memory management:

   − The compiler front-end uses reference counting to handle AST nodes.
   − The PVM uses the Boehm GC collector for values and the run-time
     environment.
   − Everything else uses ‘malloc’/‘free’.

   This sometimes leads to tricky situations, some of which are
documented in the sections below.

* Menu:

* Using ASTREF::
* Using ASTDEREF::
* PVM values in PVM programs::
* PVM values in AST nodes::


File: pokeint.info,  Node: Using ASTREF,  Next: Using ASTDEREF,  Up: Memory management

30.1 Using ‘ASTREF’
===================

The AST uses reference counting in order to manage the memory used by
the nodes.  Every time you store a pointer to an AST node, you should
use the ‘ASTREF’ function in order to increase its counter:

     pkl_ast_node foo = ASTREF (node);

   Note that the ‘pkl_ast_make_*’ constructors do ‘ASTREF’ internally,
so you don't need to use it in calls like:

     pkl_ast_node new = pkl_ast_make_struct (ast, 5, elems_node);

   To make sure you get the reference counting right, it's required to
use the return value of ‘ASTREF’ function.  The compiler will warn you
otherwise.


File: pokeint.info,  Node: Using ASTDEREF,  Next: PVM values in PVM programs,  Prev: Using ASTREF,  Up: Memory management

30.2 Using ‘ASTDEREF’
=====================

‘ASTDEREF’ decreases the reference counter of the provided AST node.
It's required to use the return value of ‘ASTDEREF’.  The compiler will
warn you otherwise.

   In practice you will seldom find yourself in the need to use
‘ASTDEREF’.  Just make sure that every ‘ASTREF’ is paired with a
‘pkl_ast_node_free’.

   However, there are situations where ‘ASTDEREF’ is necessary in order
to avoid a memory leak.  For example, consider transformations like ‘a
-> b’ to ‘a -> x -> b’.  In that case, you should use something like:

     b = PKL_AST_KIND_WHAT (node);
     x = pkl_ast_make_xxx (ast, ASTDEREF (b));
     PKL_AST_KIND_WHAT (node) = ASTREF (x);

   This works because ‘pkl_ast_make_xxx’ does an ‘ASTREF’ to ‘b’
internally.  The final result is that the reference counter of ‘b’
doesn't change at all.


File: pokeint.info,  Node: PVM values in PVM programs,  Next: PVM values in AST nodes,  Prev: Using ASTDEREF,  Up: Memory management

30.3 PVM values in PVM programs
===============================

PVM routines (data structures of type =pvm_routine=) are allocated by
Jitter in complicated data structures, internally relying on ‘malloc’.
Their content is therefore not automatically visible to the GC.

   Now, the instructions in a routine can contain literal PVM values,
and some of these values will be boxed.  For example, the following
routine contains a pointer to a ‘pvm_val_box’:

     ;; Initialize the element index to 0UL, and put it
     ;; in a local.
     push ulong<64>0
     regvar $eidx

   There is only one place where jitter routines are stored in other
data structures: the PVM program abstraction.

   PVM programs are allocated using ‘pvm_alloc’, i.e.  by the GC. They
are composed by a PVM routine and other fields:

     struct pvm_program
     {
       pvm_routine routine;
       [...]
     }

   However, since ‘routine’ is malloc-allocated, the GC can't traverse
it.  Consequently, the references to contained boxed values won't be
accounted for, and these values will be collected if there are no more
references to them!

   The solution, recommended by Luca Saiu, is to keep an array of
pointers in the PVM program structure, containing the pointers to every
boxed value used in ‘routine’:

     struct pvm_program
     {
       pvm_routine routine;
       void **pointers;
       [...]
     };

   The pointers are collected in the ‘pvm-program.c’ functions that
append parameters to instructions.


File: pokeint.info,  Node: PVM values in AST nodes,  Prev: PVM values in PVM programs,  Up: Memory management

30.4 PVM values in AST nodes
============================

Storing a PVM value (whose memory is handled by GC) in an AST node
(reference-counted) leads to a problem: the GC is unable to see the
reference to values, and will therefore collect the memory if there are
no other reachable references.

   Let's see an example of this.  Both array and struct types contain
closures for mappers, writers, bounders, and the like.  For example,
let's take arrays:

     struct pkl_ast_type
     {
         ...
         struct
         {
           union pkl_ast_node *bound;
           union pkl_ast_node *etype;
           pvm_val mapper;
           pvm_val writer;
           pvm_val bounder;
         } array;
     }

   In this case, ‘mapper’, ‘writer’ and ‘bounder’ are managed by GC.
However, the contained ‘struct pkl_ast_type’ is not.  There is no way
the GC can find these nodes thru the AST node.

   The solution is to allocate memory for these values as
“uncollectable”.  This is done with ‘pvm_alloc_uncollectable’.  For
example:

     pkl_ast_node
     pkl_ast_make_array_type (pkl_ast ast, pkl_ast_node etype, pkl_ast_node bound)
     {
       ...
       /* Prevent GC to collect these PVM values.  */
       PKL_AST_TYPE_A_CLOSURES (type)
           = pvm_alloc_uncollectable (nclosures * sizeof (pvm_val));
       PKL_AST_TYPE_A_MAPPER (type) = PVM_NULL;
       PKL_AST_TYPE_A_WRITER (type) = PVM_NULL;
       PKL_AST_TYPE_A_BOUNDER (type) = PVM_NULL;
       PKL_AST_TYPE_A_CONSTRUCTOR (type) = PVM_NULL;
       PKL_AST_TYPE_A_INTEGRATOR (type) = PVM_NULL;
       PKL_AST_TYPE_A_FORMATER (type) = PVM_NULL;
       PKL_AST_TYPE_A_PRINTER (type) = PVM_NULL;

       return type;
     }

   And of course, before the memory of the AST node is freed, the
uncollectable memory shall be explicitly fred in ‘pkl_ast_node_free’:

     void
     pkl_ast_node_free (pkl_ast_node ast)
     {
       ...
       case PKL_AST_TYPE:
         switch (PKL_AST_TYPE_CODE (ast))
           {
           ...
           case PKL_TYPE_ARRAY:
             pvm_free_uncollectable (PKL_AST_TYPE_A_CLOSURES (ast));
           }
     }

   If you add PVM values or PVM environments to an AST node, please
follow the strategy above.


File: pokeint.info,  Node: Concept Index,  Prev: Memory management,  Up: Top

Appendix A Concept Index
************************


Tag Table:
Node: Top572
Node: Nomenclature2750
Node: Development environment3538
Node: Autotools3855
Node: Dejagnu4318
Node: Flex4593
Node: Bison4863
Node: GNU Jitter5144
Node: Version control system5906
Node: Git repos6134
Ref: Git repos-Footnote-17697
Ref: Git repos-Footnote-27734
Node: Personal branches7773
Node: Running an uninstalled poke8516
Node: Picking something to work on9085
Node: Submitting a patch10040
Node: Notifying about pushed commits11547
Node: Building basics12295
Node: Partial builds12771
Node: Building after a git pull13508
Node: Building with local changes in gnulib or Jitter14320
Node: Updating gettext14836
Node: Writing tests15332
Node: Test framework16033
Node: Run selected tests16478
Node: Unit tests17084
Node: Naming tests17353
Node: Always set obase17758
Node: Put each test in its own file18252
Node: List the test files in testsuite/Makefile.am19129
Node: dg-output may require a newline19735
Node: Using data files in tests20265
Node: Using NBD connections in tests21302
Node: Writing tests that depend on a certain capability22538
Node: Writing REPL tests23678
Node: Command REPL tests24831
Node: General REPL tests25563
Node: Testing pickles26526
Node: Continuous Integration27023
Node: Grammarinator27897
Node: Building with debugging support29416
Node: Invoking GDB to debug an uninstalled poke31927
Node: Debugging poke with GDB32353
Node: Valgrind and poke33178
Node: Debugging PVM assembly code34497
Node: Building poke with profiling support38667
Node: Maintenance targets39190
Node: Writing compiler diagnostics41246
Node: Writing C42608
Node: Writing Poke45205
Node: Writing RAS45986
Node: Documenting pickles47141
Node: The Poke compiler47753
Node: Compiler overview48045
Node: The bison parser in pkl-tab.y48782
Node: The AST49679
Node: loop_stmt50066
Node: try_stmt51318
Node: exp51884
Node: cond_exp53424
Node: Lexical handling53746
Node: Compiler passes and phases55623
Node: The pkl-passes.def file55908
Node: Naming conventions for phases56805
Node: Naming conventions for phase handlers58511
Node: The Poke Virtual Machine59386
Node: PVM types59760
Node: Exception handling60300
Node: Signal handling61687
Node: Offsets and bit-offsets in the PVM62581
Node: Writing relocatable instructions64340
Node: Avoiding branching instructions65531
Node: Wrapping globals66860
Node: Memory management69012
Node: Using ASTREF69663
Node: Using ASTDEREF70391
Node: PVM values in PVM programs71423
Node: PVM values in AST nodes73083
Node: Concept Index75439

End Tag Table


Local Variables:
coding: utf-8
End:
